Key takeaways
- NVIDIA Dynamo can split large-language-model inference into separate prefill and decode worker pools, allowing each phase to be scaled and placed independently.
- NVIDIA's NIXL transfer layer moves the key-value cache from the prefill engine to the decode engine without making the GPU wait for the copy to finish.
- In one Microsoft AKS test with a prefix-heavy agent workload, KV-aware routing cut average time to first token from 53.877 seconds to 2.658 seconds and average end-to-end latency from 84.517 seconds to 19.761 seconds.
- Those results are workload-specific. They show the value of reusing cached prefixes in that test, not a guaranteed speedup for every model or deployment.
Why Dynamo separates prefill and decode
An inference request has two phases with different jobs. Prefill processes the input prompt and builds the key-value cache the model needs. Decode uses that cache while producing output tokens.
NVIDIA's Dynamo documentation describes prefill as more compute-bound and decode as more constrained by memory and active cache. When both phases share one worker pool, a long prompt can compete with requests that are already generating output. Disaggregation puts the phases in separate pools, so operators can scale their prefill and decode capacity independently.
That architectural change does not make a deployment faster by itself. It adds a cache transfer between pools and more routing decisions. The payoff depends on whether better specialization, queue isolation and cache reuse outweigh that extra movement and coordination.
How a disaggregated request moves through Dynamo
NVIDIA documents a four-part flow:
- A prefill router selects a prefill worker, using either load balancing or information about cache overlap and load.
- The selected worker processes the prompt and returns transfer metadata with the resulting KV cache.
- The router attaches that metadata to the decode request and sends it to a decode worker.
- NIXL coordinates the direct cache transfer before decoding continues.
The important boundary is the KV cache. The decode worker needs the state produced during prefill, so moving that state efficiently determines whether separating the pools is worthwhile.
What NIXL does
NIXL is NVIDIA's data-transfer layer for distributed inference. In Dynamo's documented disaggregated path, it transfers KV cache directly from the prefill engine's GPU memory to the decode engine's GPU memory. The transfer is non-blocking, allowing the prefill GPU to continue serving other work while data moves.
Dynamo can use the transport available to the deployment, including NVLink on a single system or InfiniBand with UCX across suitable nodes. This is why deployment topology matters: a design that works well inside one rack may behave differently once cache traffic crosses a network.
NIXL handles movement; it does not decide where a request should go. That choice belongs to Dynamo's routing layer.
Why KV-aware routing can beat round robin
Round-robin routing spreads requests evenly, but it ignores whether a worker already holds a useful prefix cache. KV-aware routing considers both cache overlap and current worker load. A request may be routed to a worker that can reuse cached prompt state instead of recomputing it.
This is most relevant when prompts share substantial prefixes. Agent systems are a clear example: many requests repeat the same tool definitions and instructions before adding new user input. With little prefix reuse, there is less redundant prefill work for the router to eliminate.
Microsoft's AKS Engineering Blog tested this idea with NVIDIA Dynamo using Qwen3-32B on eight H100 GPUs across four Azure nodes. The trace had high prefix sharing, making it a favorable test of cache-aware routing.
| Metric | Round robin | Dynamo KV Router | Reported change |
|---|---|---|---|
| Average time to first token | 53,877 ms | 2,658 ms | about 20.4× faster |
| P99 time to first token | 280,221 ms | 17,585 ms | about 15.9× faster |
| Average end-to-end latency | 84,517 ms | 19,761 ms | about 4.3× faster |
| P99 end-to-end latency | 340,006 ms | 90,299 ms | about 3.8× faster |
These are the strongest numbers in the evidence, but their scope matters. They compare two routing strategies in a specific cluster, model and request trace. They do not establish a universal Dynamo gain, a cost-per-token reduction, or a hardware recommendation.
What operators should measure
A useful evaluation starts with the workload, not the headline speedup:
- Measure how often prompts share reusable prefixes and how long those prefixes are.
- Track time to first token separately from total response time; routing can affect them differently.
- Compare an aggregated baseline with the disaggregated design under the same request mix.
- Watch prefill and decode queues independently to see whether one pool is becoming the bottleneck.
- Include cache-transfer time and network saturation in capacity tests.
- Re-test after changing the model, context length, serving engine or node topology.
The AKS result makes a strong case for testing KV-aware routing when prefix reuse is high. It does not remove the need to benchmark the exact production workload.
Bottom line
NVIDIA Dynamo disaggregation is best understood as a resource-allocation and routing tool. Separate prefill and decode pools create room to scale the two phases independently. NIXL moves the KV cache between them, while the router tries to avoid repeated prefill work by considering cache locality as well as load.
For prefix-heavy requests, Microsoft's AKS test shows that this can materially reduce both first-token and end-to-end latency. For other workloads, the right decision depends on measured cache reuse, transfer overhead and queue behavior—not on applying the reported 20.4× result as a blanket promise.
Sources
This article was researched and fact-checked against the following sources:
- Scaling multi-node LLM inference with NVIDIA Dynamo and NVIDIA GPUs on AKS (Part 3) | AKS Engineering Blog (blog.aks.azure.com)
- Disaggregated Serving | NVIDIA Dynamo Documentation (docs.nvidia.com)
