Disaggregated Prefill and Decode for Long-Context Request Handling
Separating prompt and token generation onto different hardware eliminates bottleneck conflicts.

Prefill and decode are not two halves of one computation. They are two separate computations with opposite bottlenecks, running on the same hardware largely because early inference serving systems had no reason to split them apart. Prefill takes the full input prompt and processes it in a single dense forward pass: every token attends to itself and to every token before it, subject to the causal mask, and the resulting matrix multiplications fill up the GPU's compute budget. Prefill is compute-bound. Decode works differently. It produces one token at a time, and each new token requires loading the entire accumulated key-value cache, covering every prior token, from memory before the GPU can do anything with it. The actual arithmetic at each decode step is small relative to the volume of data that has to move. Decode is memory-bandwidth-bound.
This split has been framed as a dynamic operator scheduling problem: prefill's operator mix is dominated by GEMM, the dense matrix multiplications that reward raw compute throughput, while decode's operator mix is dominated by attention over a KV state that keeps growing with every step and rewards memory bandwidth instead. Inside a single GPU, prefill and decode draw on different resource pools of the same chip at the same time, competing rather than cooperating.
The quantization literature shows the same split from a different angle. During prefill, the cost of loading model weights gets amortized across many tokens processed in parallel, which makes quantizing activations worthwhile. During decode, where batch size per request effectively drops to one, each weight gets used for exactly one multiply-add, so compressing the weights themselves matters far more than compressing activations. The hardware mismatch between the two phases carries a quantization-format mismatch along with it.
Running both phases on the same GPU turns the mismatch into an operational problem, as a long prompt arriving mid-generation forces its way through the prefill computation before the GPU can return to the decode batches already in flight. This is head-of-line blocking, and it is the direct, measurable cost of asking one piece of hardware to serve two workloads with opposite resource profiles.
Splitting Pools and Hardware in Disaggregation

Disaggregation addresses the compute-versus-bandwidth conflict by giving each phase its own hardware pool, matched to its own bottleneck. It does not make the coupling between prefill and decode go away. It moves that coupling somewhere specific: into the transfer of the KV cache from one pool to the other.
The architecture itself is simple to state. One set of GPU nodes handles prefill: you run the full prompt through a forward pass, and it produces a KV cache block as output. A second set of nodes receives that cache and carries out the autoregressive generation, one token at a time. Because the two workloads have different bottlenecks, they can be matched to different chips: prefill nodes benefit from compute-dense GPUs such as the H100, while decode nodes benefit from memory-bandwidth-dense GPUs such as the H200. The B200 sits across both categories: it offers substantially higher compute throughput than the H200 alongside greater memory bandwidth, so it can suit either role. The hardware choice follows from which bottleneck a given pool needs to solve.
Separating the pools also separates their scaling behavior. Operators facing a surge of long-prompt traffic can add prefill nodes without touching decode capacity, and operators facing a burst of short, chatty requests can add decode nodes without over-provisioning compute they don't need. A collocated serving setup can't make that trade at all, since every node has to be sized for both jobs at once.
But provisioning prefill and decode separately raises its own allocation problem: the ratio of prefill nodes to decode nodes has to satisfy latency targets across a mixed request population, and that ratio shifts with the distribution of prompt lengths and the level of concurrency the system is handling at any given moment. It is not a setting operators choose once.
Disaggregation sits in a wider family of serving optimizations, and it is worth placing against its neighbors to see what it is and isn't. Chunked prefill interleaves prefill and decode work on the same node rather than splitting them onto separate hardware, trading some of disaggregation's elasticity for simpler deployment. Speculative decoding attacks a different problem entirely, using a smaller draft model to propose tokens that a larger model verifies, speeding up decode's own token generation. Each technique solves a distinct failure mode, and disaggregation's target is specifically the hardware mismatch described above.
What disaggregation buys, concretely, is the removal of head-of-line blocking: a long prompt arriving on the prefill pool no longer stalls decode batches running elsewhere. What it costs is a new sequential step between the two phases. The interface between prefill and decode is no longer shared GPU memory sitting on one chip. It is a KV cache block that has to be physically carried across a network boundary for every single request, and the latency, bandwidth, and placement of that handoff now set the pace for the whole system.
The KV cache handoff as the engineering core of disaggregation

The KV cache handoff is not a networking detail to be wired up after the compute split is working. At realistic context lengths and realistic concurrency, it is the binding constraint on whether disaggregation delivers the latency and throughput gains it promises.
The size of that handoff follows directly from the shape of the model and the request. KV cache volume scales with sequence length, the number of layers, the number of KV heads, and the head dimension, and the formula is fixed: twice the number of layers, times the number of KV heads, times the head dimension, times the sequence length, times the number of bytes per stored element. For a large model handling a long prompt, that arithmetic produces a cache block substantial enough to matter for a single request on its own, and the entire block has to cross from the prefill node to the decode node before decode can do anything.
That transfer sits directly on the critical path for time-to-first-token. The decode node cannot emit a single output token until it holds the complete KV cache handed off by the prefill node, making the transfer a hard sequential dependency inserted between prefill finishing and decode starting. Where network bandwidth is thin relative to the size of the cache being moved, the case for disaggregation weakens, because the gains promised by splitting the hardware get absorbed by the time spent waiting for data to arrive.
The scale of this problem is growing. As context lengths increase, KV cache state in modern serving architectures can exceed the footprint of the model weights themselves, so the amount of data that must transfer between pools keeps growing as a share of overall traffic. Fragmentation inside the cache adds a separate complication: naive contiguous memory allocation wastes a large share of reserved memory, and paged allocation schemes such as PagedAttention cut that waste substantially. Even a well-managed, tightly packed set of KV blocks still has to move as a unit before decode can begin, so better allocation reduces memory pressure but does not remove the sequential handoff itself.
The practical consequence for engineers building these systems is specific: treat KV transfer as an afterthought to be solved once compute disaggregation works, and transfer latency will end up dominating the time-to-first-token budget while the expected throughput gains fail to show up under real load. The handoff has to be designed as a first-class part of the system from the start, not patched in once the rest of the architecture is in place.
Transfer mechanisms and network-layer failure modes
The transport layer that carries the KV cache from prefill to decode is not just a question of raw bandwidth. Protocol choice, path selection, and fallback behavior together decide whether disaggregation runs reliably in production or degrades quietly while every health check still reports green.
NIXL, the library used for this transfer, supports several backends, each with different latency and bandwidth properties: UCX, which itself covers RoCE, InfiniBand, and TCP; GDS; Libfabric; Mooncake's own transfer engine; DOCA GPUNetIO; S3-compatible object storage; and Azure Blob. If you need a high-performance route for moving KV cache data, RDMA-based paths, specifically InfiniBand and RoCE, give you direct memory-to-memory transfer with minimal CPU involvement. TCP exists as a fallback, and it carries substantially higher overhead. It can also become the active transport silently, if RDMA is misconfigured, without triggering any obvious error.
That silent failure mode is not hypothetical. A practitioner debugging a production vLLM disaggregated deployment running a large FP8 model on bare-metal Kubernetes found that the prefill and decode workers came up correctly and the model loaded without any visible problem, yet KV cache transfer had fallen back to TCP. Nothing in the compute-level health checks caught it, because those checks confirm that the model loaded and that forward passes run, not that the transfer path between pools is using the fast route. The case points to a structural asymmetry in how these systems fail: the compute path is straightforward to verify, since a loaded model and a working forward pass are easy to confirm, while the transfer path can degrade substantially and still look healthy from the outside.
Not every KV transfer need is about per-request handoff speed. NVMe over Fabrics and S3-compatible backends matter when KV state has to be persisted or shared across nodes that don't sit on a common RDMA fabric, which comes up directly in prefix caching and cross-instance cache reuse rather than in the single prefill-to-decode handoff for one request.
Recent research on intra-GPU KV management treats the transport layer as a design variable chosen deliberately, rather than infrastructure to be assumed, because the backend selected has a direct effect on end-to-end time-to-first-token. On the orchestration side, NVIDIA Dynamo sits above vLLM and routes prefill and decode work to dedicated worker pools, relying on NIXL to carry out the actual transfer between them. NVIDIA's ICMSP, announced at CES 2026, takes a different approach to the same problem, using BlueField-4 DPUs to manage KV cache movement off the GPU's compute path entirely, so the GPU's streaming multiprocessors stay occupied with math while I/O proceeds in parallel.
KV placement strategy across the memory hierarchy
Where a KV cache block physically lives matters as much as how fast it moves. Deciding which blocks sit in GPU HBM, CPU DRAM, local NVMe, or remote object storage, and when to promote a block to faster memory or evict it to slower memory, can eliminate large portions of prefill computation outright, which changes when full disaggregation is even necessary.
The memory hierarchy available for KV storage runs from GPU HBM, fastest and smallest, down through CPU DRAM, local NVMe SSD, and remote object storage, each tier trading latency against capacity. Mooncake, built by Moonshot AI and presented at USENIX FAST 2025, with a related paper accepted at ACM Transactions on Storage in October 2025 and published online that November, treats the KV cache as a first-class distributed storage object that spans this entire hierarchy. Rather than confining KV state to GPU memory, Mooncake's architecture draws on CPU, DRAM, SSD, and network-interface resources that would otherwise sit underused, turning them into distributed cache storage capacity.
Prefix caching shows what good placement is worth in concrete terms. LMCache extends prefix caching so that previously computed KV blocks persist across server restarts and across multiple server instances, backed by Redis or by disk storage. On a long shared system prompt, that persistence drops time-to-first-token from roughly eleven seconds to roughly one and a half seconds, because the system replaces the full prefill computation with a cache hit instead of recomputing the same prompt prefix from scratch.
The same placement logic is being studied at finer grain. Work on intra-GPU KV management shows that even inside a single device, where KV blocks sit relative to the compute units reading them affects whether prefill and decode interfere with each other, a finding that generalizes outward to how blocks should be placed across multiple nodes. Separate work proposing resolution-adaptive KV caching with hierarchical semantic memory keeps high-resolution KV data for the tokens that carry the most semantic weight on GPU, while storing compressed SVD representations of everything else on CPU, reconstructing detail on demand rather than keeping every token at full resolution all the time. Research on asymmetric KV treatment has also found that keys and values carry different sensitivity profiles: concentrated spectra in query and key weights produce feature homogeneity, but more dispersed spectra in value weights preserve heterogeneity, and this can guide which blocks deserve fast memory and which you can safely offload.
You don't make placement a decision just once per request and then leave it alone. As a session's context grows, the right tier for a given cache block changes, so cache management has to run as an ongoing scheduling process, not a fixed allocation made at the start of a request. The frontier of this work pushes the idea further still: a proposed "Prefill-as-a-Service" direction, from Qin and colleagues at Moonshot AI, describes serving the KV caches of future models across datacenters entirely, decoupling where prefill happens geographically from where decode happens.
How long-context workloads change the disaggregation calculus
Long-context requests do not scale up the ordinary serving picture. They turn trade-offs that are optional at short context into structural requirements, because KV cache volume, transfer time, and memory pressure all grow with sequence length in ways that remove collocated serving as a realistic option.
KV cache volume grows with every additional token, since each one adds entries across every layer and every attention head in the model. At very long context, a large model can produce cache blocks big enough to exhaust GPU HBM across even a modest batch size, well before compute becomes the limiting factor. Surveys of long-context serving describe a regime where KV state routinely outgrows the model's own weight footprint, and once that happens, standard HBM allocation strategies stop working on their own terms. Tiered placement across the memory hierarchy becomes mandatory for these workloads.
MLPerf's reasoning benchmark class targets a reasoning-model class of comparable scale in its September 2025 round, and it puts this pressure under direct measurement. Chain-of-thought inference extends the effective sequence length of a request well beyond its visible prompt and output, so it hits both KV capacity and transfer bandwidth hard, at the same time. In that workload class, disaggregation is no longer a design choice weighed against collocated serving but the only architecture capable of meeting competitive latency and throughput targets.
Sources
- SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
- Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling
- Disaggregated Quantization:Specializing LLM Prefill and Decode
- Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving
- Taming the Titans: A Survey of Efficient LLM Inference Serving
- Beyond the Buzz: A Pragmatic Take on Inference Disaggregation
- KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging
- SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference


