RDMA Over InfiniBand for KV Cache Transfer Between Prefill and Decode Nodes

Disaggregated inference requires efficient KV cache transfer to keep decode nodes from stalling.

· 8 min read
Cover illustration for “RDMA Over InfiniBand for KV Cache Transfer Between Prefill and Decode Nodes”
Distributed Serving · October 23, 2026 · 8 min read · 1,897 words

RDMA over InfiniBand for KV cache transfer between prefill and decode nodes is a memory, network, and scheduling problem. For LLM inference throughput, tiered storage works when hot KV blocks stay in GPU HBM and reusable blocks in DRAM or NVMe can return without stalling decode. NVMe adds capacity, but its effect on cost per token depends on whether retrieval preserves useful GPU work.

Why disaggregated inference puts KV transfer on the critical path

GPU utilization dashboards crater during decode on colocated serving systems. Prefill saturates tensor cores because it is compute-bound, while decode runs at a fraction of GPU utilization because it is memory-bandwidth-bound. These workloads compete when they share the same hardware. A long prompt arriving mid-generation can stall every concurrent decode stream, creating head-of-line blocking that continuous batching alone cannot fix.

Disaggregation puts each phase on hardware that suits it. That separation creates a new dependency, however. The KV cache produced during prefill must arrive at the decode node before generation begins, putting the handoff on the critical path for Time-To-First-Token (TTFT). Microsoft's Splitwise, presented at ISCA 2024, documented that compute scales faster across GPU generations than memory bandwidth, strengthening the case for specialization while making efficient transfer increasingly consequential.

What the KV cache actually is and why its size makes transfer hard

The KV cache contains the key and value vectors computed for every token in the context. It is not a small piece of request metadata. The complete state needed for generation must reach the decode node, and its size grows with context length and model depth. A request can therefore require tens of gigabytes of network transfer before decode emits a single token.

That size makes bandwidth a request-level latency constraint. A larger context means more bytes must cross the fabric within the time available for the handoff. Payloads also differ from one request to the next, so a transfer path that works for shorter prompts can cause most of the delay for longer ones.

How CPU-staged transfers create a multi-hop bottleneck

The default CPU-staged path moves data from GPU HBM across PCIe into host DRAM, through the NIC to remote host DRAM, and across PCIe again into remote GPU HBM. This six-hop route uses cudaMemcpy or an equivalent operation for host staging. Both endpoints pay for the intermediate copies. Each stage adds latency and consumes host memory bus bandwidth, with total penalties on the order of tens to hundreds of milliseconds that can dominate TTFT at long context lengths.

NCCL, the standard GPU collective library, is optimized for bulk transfers of contiguous memory, while PagedAttention manages KV cache in non-contiguous physical blocks. With those layouts, the data must move in many small transfers or go through extra reshaping and copying before it can move in one bulk operation. The host memory bus can become saturated while the NIC remains underutilized, leaving the decode GPU idle as the cache drains through the staging path.

GPUDirect RDMA and one-sided verbs as the engineering response

GPUDirect RDMA changes the data path by allowing the NIC to access registered GPU memory directly, so KV tensors can move between GPU HBM regions over InfiniBand or RoCE without staging through host DRAM. Memory region registration through ibv_reg_mr or DMA-BUF makes that access possible. The defining property is zero intermediate staging copies at both endpoints, rather than faster execution of the same CPU-bounce sequence.

One-sided RDMA Write also removes the decode-side send/receive handshake. The prefill node can write into a pre-negotiated remote VRAM address as each layer's KV tensors become ready, without the decode CPU posting a receive for that transfer. Scatter-gather-capable verbs can address fragmented KV layouts without first consolidating every block into a contiguous staging buffer. This does not eliminate all coordination costs. Memory registration, address negotiation, and arrival signaling remain engineering responsibilities even though host CPUs no longer relay the payload.

Push versus pull: how transfer mode shapes TTFT and orchestration complexity

AMD's MORI-IO connector for vLLM, built on the open-source MORI (Modular RDMA Interface) framework, implements both write and read modes through VLLM_MORIIO_CONNECTOR_READ_MODE. In write, or push, mode, the proxy dispatches to prefill and decode simultaneously. Prefill pushes each layer's KV data into decode memory as computation proceeds. Decode can then begin when prefill finishes and the required state is available, without a serial dispatch step between the phases.

Read mode makes that orchestration simpler but serial. The proxy waits for prefill to finish, extracts remote block IDs, and forwards them to the decode node, which pulls the cache through RDMA. The connector demonstrates lower TTFT with write mode, while both modes outperform standard colocated serving. Push therefore exchanges simpler orchestration for tighter coordination, because decode buffers must be registered and ready before prefill begins.

The proxy's role extends beyond choosing transfer direction. The proxy must also track which decode nodes hold the KV blocks needed by repeat requests or multi-turn sessions to avoid moving them needlessly. Sending a request to the node that already holds its cache can avoid the transfer altogether. Transfer coordination and prefix-cache routing thus need the same information on where blocks are held.

How InfiniBand topology and bandwidth tier determine whether transfer stays off the critical path

Even when RDMA is used correctly, the fabric still needs enough bandwidth. Context length, model size, and concurrency determine how much KV data the network must drain within the latency budget. InfiniBand rail topology provides multiple paths between compute nodes, but those paths may also serve model-parallel all-reduce traffic. Without isolation or deliberate scheduling, KV transfers compete with tensor-parallel communications and introduce unpredictable latency.

Huawei's CloudMatrix-Infer isolates KV transfers on a dedicated RDMA plane, separate from control and storage operations, while model-parallel compute uses the distinct UB plane. It combines that separation with asynchronous scheduling and model-aware connection grouping to distribute traffic across the fabric. Physical separation is only part of the design. Scheduling must also prevent transfer readiness from unnecessarily coupling prefill progress to decode scheduling.

DeepSeek-AI and collaborators' DualPath paper identifies a different topological imbalance in agentic workloads with high KV reuse. Prefill storage NICs become saturated while decode storage NICs remain idle. Its additional storage-to-decode path loads cache into decode engines and forwards it to prefill through RDMA over the compute network. The design demonstrates why KV movement needs its own bandwidth budget and path planning, including traffic that begins in storage.

Memory registration, buffer lifecycle, and the hidden costs of managing RDMA at request scale

Every GPU memory region used for RDMA must first be registered with the HCA. Registration pins physical pages, assigns an rkey, and makes the region addressable by the remote NIC. That setup is not free at request scale. With PagedAttention-style allocation, a naive implementation registers and deregisters regions each time requests allocate or release blocks, and for small blocks this extra work can take almost as long as the transfer.

Moonshot AI and Tsinghua University's Mooncake treats fragmented GPU memory indices as a first-class transfer primitive. Its RDMA queue pairs move fragmented KV data without additional copying, while a common transfer semantics API covers RDMA, NVLink, TCP, and NVMe-oF. One interface can thus handle registration and path selection for all these protocols. The Mooncake ACM paper reports that topology-aware selection reduced cross-node latency by 40% compared with Gloo, showing that queue pair and path management cannot be reduced to blind assignment.

A fixed GPU memory pool registered at startup avoids per-request registration, but reserves HBM for the transfer machinery, reducing the memory available for KV storage. Session setup adds another lifecycle cost because rkeys and remote VRAM addresses must be exchanged before data moves. Amortizing that negotiation across sessions or batches avoids repeatedly inserting the same round-trip into the request path.

Where the storage tier enters: NVMe and CXL as overflow for KV cache that cannot stay on GPU

Long contexts and high concurrency can exhaust GPU HBM. The canonical hierarchy keeps hot blocks in HBM, completed but reusable warm blocks in CPU DRAM, and cold historical context or future prefix-cache candidates on NVMe SSDs. ICMSP describes the bandwidth differences across these tiers. HBM delivers multiple terabytes per second, PCIe 5.0 access to CPU DRAM delivers tens of gigabytes per second, and NVMe SSDs deliver single-digit gigabytes per second with microsecond-range latency.

NVMe has more room to keep KV state, but loading cold blocks while requests are running can stall GPUs because the I/O path cannot feed data as fast as decode uses it. Cost per token thus depends on how much state is kept and reused and how long compute waits to retrieve it, as well as how much storage is available.

Marvell Technology's A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference explores pooled memory beyond conventional DRAM overflow. Its photonic-CXL design addresses limitations of electrical CXL pooling, rather than making lower-tier access equivalent to HBM. Across NVMe, DRAM, and new memory pools, cache placement metadata and RDMA path selection must agree on where blocks are held so that requests reach the tier that has their state.

What production deployments reveal about where the engineering actually breaks

Mooncake's FAST '25 Best Paper documents production use at Kimi, processing over 100 billion tokens daily across thousands of nodes, with its RDMA-based Transfer Engine enabling higher long-context throughput. SGLang's 96-H100 production deployment of DeepSeek-R1 used a 3-to-9 prefill-to-decode node ratio and achieved tens of thousands of input and output tokens per second per node, demonstrating that the ratio is a tunable outcome of transfer constraints.

MORI-IO demonstrated substantially higher goodput than colocated serving on the same 8-GPU MI300X node, showing that the benefit is not confined to multi-node deployments. DualPath shows that high cache hit rates still leave throughput constrained by storage ingress on the wrong node type. A faster compute fabric alone cannot fix that imbalance. The successful intervention changes where storage loads enter the system.

The engineering checklist for building a transfer layer that does not become the bottleneck

For practitioners, the lesson is to make these decisions together. Choose GPUDirect RDMA over CPU staging where transfer materially affects TTFT, and prefer scatter-gather-capable verbs for fragmented KV layouts. Use one-sided RDMA Write with Immediate to avoid waiting on a two-sided send/receive, and build arrival signaling into the handoff. Pre-register a fixed HBM pool and amortize rkey and remote-address negotiation across sessions or batches.

Transfer mode should follow the workload's latency objective. Default to push for latency-sensitive requests, and use pull where simpler orchestration justifies the added serial delay. Build the proxy to track where KV blocks are held so repeat requests can reuse state where it already is. That routing decision should precede paying for another transfer.

Fabric sizing must account for the worst-case payload at target context length and concurrency. InfiniBand HDR or NDR is the practical minimum identified for latency-sensitive long-context production workloads. Keep KV traffic apart from tensor-parallel communications with dedicated planes, traffic prioritization, or separate rails, and choose paths based on queue depth and how close the hardware is. Bandwidth planning must include storage ingress as well.

Finally, co-design tier placement metadata with RDMA path selection. Prefer in-HBM blocks for latency-critical requests, while using DRAM and NVMe for prefill reuse candidates with greater latency tolerance. Lower-tier capacity comes with a bandwidth cliff. A serving architecture delivers on disaggregation only when its scheduler accounts for that retrieval cost before committing scarce GPU time.

Sources

  1. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
  2. Serving Large Language Models on Huawei CloudMatrix384
  3. A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference

More in Distributed Serving