Distributed KV Cache Coordination Across GPU Node Boundaries in Multi-Host Inference Clusters

Cross-node KV reuse hinges on routing, metadata, and fast transfer.

· 8 min read
Cover illustration for “Distributed KV Cache Coordination Across GPU Node Boundaries in Multi-Host Inference Clusters”
Long-Context Serving · November 10, 2026 · 8 min read · 1,882 words

Managing KV cache storage for production LLM serving requires more than freeing GPU memory. Across GPU nodes, reuse depends on prefix-aware routing, sufficiently fresh metadata, and transfers that cost less than recomputation. A tiered architecture spanning GPU HBM, CPU DRAM, NVMe, and shared object storage makes capacity available, but reducing serving latency and wasted GPU work with that capacity requires distributed coordination.

Why KV cache reuse fails at the boundary between GPU nodes

On a single node, an inference worker can look up a prompt prefix and reuse the KV state already held in its memory. Towards Distributed Inference of LLMs on a P2P Network describes this as “largely a local data-structure and memory-management problem”. That mental model breaks when replicas are on different nodes. A miss on the receiving worker may mean only that another worker has the computed prefix, where the receiving worker cannot see it.

The memory hierarchy makes that separation expensive. HBM provides orders-of-magnitude more bandwidth than the CPU–GPU PCIe link, so reloading cached state can take longer than computing the new tokens. SwiftCache shows PCIe loading dominating total latency even at modest context lengths. Its comparison of NVLink at 400 GB/s with PCIe at 64 GB/s shows why sharing GPU memory within a server can cut transfer overhead sharply. That intra-node solution needs another mechanism to work across node boundaries.

So a routing decision made without information about the cache can force a full prefill even when the state it needs is already on another node. GPU cycles go into doing the same work again, time-to-first-token rises, and throughput falls. More memory alone cannot resolve that disconnect. The boundary creates three linked failures, namely routing blindness, metadata fragmentation, and transfer cost.

How prefill–decode disaggregation restructures the problem

Prefill–decode disaggregation deliberately separates the workers that create KV state from those that consume it. Moonshot AI’s Mooncake places prefill and decode on separate node pools, making KV transfer part of the architecture. Every completed prefill must ship its cache to a decode node. Coordination is therefore part of the request’s critical path.

Mooncake also makes cache capacity a resource for the whole cluster. Its disaggregated KV pool uses otherwise underutilized CPU, DRAM, and SSD resources, extending placement beyond GPU HBM into a tiered hierarchy that includes CPU DRAM and NVMe. This changes what efficient memory management means. Efficient memory management must account for both the space a GPU has for another context and where that context can reside while remaining economical to retrieve. Placement and transfer become inseparable decisions.

The size of the handoff explains the pressure. A large model in BF16 creates hundreds of KiB of KV state for each token, so a multi-thousand-token prompt can leave several gigabytes waiting to reach the decode node. Decoding must wait for the state it needs to arrive. Disaggregation does not remove the coordination problem that node boundaries create, and it can make KV transfer latency the principal determinant of time-to-first-token. Routing, metadata, and transport are therefore dependencies of the architecture.

Prefix-aware routing as the first line of coordination

A prefix-aware router directs requests toward workers likely to hold reusable state while preserving load balance. These goals can conflict. Sending every request with a popular prefix to the same worker can create a hotspot, while spreading those requests evenly can discard the benefit of prefix reuse. The router must also act on metadata that may be stale or incomplete.

The P2P paper uses a decentralized design in which each node keeps a local radix tree and approximate read-only replicas of its peers’ trees. Periodic anti-entropy refreshes those estimates, and requests go to the node with the longest estimated prefix match. This scheme moves requests while keeping KV state in place. Its evaluation on simulated MMLU workloads finds latency benefits under low communication delay and skewed prefix distributions, but high network latency and affinity-induced hotspots limit those gains.

A centralized router can cut stale-view errors by keeping a view of almost all caches, but coordination can become a bottleneck and partial failures can put availability at risk. LMCache’s MP Coordinator separates those responsibilities more narrowly. It holds the control-plane metadata needed to discover and manage reusable caches while leaving KV data movement outside the coordinator.

The Unified AI Gateway paper extends the same problem to heterogeneous models and providers. A gateway must choose the target model, execution site, and cache action together because a switch that makes the state built up so far invalid can wipe out the cost or latency gains it was expected to bring. In each design, routing accuracy depends on the metadata available at decision time.

Metadata consistency: how much agreement the routing layer actually needs

Cache metadata requires performance consistency. If stale metadata sends a request to a worker that has already evicted the prefix, that worker can recompute the prefill and still produce a correct output. The optimization fails while inference remains correct. This allows eventual consistency without distributed agreement before each routing decision.

The P2P paper formalizes cache discovery as an “eventually consistent metadata-replication problem”. Peer trie estimates converge through periodic anti-entropy, while incoming requests use only the receiving node’s local view. Overlong refresh intervals under high churn or skewed prefix distributions increase the chance of a wasted round trip followed by recomputation. A global coordinator reduces that uncertainty but concentrates control-plane work, which explains the value of keeping bulk KV movement peer-to-peer.

Freshness is not the only requirement for useful metadata. UNISON shows that agent sessions retain growing prefixes across tool waits, while ordinary recency and timeout policies can classify those live waits as cold state. Even an up-to-date record of the last access falls short if the eviction policy has no information about when the loop is expected to return. The paper’s awareness-based policy cuts time-to-first-token by up to 89% on long-horizon agent traces.

RDMA and GPUDirect Storage as the transfer layer underneath coordination

The physical layer faces a simple test. Transfer must beat recomputation. A router may correctly locate several gigabytes of reusable state, yet retrieving that state over a slow path can cost more than regenerating it on the GPU. Logical availability is not enough to make a cache hit useful.

RDMA over InfiniBand or RoCE bypasses the CPU on both ends and makes large KV transfers competitive with prefill recomputation. NVMe-oF with GPUDirect RDMA extends that path to remote storage pools at near-local speeds. Standard 100GbE Ethernet without RDMA cannot use this path. NVIDIA Magnum IO GPUDirect Storage provides direct movement between storage and GPU memory across local and remote backends, avoiding CPU involvement.

NVIDIA cuObject, shipped January 2026, provides a public API for the RDMA fast path between S3-compatible object storage and GPU memory. NVIDIA NIXL, announced March 2026, unifies movement across GPU, CPU, NVMe, and cloud object stores, with integrations into vLLM, SGLang, and Ray. NVIDIA ICMSP, announced at CES 2026, uses BlueField-4 DPUs to manage KV movement off the GPU compute path.

Within a server, SwiftCache provides the complementary evidence. Sharing idle GPU memory over NVLink reduces P99 TTFT by up to 69% and extends maximum context length by up to 3.98× compared with vLLM and SGLang. NVLink addresses intra-node movement, while RDMA addresses the node boundary. With either path, transfer still has to beat recomputation for the request.

Lossless compression during transfer and why it matters for output fidelity

Compression can reduce transfer time, but unchanged KV state requires a lossless method. A lossy representation changes the model’s effective computation. That distinction defines the fidelity constraint. When the requirement is to preserve the original KV tensors exactly, bandwidth savings cannot come from discarding information.

SplitZip (2026) shows that this constraint is tractable in prefill–decode disaggregated serving. Its GPU-friendly design supports GPU-parallel decoding, uses authentic BF16 KV-cache activations in all main experiments, and reconstructs the original cache bit-exactly. It exploits otherwise underutilized GPU compute during transfer for compression and decompression.

FP8 quantization presents the strongest alternative argument because it already halves KV-cache size relative to BF16. Quantization-first approaches can capture much of the bandwidth gain while also reducing compute, leaving less opportunity for subsequent lossless compression. The two approaches nevertheless serve different requirements. Quantization changes model arithmetic and the output distribution, whereas lossless transfer preserves the representation supplied to it.

Object storage as a shared KV pool: what MLPerf Storage v3.0 established

S3-compatible object storage gives nodes a shared pool they can all reach without first agreeing on placement. The storage can grow without adding compute, and it runs on commodity non-GPU hardware. Its suitability for runtime KV reuse, however, requires workload evidence. General storage throughput does not by itself establish inference performance.

MLCommons released MLPerf Storage v3.0 results on September 1, 2026, with new KV Cache and Vector Database tests and support for an S3 access layer alongside filesystem-based access. The KV Cache test simulates multi-turn conversations that write a context once and read it repeatedly, using transfers from 64 MiB to about 3 GiB. This supplies a public baseline for comparing runtime cache storage across access types. MLPerf Storage working group chair Curtis Anderson stated, “As the scale of AI contexts reaches into the trillions, we expect object-based storage systems to emerge as a viable — and possibly preferred — alternative to filesystem-based storage.”

Nebius ran all four Closed division v3.0 results on its Object Storage with the Enhanced storage class through S3, feeding more simulated accelerators than any other result that round. S3-layer throughput trails parallel file systems, and request overhead remains significant for small, latency-sensitive accesses.

ObjectCache (2026) addresses that gap through RDMA transport, aggregation, layerwise delivery, and bandwidth-aware scheduling. Object storage works better for large sequential reads of contexts already written than for small accesses. The benchmark makes that fit measurable, while the universal S3 API lets nodes reach a shared pool through a configuration change alone.

How these mechanisms compose in production multimodal and multi-role systems

Multimodal serving turns cross-request reuse into cross-role coordination. One component can produce state that another consumes, across nodes with different execution roles. A cache location is therefore only part of the picture. Placement, compatibility, transfer, and lifetime must stay in step as work moves between components.

Omni-Flow, from Meituan and Peking University and targeting EuroSys’27, addresses this through its Data Flow and Compute Flow abstractions. Data Flow manages the placement, transfer, and lifetime of shared tensors and paged KV caches across roles. Compute Flow matches multimodal conversation histories and enables cross-role prefix-KV reuse between compatible LLM and diffusion components. Its Qwen3-Omni experiments show that server-side state management in a 50-turn session reduces mean job completion time by 19.3% relative to SGLang-Omni, with a smaller improvement relative to vLLM-Omni.

The gateway paper exposes the same dependency when a session switches models. Reuse, mapping, transfer, and recomputation must be decided together under quality, latency, cost, and resource constraints. Prefix-aware routing cannot make those decisions effectively without metadata, and metadata cannot make reuse economical without an adequate transfer path. Offload-backed storage supplies shared capacity beyond the memory of an individual worker.

These mechanisms are complementary layers. A shared S3-compatible pool decouples state production from consumption without requiring every role to share a memory space or agree on placement in advance. Storage supports coordination when it lets roles scheduled on their own reuse computation across node boundaries.

Sources

  1. Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference
  2. Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management
  3. UNISON: A Co-Designed Near-Memory Scheduler of Session KV Residency for LLM Agents
  4. SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing
  5. Towards Distributed Inference of LLMs on a P2P Network
  6. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

More in Long-Context Serving