Networked NVMe Fabrics for Shared KV Cache in Multi-Node Serving Clusters
Shared NVMe fabric turns isolated GPU cache into a cluster-wide resource.

Networked NVMe fabrics improve LLM inference throughput by turning isolated KV cache capacity into a shared resource that prefill and decode workers can access across nodes. Their value depends on flash capacity together with transport, topology, namespace design, I/O scheduling, and coherence. For multi-node serving, effective tiering preserves reusable KV blocks without making their movement the bottleneck on the path to generation.
Why KV cache stops being a per-node problem at scale
Long contexts and concurrent users push KV state beyond a single node’s capacity. On an H100 serving a large long-context model, one user’s KV cache can consume tens of gigabytes, while eight concurrent users can require several times the device’s entire HBM. More VRAM does not resolve the structural pressure. The first response is a hierarchy of GPU HBM, CPU DRAM, and NVMe, with hot, warm, and cold blocks placed respectively. That arrangement still assumes local ownership.
Prefill-decode disaggregation breaks that assumption by separating the nodes that compute KV tensors from those that consume them. After each prefill, its cache must move to a decode worker before generation can begin, which can mean moving multiple gigabytes per request for a large model. The state must also survive eviction, support future lookups, and eventually be freed. These tasks go beyond GPU memory allocation and call for distributed storage.
What local NVMe and DRAM tiers cannot provide once requests span multiple nodes
Local tiering solves capacity without solving cross-node reuse. vLLM’s built-in prefix cache works within a worker, but each replica in a multi-replica deployment maintains its own cache. A request that reaches worker B thus gets no benefit from an identical prefix that worker A has already computed. The miss has a substantial latency cost. For a long context, loading cached KV blocks from NVMe is far faster than recomputing them, so lost reuse directly increases time to first token.
Agent workloads widen this sharing gap because their retained state extends beyond active generation. Fudan University’s UNISON research documents growing KV prefixes held across tool waits in many simultaneous sessions, producing a larger live set than multi-turn chat alone. Routing by prefix cannot guarantee reuse across spot restarts, node failures, or heterogeneous pool rebalancing. Worker affinity is therefore an incomplete answer. DRAM and local NVMe are still useful tiers, but a shared cache that works regardless of worker placement needs more than local capacity or a preferred route.
How NVMe-oF turns per-node flash into a cluster-wide KV cache substrate
NVMe-oF separates flash namespaces from the compute nodes that access them. Any connected inference worker can address the shared pool, so it can use storage beyond what is locally attached. Namespace assignment is independent of physical compute placement. Namespaces can be carved, assigned, and remapped into per-model or per-tenant pools without physical reconfiguration, extending the hierarchy from HBM through DRAM and local NVMe into shared network storage.
The transport makes that addressability available over a fabric, using RDMA or TCP. TCP supports commodity Ethernet deployments and can improve substantially on legacy NFS or iSCSI without requiring an RDMA rebuild. Western Digital’s OpenFlex Data24 NVMe-oF Storage Platform illustrates this disaggregated approach by extending NVMe flash over Ethernet and supporting independent storage and compute scaling. Once a prefill write is complete and available under the cache’s coherence contract, decode workers can read those blocks from the common namespace without a separate replication step.
Fabric topology choices and their effect on KV cache I/O latency
RDMA over RoCEv2 or InfiniBand is the preferred transport for latency-sensitive KV reads because it keeps the CPU off the data path and delivers bandwidth close to the physical medium. TCP trades that path for deployment on commodity Ethernet. Existing infrastructure constrains the practical choice. A transport decision based only on peak specifications misses whether the cluster can sustain concurrent transfers through its actual network and software stack.
Co-locating NVMe-oF targets with GPU nodes in the same rack reduces fabric latency but limits independent scaling, while dedicated storage racks enable more flexible expansion at the cost of an additional switch hop. Moonshot AI’s Mooncake uses otherwise underutilized CPU, DRAM, SSD, and RDMA resources in the GPU cluster to build a disaggregated KV cache. Separate storage racks are not its starting point.
All-to-all designs such as fat-tree or dragonfly differ from a two-tier spine-leaf arrangement that can concentrate east-west KV traffic at the spine under concurrency. Low-load latency does not expose that bottleneck. MLPerf Storage v3.0 results show file payload traffic over two independent 400 Gb/s NDR InfiniBand/RDMA links on both client and storage nodes, simulating 1,024 accelerators. That benchmark configuration illustrates the bandwidth envelope against which topology must be judged, rather than establishing that any particular KV workload will reach it.
I/O path latency in detail — where microseconds are lost between the GPU and remote NVMe
CPU staging, interrupt handling, and software scheduling account for much of the gap between local and networked NVMe, with less of the gap coming from flash itself. Naïve offload stalls GPU decoding while it waits on storage. Bandwidth specifications do not capture those lost cycles. NVIDIA’s Inference Context Memory Storage Platform, announced at CES 2026, uses BlueField-4 DPUs to manage KV movement off the GPU compute path, with NIXL as its KV block transport library. Tutti aims to cut the same kind of stalls by keeping the CPU out of the critical data and I/O control paths between HBM and SSDs.
GPU Direct Storage removes CPU staging by allowing direct transfers between NVMe and GPU memory over PCIe, with particular significance for KV spillover writes. Removing a copy is not sufficient by itself. Pipelining that works well for DRAM-backed caches can fragment SSD transfers, reduce effective bandwidth, and add CPU scheduling overhead. The NVMe tier consequently needs different I/O scheduling from the DRAM tier. UNISON proposes using agent tool-wait intervals as a DMA budget through a near-memory scheduler, moving blocks while the session is idle so that migration latency falls outside the generation path.
Block-level cache coherence across decode nodes sharing a common NVMe namespace
In a local system, the PagedAttention allocator controls the address space, but a shared block written by prefill node A may be read by decode nodes B and C while a garbage collector on node D attempts eviction. When nodes share reads, writes must finish before reads begin. Pinning must protect blocks that are in use, and reclamation must leave out any block that another node is still reading. FlexKV, developed by Tencent and NVIDIA, addresses distributed reuse at the cache layer, with Mooncake Transfer Engine integration in January 2026.
Content addressing simplifies writes but does not solve reclamation. With the same model weights, blocks keyed by token prefix hash make duplicate writes idempotent and allow deduplication. LMCache’s shared Redis tier uses content-addressed prefix keys and LRU eviction as a soft coherence contract, leaving individual evictions without active coordination. When a block has been held for many seconds and is near its eviction timeout, it needs either lease renewal or a TTL that covers worst-case generation time, and either approach can fail.
Where the real bottlenecks land under production concurrency
Production diagnosis should begin with the fabric, coherence coordination, and GPU-to-storage software path before assuming flash saturation. Concurrent reads can exhaust east-west bandwidth while individual NVMe devices still have headroom. Device headroom is an important diagnostic distinction. If transfer demand exceeds fabric capacity, topology and byte volume are the intervention points, while GPU stalls associated with CPU staging call for I/O-path changes. Coordination contention instead requires attention to the cache’s pinning and eviction contract.
UCLA and Columbia’s MESHKV research, arXiv 2609.19207, describes a tiled accelerator whose KV traffic funnels through a logical memory controller, consuming link cycles in back-pressure and reducing useful transfer. The spoke-hub failure pattern provides a cluster-level analogy for NVMe-oF. MLPerf Storage v3.0, published September 1, 2026, adds KV Cache and Vector Database workloads, giving practitioners workload categories beyond training and checkpointing for evaluating storage behavior.
Reducing bytes can be as important as expanding links. FP8 KV quantization halves cached-token bytes relative to BF16 and therefore halves I/O bandwidth demand per attention step. Disaggregation nevertheless has a break-even point because short prompts at small batch sizes can incur more transfer overhead than they recover through separation. Teams should benchmark their own prefix reuse and interconnect behavior before committing to full prefill-decode disaggregation. Mooncake’s Kimi K2 deployment provides a positive production anchor, achieving very high prefill and decode throughput with that architecture.
The memory-native fabric argument and what it means for NVMe-oF’s architectural ceiling
The strongest objection to NVMe-oF concerns its abstraction. Decode needs fine-grained, session-keyed, latency-critical access, while block storage was not designed around live KV state. Tens-of-microseconds access remains far slower than HBM. Kiyawat and Skadron’s HotInfra 2026 paper argues for KV cache as a first-class shared data structure served by a memory-native fabric, analogous in purpose to accelerator interconnects such as NVLink and UALink but realized through memory-to-memory connectivity.
The paper compares CXL 3.0 pooled-memory with 19 H100s doing the same job and reports 2.4× more bandwidth and 39.7× lower cost per million tokens. Those figures apply exclusively to that comparison, with NVMe-oF savings outside its scope. They give the memory-centric objection economic force as well as an access-pattern rationale. However, datacenter-scale CXL 3.0 fabrics are not available in production deployments today, while NVMe-oF with RDMA is deployable. In practice, networked flash therefore serves as the cold-to-warm tier, with memory-native warm storage still needed in the long run.
Engineering decisions that separate a functional shared KV fabric from one that saturates under load
For practitioners designing this tier, the first decision is to match transport to the installed fabric, using RDMA for latency-sensitive reads and TCP as a deployable fallback. Topology must then support east-west traffic without a spine or central controller becoming the choke point. Peak link rates are not enough here. The design should use fat-tree or equivalent non-blocking connectivity and put targets in the same failure domain as the GPU nodes they serve. Per-model or per-tenant namespaces should isolate eviction pressure, while content-addressed blocks simplify duplicate writes.
The I/O path and coherence protocol must fit the workload together. GPU Direct Storage or DPU offload removes CPU staging, while background migration during tool waits keeps avoidable transfers outside active generation. Prefix caches can use TTL-based eviction with content addressing. Long-running decode needs explicit pinning or lease renewal, with the choice based on the session-duration distribution. Quantization belongs in this storage design as well, since INT8 offers up to 2× memory reduction versus FP16 or 4× versus FP32, while sensitivity-aware mixed precision such as KVTuner, arXiv 2502.04420, targets near-lossless inference at lower I/O cost.
Mooncake, FlexKV, and SGLang’s EPD disaggregation using Mooncake’s RDMA engine share an architectural stance that treats KV storage as a system with its own needs for scheduling, coherence, and transfers. Capacity alone is not the design criterion. Storage is not a commodity problem for LLM inference, and a cluster scales or saturates depending on how deliberately this tier was designed.
Sources
- MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators
- UNISON: A Co-Designed Near-Memory Scheduler of Session KV Residency for LLM Agents
- Reimagining LLM Inference Infrastructure with Memory-Centric KV Cache Servers
- Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving


