Disaggregated Storage Backends Versus Monolithic GPU Memory the Infrastructure Stack Vendors Are Actually Shipping
How vendors tier KV cache across HBM, DRAM, and NVMe.

Long-context LLM serving requires a memory hierarchy that extends beyond GPU memory to retain sessions. For long contexts, extending into millions of tokens, the architecture places active KV blocks in HBM, reusable state in DRAM, and cold state on NVMe, with shared access across nodes. Shipping systems favor this disaggregation because GPU-bound storage turns inference capacity limits and checkpoint delays into paid GPU idle time.
Why GPU HBM became the bottleneck that forced disaggregation
GPU HBM is the scarcest and most expensive resource in an LLM cluster. Two workloads expose its limits in different ways. During inference, KV cache grows linearly with context length, batch size, model width, and layers, so long-context and agentic sessions can exceed VRAM before completion. During training, a large BF16 checkpoint written through CPU staging to NVMe takes several minutes per save, accumulating tens of GPU-hours of idle time across a week-long job.
SK Hynix’s ITME frames KV cache as long-lived inference state that persists across turns and sessions. The py-kvcache characterization shows that once the reusable working set exceeds VRAM, the system must evict useful blocks or move them elsewhere. GPUs remain billable during both recomputation and storage stalls, making data placement an economic decision as well as a capacity problem.
What the three-tier memory hierarchy actually looks like in production
HBM provides the highest bandwidth but the smallest capacity and highest cost per GB, holding model weights and active KV blocks during computation. CPU DRAM accommodates idle or recently evicted sessions, although transfers back to the GPU are PCIe-limited at around 40 GB/s. NVMe provides cheaper capacity for cold blocks and checkpoints. Its latency requires prefetching or overlap with compute, but py-kvcache finds that scheduler-aware disk loading can beat recomputation at long context lengths.
ITME identifies node-local SSDs as a source of fragmentation when requests migrate, motivating shared pools accessible over RDMA. Its proposed CXL-hybrid memory adds a byte-addressable, TB-scale layer between DRAM and NVMe, with predictive DMA prefetching for prefixes and weights that exceed host memory. ITME reports up to a 35.7% throughput improvement over conventional CPU-offloading. The tests use a programmable hardware prototype and a production SK Hynix memory module and have yet to cover an at-scale deployment, so CXL is still an emerging option that has yet to become an established fourth tier.
How Mooncake established disaggregated KV cache as a production pattern
Moonshot AI’s Mooncake makes the shared-store architecture concrete in Kimi’s serving platform. It separates prefill and decoding clusters, then pools CPU DRAM and SSDs across the GPU cluster as distributed KV storage. Mooncake describes the principle as “trading more storage for less computation”, because a cache hit avoids full prefill recomputation only if retrieving that state costs less than computing it again.
Prefill–decode disaggregation makes that handoff explicit and allows separate provisioning of the two phases. Mooncake is bounded by RDMA bandwidth of approximately 12.5–25 GB/s per NIC at 100–200 Gbps, with RDMA providing the latency needed to make remote DRAM viable. Mooncake Store also appears in SGLang’s HiCache and TensorRT-LLM. Those integrations show that the distributed KV layer can serve engines beyond the platform that introduced it.
Where prefill-side storage bandwidth becomes the new bottleneck in agentic workloads
In multi-turn agentic workloads, prefill engines repeatedly load large KV volumes while decode engines generate tokens and write back comparatively little state. Prefill storage NICs saturate while decode storage NICs remain underused. This imbalance comes from how the work is split, so more cache capacity alone will not fix it.
DualPath, from DeepSeek-AI, Peking University, and Tsinghua University, opens a second loading route through decode engines. Those engines receive cached state and forward it to prefill engines over the compute network using RDMA, while the design avoids interference with latency-critical model execution traffic. A global scheduler balances both paths dynamically. DualPath reports up to 1.87× higher offline inference throughput and an average of 1.96× higher online agent runs per second across three models under realistic agentic workloads. The result makes storage NIC provisioning and scheduling integral parts of the disaggregation design.
What the vLLM and SGLang connector ecosystem reveals about how teams actually wire storage backends
The connector ecosystem places an API boundary between inference engines and external cache tiers. LMCache extends prefix caching with persistent backends, supports vLLM, SGLang, and NVIDIA Dynamo, and integrates with Mooncake Store. The llm-d implementation works with POSIX filesystems including CephFS, Lustre, IBM Storage Scale, AWS EFS, and local NVMe. Connectors therefore let backends compete independently of complete engine integrations.
The API does not make transfer policy interchangeable. The py-kvcache implementation combines asynchronous direct I/O, bounded shared staging, and preloading while requests remain queued. At long contexts, it loads from disk faster than LMCache and within a few percent of native vLLM KV Offload. Its H100 results also show that average requests fall below the external-cache break-even point when GPU memory holds enough prefixes. NVIDIA NIXL, announced March 2026, sits below this connector layer, unifying RDMA and GPUDirect movement across GPU, CPU, NVMe, and cloud object stores, with integrations into vLLM, SGLang, and Ray.
Why GPUDirect Storage and io_uring are not interchangeable — and which phase each belongs to
GPUDirect Storage and io_uring address different I/O patterns. GDS removes CPU DRAM staging from large sequential transfers by using a direct GPU DMA path to NVMe. The supplied H100 SXM5 checkpoint comparison reports an 8.6× speedup over CPU-staged checkpointing. This path is a good fit for pre-training and checkpoint writes. By contrast, fine-grained, frequent KV reads favor io_uring, whose asynchronous submissions reduce system call overhead and provide competitive IOPS with lower latency than GDS on NVMe devices.
Remote storage adds a separate transport requirement. NVMe-oF with the direct GDS DMA path requires InfiniBand or RoCE and GPUDirect RDMA-capable NICs. NVMe-oF still works over standard TCP/Ethernet, but GDS falls back to CPU-buffered compatibility mode. At 400GbE and above, TCP interrupt handling and user-space packet copies can consume host cycles needed to feed GPUs. The ROS2 SmartNIC research at SC Workshops ’25 found that BlueField-3 SmartNIC RDMA matched host RDMA performance while SmartNIC TCP was markedly lower, distinguishing protocol choice from offload location. NVIDIA’s CMX Context Memory Storage Platform, powered by a data processing unit, applies RDMA to a cluster-shared KV pool for multi-node context reuse.
How MoE disaggregation adds a second dimension — separating attention workers from FFN workers
Mixture-of-Experts deployments introduce a compute split that changes storage requirements. Attention depends on memory capacity and bandwidth, while routed FFN experts depend on batch size and FLOPs. In collocated deployments, expert weights compete with KV cache for GPU memory. Attention–FFN Disaggregation moves those weights into a dedicated FFN pool, deliberately freeing attention workers to retain more cached state.
Tsinghua University and Meituan’s AFD-Ledger, published in August 2026, models hardware assignment under a fixed budget and time-per-output-token service-level objective. It finds that AFD improves throughput in only a minority of studied homogeneous-hardware settings, while heterogeneous deployments require hardware chosen for complementarity. FFN utilization or KV capacity alone cannot predict the outcome. Storage sizing consequently depends on the joint attention and FFN provisioning model, including the number of attention workers available to hold the working set.
What routing KV cache across models and providers requires from the storage layer
When routing sends a request to a model other than the one that produced its KV state, that state can become incompatible and force a full prefill. The Unified AI Gateway architecture frames this as cache-induced model lock-in. Storage savings stay tied to the original model unless the system can manage reuse across models.
The gateway requires four capability groups, including cross-granularity reuse across prefixes, semantic matches, and non-prefix state, adaptive compression, cross-model KV mapping with fidelity constraints, and distributed storage and transfer. These capabilities need more than a common filesystem. In the analytical simulation across eight workload profiles, time-to-first-token speedups range from modest to more than 13×, and input-cost reductions are more than 6×. The gains depend on switching probability and workload type, so these simulation results do not show that all production systems will see the same gains. The engines that take part still need a shared backend in place of isolated local caches.
What lossless BF16 compression changes about storage bandwidth requirements
Lossless compression reduces the bytes crossing every boundary without changing numerical fidelity. GAAC, published in 2026, encodes BF16 KV exponents with short codes and reports roughly 1.4× compression across five tested models. On a 15 GB/s RDMA link, that corresponds to roughly 21 GB/s of effective KV throughput. The physical link has not become faster. Each transfer simply represents more uncompressed state.
Brevis, published in 2026, reduces checkpoint data by roughly a third across ten public checkpoints spanning language, audio, and image generation, shortening the write window and reducing NVMe wear. The open-source lmz implementation, released in 2026, preserves checkpoints byte for byte and decodes on the GPU well above PCIe link speed. Its decoder therefore does not become the transfer bottleneck. Across the hierarchy, lossless compression reduces checkpoint size, NVMe write time, RDMA transfer time, and cold-tier capacity requirements while preserving the quality constraint that motivates exact KV storage.
What MLPerf Storage v3.0 results reveal about which backends can actually feed a GPU cluster
MLPerf Storage v3.0, with results announced September 1, 2026, adds KV cache and vector database workloads to training and checkpointing tests. It also adds an S3 data layer beside POSIX. Roughly one-sixth of submissions used S3 access, showing that object storage has a growing role in training measurements as well as archival use. First-time submitters included Azure, Nebius, NVIDIA, XSKY, and ZettaLane. On-premises checkpoint write results vary widely, with the top result far above the median, so differences between backends can be measured as well as claimed on the basis of their architecture.
Earlier tests showed why checkpointing should be part of buying decisions. Volumez’s MLPerf Storage v2.0 result in October 2024 provided an AWS throughput and GPU utilization benchmark. Version 2.0 also added checkpoint tests to show the trade-offs between failure-recovery and save-frequency across a cluster. Its working group explicitly warned against storage becoming an unacceptable checkpointing bottleneck. Version 3.0 extends that scrutiny to the shared KV tier itself, allowing assessment of disaggregated inference storage rather than relying on training throughput as a proxy.
How to choose a storage backend given the workload phase, cluster scale, and hardware available
For single-node or small-cluster inference without RDMA, local NVMe with io_uring and scheduler-aware preloading fits the py-kvcache pattern. External caching still needs an admission decision based on GPU speed and prefix hit rate. For multi-node inference with RDMA, the Mooncake pattern combines shared DRAM for hot KV with NVMe for cold state, while DualPath-style scheduling addresses prefill NIC saturation. MoE deployments should model attention and FFN provisioning before sizing that store.
Large sequential checkpoint writes favor GDS, with lossless BF16 compression reducing the volume written. Multi-model deployments additionally require shared access and coordination across engines, while S3-compatible storage over TCP suits checkpoint archives and cold dataset retrieval rather than serving as the primary live-inference or hot-checkpoint path at scale. Backends should be compared against the phase and fabric they will actually serve. Public, comparable MLPerf Storage results and reproducible workload benchmarks provide a broader basis for that choice than a single advertised bandwidth figure.
Sources
- AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation
- ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories
- DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
- Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
- Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management


