Disaggregated KV Cache Architectures Across GPU Memory and NVMe Tiers
Disaggregated KV cache tiers trade memory bandwidth for storage capacity strategically.

Disaggregated KV cache architectures across GPU memory and NVMe tiers improve LLM inference throughput when active blocks stay in HBM, reusable blocks sit in DRAM or CXL memory, and cold blocks reach NVMe without delaying decode. NVMe adds capacity, but generation still needs memory that can keep up. Its effect on latency and cost per token depends on reuse, prefetching, and the cost of moving blocks between tiers.
Why KV cache has outgrown the GPU and what disaggregation actually means
Autoregressive decoding repeatedly reads the keys and values of preceding tokens. Discarding those tensors without retaining an offloaded copy forces recomputation of the preceding context, bringing quadratic attention work back into the generation path. Offloading preserves that work but introduces a retrieval cost. The capacity problem remains unavoidable for large-scale models at long context, where a single user’s cache can exceed what many accelerators hold. Serving even a handful of concurrent users can therefore become physically impossible within GPU HBM alone.
Long contexts also outlive individual generation steps. Agentic and multi-turn workflows need state across turns and sessions, so their cache may still be needed after a request ends. Keeping that state competes with the memory needed for active users. Adding HBM means adding GPUs, tying cache expansion to accelerator purchases even when storage capacity is the immediate constraint.
Local offload initially moved cache from HBM to CPU DRAM, then to local NVMe. Both steps expanded capacity without changing ownership. The cache still belonged to one inference worker. Disaggregation makes KV cache shared system state that the whole serving fleet can see, so workers beyond the one that made it can place and reuse it.
The six-tier memory hierarchy that disaggregated KV cache must navigate
GPU HBM provides the highest bandwidth and sub-microsecond access, but its capacity is constrained and expensive to expand. CPU DRAM offers substantially lower bandwidth and mid-range latency. It has more local room for reusable blocks. Marvell’s photonic-CXL paper describes server DRAM limits that leave room for only minutes of KV state in a busy cluster, showing why host memory alone cannot hold enough KV state.
CXL-attached pools extend byte-addressable memory across a rack, with latency well below NVMe. ITME shows how CXL-hybrid memory can add TB-scale capacity while keeping direct memory addressing. Electrical CXL has deployment limits, however. The Marvell paper shows how switch-induced latency per hop, limits on cable reach, and power scaling make TB-scale electrical deployments hard to put into practice.
NVMe is a lower storage tier for historical prefixes and completed sessions, but retrieving them costs much more than reading them from memory. RDMA fabric connects placement across nodes through high-bandwidth transfers whose latency varies with topology. Parallel filesystems round out the hierarchy with the most capacity and the slowest access, which suits offline or archival use but is too slow for active generation.
Sanjeev Rao Ganjihal’s predictive placement preprint maps these six tiers explicitly. Its hierarchy includes both the places that store blocks and the fabric that connects them, so access cost depends on more than where a block is stored. The path to where it is stored must also be known. Both the place and the path matter when remote memory and local storage compete to serve the same request.
How the hot-warm-cold classification maps cache blocks to tiers
Predictability connects workload behavior to tier placement. Hot blocks are actively referenced during generation and must remain in HBM, because eviction forces a stall on the decode path. Warm blocks belong in DRAM or CXL-range storage when likely reuse justifies fast restoration without dominating time-to-first-token, or TTFT. Cold blocks can reside on NVMe when they are candidates for prefetching ahead of decode. Temperature describes expected use, which can change over time.
Misclassification wastes either capacity or time. Keeping cold blocks in HBM displaces active work, while pushing warm blocks to NVMe delays reuse. Agentic workflows make this distinction especially important because tool-calling sequences and system prompts can recur regularly despite gaps between accesses. Standard least-recently-used eviction can discard those blocks as cold, forcing recomputation when the workflow returns.
Mooncake’s Conductor scheduler implements placement-aware dispatch, while Mooncake Store supports LRU, LFU, and request-characteristic-based eviction. Ganjihal’s preprint proposes a Bayesian reuse predictor over block-type and transition-type pairs, treating placement as a forecast made before memory is exhausted.
ITME exploits deterministic access patterns in model weights and long-context prefix caches to prefetch from NVMe into DRAM before decode needs the data. The timing is the engineering advantage. A slower tier can contribute useful capacity when the system predicts demand early enough, but the same tier becomes a bottleneck when retrieval starts only after generation waits for a block.
The prefill–decode split and why it puts KV transfer on the critical path
Prefill–decode disaggregation widens placement from a local memory decision into a distributed systems problem. Prefill performs compute-bound dense matrix operations over the input prompt and produces KV tensors. Decode reads that cache over and over through operations that are memory-bandwidth-bound. Separating these phases permits independent scaling for bursty prompt ingestion and concurrent streaming, but it also creates a mandatory handoff between workers.
The handoff is now part of serving latency. vLLM, SGLang, TensorRT-LLM, LMDeploy, and NVIDIA Dynamo support prefill–decode disaggregation, and providers including DeepSeek and Gemini deploy it at production scale. Specialization therefore relocates part of the cache constraint onto the connection between the workers.
Mooncake, developed by Moonshot AI / Kimi and awarded Best Paper at FAST 2025, pools CPU, DRAM, SSD, and RDMA resources into shared KV infrastructure. Its Conductor scheduler manages placement across the fleet. Early disaggregated designs route tensors over InfiniBand or RoCE through a CPU-orchestrated, six-hop bounce-buffer path. Mooncake’s GPUDirect RDMA Messenger instead moves blocks directly between GPU HBM and CPU DRAM across nodes without CPU bounce copies.
Long context magnifies the cost of that movement. The Marvell paper describes handoff costs ranging from a sub-millisecond RDMA operation to a multi-second NVMe read, depending on where the blocks reside. The tier occupied at handoff therefore determines which transport costs enter TTFT.
What moves data across tier boundaries and why the transport mechanism is not interchangeable
HBM-to-DRAM movement within a node is governed by PCIe bandwidth. CPU-staged copies add overhead and latency variability, while KV movement can also consume GPU streaming multiprocessor cycles. NVIDIA ICMSP, announced at CES 2026, uses BlueField-4 DPUs to move data management off the GPU compute path. The boundary needs more than available memory. It needs a copy path that does not compete unnecessarily with inference.
At the DRAM-to-NVMe boundary, retrieval adds storage access to the memory path, making bandwidth and staging costs part of placement. GPU-direct paths can avoid host staging. ICMSP uses cuFile for GPU-direct NVMe transfers, while its NVIDIA Inference Xfer Library, or NIXL, supports UCX for RDMA, RoCE, and TCP, GPUDirect Storage for NVMe, and object storage backends. Tutti’s 2026 research uses P2P DMA to eliminate CPU intervention from HBM-to-NVMe control and data paths and integrates with Mooncake for globally discoverable replica metadata.
Across nodes, GPUDirect RDMA transfers tensors between GPU memories without host CPU involvement. CXL offers a different access model. TraCT and Beluga demonstrate byte-addressable shared memory as a KV transfer substrate and rack-wide prefix cache, avoiding explicit message-passing and RDMA network-stack overhead.
Electrical CXL’s reach, switch latency, and power limits constrain that alternative. Marvell’s Photonic Fabric Memory Appliance replaces electrical switches with a passive fiber shuffle in a switch-free full-crossbar topology. Its emulation results show over 50% latency reduction versus electrical CXL pools. That result measures latency reduction on the memory access path alone.
Lossless compression as a first-class strategy at the inter-node tier boundary
RDMA-bound transfer depends on both link bandwidth and the encoding of the tensors crossing it. BF16 stores more exponent bits than mantissa bits, and KV tensor exponents often cluster around a small set of values across the token dimension. Lossless compression therefore operates as a transport strategy, reducing the bytes sent while preserving the cached state exactly.
SplitZip’s 2026 design encodes frequent BF16 exponent values with fixed 4-bit codes and places rare values in a small escape buffer. The layout permits GPU-parallel decoding and bit-exact reconstruction. SplitZip reports 613.3 GB/s compression throughput on BF16 tensors and up to a 1.32× end-to-end speedup. Its compression throughput exceeds the bandwidth of the RDMA links it serves, making the encoding stage a net accelerator at that boundary.
The distinction from lossy quantization is important. Lossless encoding adds no model-quality loss and can compose with a pipeline that also uses quantization, whereas information removed by lossy KV quantization cannot be restored during retrieval. The BF16 result remains specific to its format-aware mechanism. It is not a performance guarantee for every representation or link.
ZipCCL’s 2026 work demonstrates the broader principle in training, where lossless compression of near-Gaussian collective tensors accelerates communication and end-to-end execution on 64-GPU clusters. For inference, the relevant design question is narrower. Compression is useful when its processing cost is outweighed by the transfer time it removes.
When disaggregation hurts: workload shapes that break the standard tier assignment
Disaggregation introduces scheduling, transfer, and metadata overhead before it delivers any reuse benefit. Short, concentrated requests with little shared context may not create a useful warm-block population. For short prompts, transferring KV blocks between nodes can cost more than recomputation. Extra coordination then adds latency without recovering enough capacity to justify the architecture.
The Unified AI Gateway paper describes cases where switching the target model makes the KV cache built up so far unavailable or incompatible, forcing full-context prefill. Optimizing either independently risks cache-induced model lock-in or a handoff that discards the value of retained state.
NVMe is particularly unforgiving on latency-sensitive decode paths. The Marvell paper notes that loading blocks from NVMe can take longer than recomputing short sequences. Cold-tier placement is viable when prefetching keeps retrieval away from live decode, when recomputation is infeasible because context is too long, or when the workload tolerates retrieval latency. Capacity alone does not justify the read.
The practical recommendation is consequently workload-conditional. Low-concurrency, short-context, single-turn workloads should use simpler local tiering or no offloading. With high prefix reuse, long contexts, high concurrency, and multi-turn state, shared cache pays off and blocks need to be placed in the right tiers.
Designing a tier-aware KV cache system: the placement decisions engineers must make explicitly
Engineers should explicitly coordinate block classification, eviction policy, transport selection, compression strategy, and scheduling topology. Classification should use measured reuse probability to set boundaries between HBM, DRAM or CXL, and NVMe. Agentic workloads need session-affinity routing and prefix-aware classification, while conversational workloads need recency-weighted warm-tier sizing. Reactive LRU is only a baseline. Structured access patterns warrant predictive placement rather than eviction driven solely by memory pressure.
GPUDirect RDMA fits node-to-node HBM handoffs, P2P DMA provides the Tutti pattern for intra-node HBM-to-NVMe movement, and CXL offers byte-addressable access where rack-scale shared memory and managed switch latency are available. Format-aware lossless compression should precede RDMA transmission where the SplitZip pattern applies. Scheduling should then choose among monolithic, two-pool-routed, and fully disaggregated topologies based on workload, with routing and KV placement coordinated.
Provisioning must reflect both the resulting traffic and stored bytes. A meaningful warm tier can require terabytes of DRAM per node, while NVMe selection must account for write endurance and bandwidth because KV traffic is write-heavy and continuous. Cost per token thus depends on how much concurrency and reuse the kept state allows without adding critical-path stalls. No capacity figure answers that alone.
Production operators serving Llama, Qwen, or Mixtral face these choices directly. As reuse patterns and concurrency change, classification and placement remain ongoing engineering work. Storage is not a solved or commoditized layer for these workloads, because scaling the inference system above it depends on its access paths and placement decisions.
Sources
- Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management
- A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference
- ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories
- Predictive Multi-Tier Memory Management for KV Cachein Large-Scale GPU Inference
- Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving


