KV Cache Storage Tiering Across DRAM HBM and SSD in Million-Token Serving Pipelines

Tiering KV cache across storage requires careful I/O engineering to beat recomputation costs.

· 8 min read
Cover illustration for “KV Cache Storage Tiering Across DRAM HBM and SSD in Million-Token Serving Pipelines”
Long-Context Serving · October 31, 2026 · 8 min read · 1,894 words

Million-token LLM serving requires an engineered KV cache hierarchy across GPU HBM, CPU DRAM, and NVMe SSD. Its feasibility depends on placement policy, I/O scheduling, and data format working together within each tier’s bandwidth limits. This is the core infrastructure decision for long-context serving, because adding storage capacity without controlling data movement can make cache reuse slower than recomputation.

Why million-token context windows turn KV cache into a storage sizing problem

KV cache keeps key and value vectors for each token it retains across every model layer. Its size grows with layer count, KV heads, head dimension, sequence length, and batch size. These factors compound as serving concurrency rises. At million-token lengths, a single user running a large frontier model can need several times as much HBM as a high-end accelerator has. Even with shorter contexts, a modest multi-user batch can use more HBM than the accelerator has.

Host memory extends retention, but not indefinitely. For long-running conversations and agentic workloads, Tutti reports that even 2 TB of CPU DRAM holds only around five minutes of KV cache at scale, while commercial servers with over 100 TB of NVMe capacity can hold more than one hour. At this scale, the question becomes how to engineer the tiers.

The bandwidth physics at each tier boundary that governs what tiering can actually deliver

HBM, DRAM, and NVMe have order-of-magnitude bandwidth gaps that constrain how quickly stored context can become useful computation. Capacity does not erase those transfer costs. TokenStack characterizes autoregressive attention as bandwidth-limited at O(1) arithmetic intensity because each new token rereads the accumulated KV context.

The PCIe boundary also affects cached-prefix prefill. In document Q&A with a large cached prefix, restoring KV state can leave the GPU idle while memory transfers dominate execution. A cache hit saves computation, but the data still has to move, and the cost depends on the path between storage and HBM. The relevant comparison is the time to restore usable state against the time to recompute it. SSD reuse can win that comparison only when the I/O path is managed correctly.

Why naive SSD offload causes GPU stall rates that make tiering slower than recomputation

PagedAttention creates a mismatch between efficient GPU allocation and efficient SSD access. Engines such as vLLM and SGLang allocate KV cache in small, fixed-size pages scattered across GPU memory, reducing internal fragmentation. Offloading that layout can lead to huge numbers of tiny random I/Os. The bottleneck then extends beyond the SSD itself. The CPU has to start each operation, even when GPU Direct Storage removes CPU staging from the data path.

In the tested configurations described in the notes, GPU stalls consume the large majority of inference latency, and GPU Direct Storage leaves substantial bubble time. SSD-backed reuse consequently becomes slower than recomputation. Pipelining does not necessarily rescue this design. Although overlapping transfers and computation works well for DRAM-backed caches, applying it to SSDs fragments transfers, reduces effective bandwidth, and adds CPU scheduling overhead. The failure follows from paged layouts meeting CPU-centric I/O initiation, so configuration tuning alone cannot resolve it.

GPU-centric I/O and placement intelligence as the two architectural levers that fix the stall problem

Tutti addresses the control path with a GPU-native KV object store. The CPU asynchronously loads I/O kernels once per layer, after which the GPU manages I/O control and completion. Its mechanisms include bulk object transfers, GPU io_uring, and slack-aware scheduling that limits GPU resource contention. This changes who schedules storage work. With 256 GB of pinned host DRAM and 14 TB of SSD volume per GPU, its evaluation approaches DRAM-backed LMCache performance while reducing time to first token and improving achievable request rate against GDS-enabled SSD-backed LMCache.

Placement addresses a different source of waste. TokenStack finds that a small minority of blocks account for a large majority of reuse events, but the distribution differs between short-form API traffic and long-form reasoning. Hot-tier admission must reflect that workload-specific skew. LMCache provides a deployable persistence layer across sessions, restarts, and server instances, supporting vLLM and SGLang with backends including CPU memory, Redis, disk, and S3. Persistence gives blocks more chances to be reused, but placement decides which blocks go in fast storage, and the I/O architecture controls whether promotions arrive in time.

How quantization changes the byte count at every tier boundary and becomes a storage architecture decision

FP8 KV cache uses half the bytes of BF16, providing 2× effective capacity across the hierarchy. The bytes are saved before any transfer is scheduled. That changes how many concurrent users each tier can hold state for and how much data must cross PCIe or reach the SSD. Teams therefore need to choose the format before sizing hardware, instead of leaving it to a final model-serving adjustment.

TurboQuant reduces throughput and increases per-token latency in normal serving, yet prevents memory saturation and sharply reduces time to first token under memory-constrained bursts. D-Quant introduces driftable entropy coding in 2026, while SplitZip targets lossless compression for disaggregated serving. These research directions extend the format choices, but smaller payloads do not by themselves remove the per-operation scheduling overhead caused by fragmented I/O.

How disaggregated prefill-decode architectures move the tier boundary problem from within a node to across a network

Mooncake, awarded Best Paper at FAST 2025, separates prefill and decode clusters around a shared KV cache pool. It draws on CPU, DRAM, SSD, and NIC resources, using its RDMA-based Messenger component and one-sided READ and WRITE operations for block transfers. In July 2025, it powered the Kimi K2 deployment with prefill-decode disaggregation and large-scale expert parallelism. Remote GPU HBM becomes another placement destination whose accessibility depends on network topology.

Mooncake remains bounded by per-NIC transfer rate, making large KV transfers a significant potential bottleneck even on a high-bandwidth network. Its transfer engine also entered vLLM v1 as a KV Connector, while SGLang introduced Encode-Prefill-Decode Disaggregation with Mooncake in December 2025. Those integrations broaden its use while leaving its workload suitability unchanged. For small short-request batches, network transfer overhead can exceed the benefit of separating prefill and decode.

What hardware-managed tiering with DPU offload changes about where I/O scheduling pressure goes

NVIDIA’s Inference Context Memory Storage Platform, announced at CES 2026 and later rebranded CMX at GTC 2026, puts dedicated hardware in charge of tier movement. BlueField-4 DPUs handle placement, eviction, I/O completion, and transfers across HBM, DRAM, and NVMe without keeping those tasks on the host CPU. The host critical path becomes less crowded. The design converges with Tutti on removing CPU-mediated I/O scheduling, assigning that work to a DPU where Tutti uses the GPU.

NVIDIA Inference Transfer Library supplies the platform’s KV transport, supporting NVLink, InfiniBand RDMA, NVMe-oF, and a TCP fallback. NVIDIA rates its STX reference design for up to 5× token throughput and 5× energy efficiency versus traditional storage, with results dependent on the workload. Hardware acceleration needs BlueField-4 in the server, requiring a hardware purchase. Offload shifts the scheduling work, but placement intelligence and bandwidth limits still control whether transfers keep up with demand.

Why High-Bandwidth Flash as a drop-in SSD replacement makes tiering worse, not better

The High-Bandwidth Flash characterization evaluates a Mooncake-style offload stack against four complete two-hour Qwen-Bailian production traces across five dense and MoE models on H100 and B200 profiles. Replacing NVMe with HBF raises average end-to-end latency 2–5.5× and reduces maximum SLO goodput. Faster reads do not produce faster serving. Writes outnumber reads on every tested trace because new blocks continually enter the far tier while reuse concentrates in the near tier.

The hardware organization compounds that imbalance. HBF-1 on the 2028 roadmap pairs package-local flash with GDDR7, while HBF-2 in 2030 shares package sites between HBM and HBF, displacing capacity otherwise available for hot KV. Thermal limits further constrain delivered bandwidth, and substantial changes in device latency barely move end-to-end latency. The study therefore calls for reuse-aware placement, write budgeting, and thermal coordination. Its general test has three conditions, namely that read I/O is the bottleneck, reads outweigh writes, and delivered bandwidth is sustainable.

Diagram: Why Faster Flash Made Serving Slower: The Write-Dominance Problem. Visualizes: Visualize the counterintuitive HBF result: replacing NVMe with High-Bandwidth Flash raised average end-to-end latency 2–5.5× and reduced maximum SLO goodput…

How PIM-enabled HBM hierarchies attempt to eliminate the tier boundary problem at the hot end

Processing-in-memory research seeks to cut how much data moves inside the memory stack. AttAcc places attention on bank-level HBM-PIM engines while the GPU executes projection and feed-forward layers. Uniform PIM allocation has a capacity cost. Cold KV, weights, and activations pay for logic they do not use, and the cited 40 GiB configuration provides only half the capacity of a conventional GPU. Moving computation toward memory thus creates another hot-versus-cold choice over how to use that memory.

TokenStack separates dense capacity layers from PIM-enabled compute layers within each HBM stack. Its HBM4 logic-die controller handles cross-layer DMA, address translation, attention-side coordination, and inline quantization during migration. Runtime placement, eviction, and bounded replication keep hot KV near compute while colder state occupies dense layers. The design improves throughput and SLO-compliant capacity over AttAcc in its evaluation while reducing per-token energy. These are research architectures on a 2028–2030 roadmap horizon, but they expose the same requirement as external tiering, which is to reserve scarce compute-adjacent capacity for blocks that benefit from it.

Placement policy, I/O scheduling, and format as a unified design space rather than independent knobs

Switching from BF16 to FP8 can change whether a working set fits in DRAM and whether NVMe is needed at a given concurrency. Placement then shapes the scheduler’s workload. Keeping blocks that are reused often in HBM cuts promotions, while putting blocks that are rarely reused in persistent storage adds writes whose transfer cost may never be repaid. The I/O architecture controls whether the data that still needs to move can do so without starving the GPU.

The HBF result demonstrates why these decisions cannot be evaluated independently. A device change without corresponding placement and write-budget changes produced a regression despite faster storage. Workload traces must therefore connect format, reuse distribution, request sizes, and bursts in the same evaluation. Block-level access telemetry supplies the feedback needed to make that connection. A storage layer treated only as a passive byte destination cannot give placement policy the evidence it needs to distinguish useful retention from expensive accumulation.

What infrastructure decisions should teams make for million-token serving today?

FP8 is the established production option in the notes, with NVFP4 an option on Blackwell, while entropy-coded formats remain research to track pending production validation. Local DRAM sizing follows the resulting bytes per token. A 1–2 TB system-memory configuration on an 8-GPU inference node is a defensible first extension, sized against the warm-block working set at target concurrency and context length.

NVMe evaluation comes after that DRAM budget is exhausted. It is useful when KV capacity is the actual constraint, with short-context single-user workloads gaining nothing from added I/O. Where SSD storage is warranted, the notes recommend GPU-centric I/O or DPU offload. Disaggregation then requires a workload-specific comparison between network transfer cost and the benefit of separating prefill and decode. Tests of any faster far-tier technology should check read pressure, read-to-write balance, and how much bandwidth it can sustain under the actual write load.

Block-level telemetry is needed in production at every step of that sequence. It allows placement policy to respond to the workload rather than assume that every retained block deserves the same treatment. The decisive purchase is a hierarchy whose format, placement, and transfer path have been evaluated together against the serving workload.

Sources

  1. HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving
  2. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving
  3. TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference
  4. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving
  5. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  6. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

More in Long-Context Serving