The Storage Layer Rethink That Unlocks Context Scaling Beyond What GPU Memory Can Offer

Long-context serving needs tiered KV cache storage, and PCIe is the real bottleneck.

· 8 min read
Cover illustration for “The Storage Layer Rethink That Unlocks Context Scaling Beyond What GPU Memory Can Offer”
Long-Context Serving · November 8, 2026 · 8 min read · 1,788 words

Long-context LLM serving requires a storage hierarchy built around the KV cache. For long context windows at scale, GPU HBM, CPU DRAM, local NVMe, and shared storage must work as coordinated inference tiers. The limiting factor is not just their combined capacity, but how quickly reusable cache blocks can reach the GPU without leaving compute idle.

Why GPU HBM alone cannot scale with context length and concurrency

KV cache demand grows as 2 × L × H_kv × D × S × B × bytes_per_element, where the terms represent layers, KV heads, head dimension, sequence length, batch size, and element size. A longer session consumes more cache, while every additional concurrent session adds its own state. At long context, that state routinely exceeds the model’s weight footprint on the same GPU.

The limits of the hardware constrain what software allocation can do. Production context windows have grown from tens of thousands to hundreds of thousands of tokens, with some reaching millions, and each token adds KV state to each layer. HBM capacity is becoming harder to scale than bandwidth across GPU generations, while manufacturing capacity is structurally constrained.

Moving state outside the GPU creates another constraint. HBM provides enormous internal bandwidth, but PCIe 5.0 delivers only a small fraction of it across the host boundary. External capacity can accommodate longer contexts and more sessions, but it cannot serve as an interchangeable extension of on-device memory. The architecture must account for both where state fits and how it reaches computation.

What the KV cache actually is and why it cannot simply be evicted

Autoregressive generation stores key and value vectors for past tokens so that their representations need not be recomputed at every new step. The KV cache is therefore accumulated inference state. Without it, attention cost grows quadratically with sequence length. Keeping the cache uses memory but saves computation.

That trade becomes especially important in multi-turn and agentic workloads. An agent can suspend during a tool call and then resume against the same long context. If its cache has disappeared, it must pay the full prefill cost again. At long context, that cost can be prohibitive, so inactive state still has operational value.

The cache can be kept without holding every block in HBM. Eviction strategies such as H2O, SnapKV, and Ada-KV reduce cache size, but introduce some risk of accuracy loss. For accuracy-critical or long-context workloads, lossless offload to a lower tier is preferable to lossy eviction. This makes the cache a storage object with a lifetime beyond its immediate use by the GPU, rather than memory that can simply be reclaimed whenever another request arrives.

How the four-tier storage hierarchy works in practice

GPU HBM holds active sessions and the blocks that are read or written during the forward pass. These blocks stay on-device because a slower access path stalls generation. CPU DRAM holds completed but likely-revisited blocks, including conversation prefixes and system prompts. Its microsecond access latency and lower cost make it a warm tier, extending effective cache capacity without reserving expensive HBM for every reusable prefix.

Local NVMe holds past context, long-tail prefixes, and sessions that have been idle. These blocks have less frequent access and can use storage with a much lower cost per GB. GPU Direct Storage paths allow their reload into GPU memory without CPU staging. Shared network storage, Redis, or object storage provides a further tier that makes cache portable across a serving fleet. That portability enables prefill and decode workers on separate nodes to exchange state, saving recomputation.

The serving engine decides when blocks are promoted, demoted, or retained, while the infrastructure provisioned for it sets how well those decisions can be carried out. PagedAttention, introduced in vLLM, provides the foundational fixed-block paging mechanism analogous to operating-system virtual memory. By eliminating fragmentation from variable-length requests and enabling dynamic allocation, it makes the cache manageable at the granularity that practical offload requires.

Diagram: The Four-Tier KV Cache Hierarchy. Visualizes: Visualize a vertical storage hierarchy showing four tiers ranked from fastest/smallest/most expensive to slowest/largest/cheapest: GPU HBM (active sessions, forward-pass blocks), CPU DRAM…

Where the real bottleneck lives — the PCIe transfer problem and what it costs

Meng et al. measured GPUs consuming only about a quarter of their rated power draw on average when offloading was active. Their measurements showed PCIe overheads reaching many multiples of prefill computation time and accounting for the overwhelming majority of execution time. The GPU was largely waiting for its data.

The cause is the gap between internal HBM bandwidth and transfers across the CPU–GPU boundary. Large KV blocks must cross a link that is orders of magnitude slower than the memory system that feeds computation on the device. The study defines κ_crit as the cached-to-prefill token ratio at which execution becomes memory-bound. Typical production workloads go over that threshold by orders of magnitude, especially document Q&A and multi-turn conversation.

Scheduling can compound this hardware limit. Standard iteration-level schedulers see the GPU as busy during long transfers and cannot exploit the compute otherwise available. NVLink C2C on tightly coupled CPU–GPU systems raises κ_crit substantially, while unified HBM designs raise it further. The study also identifies an attention technique as a model-level means of reducing KV data demand without changing the storage hierarchy.

GPU Direct Storage and RDMA — bypassing the CPU to recover bandwidth

GPU Direct Storage, or GDS, lets NVMe devices use DMA to transfer data directly into GPU memory, bypassing the CPU and its memory. This removes a staging bottleneck rather than making external storage equivalent to HBM. It is also important for KV cache spillover writes.

For remote storage, that path must also cross a network boundary. The direct GDS path over NVMe-oF requires RDMA, with InfiniBand or RoCE and GPUDirect RDMA-capable NICs providing the highest throughput. NVMe-oF TCP is also supported as a lower-performance transport.

For disaggregated prefill and decode, the same bandwidth requirement applies to cache exchange between workers. ConnectX-class NICs at 400–800 Gb/s allow useful payload bandwidth approaching line rate for KV block transfers. The benefit is specific to the cache’s journey. Cutting out needless CPU touches reduces the bandwidth tax between the worker that produces state and the GPU that must use it, but transfers still need to be scheduled carefully.

Quantization as a tier-pressure relief valve, not a tier replacement

KV cache quantization reduces the size of the state that must reside in memory or cross a tier boundary. FP8 KV cache, enabled in vLLM through --kv-cache-dtype fp8, halves memory use relative to BF16/FP16. More hot blocks can remain in HBM before offload begins. MLPerf Inference submissions have combined NVFP4 expert weights, FP8 KV cache, and BF16 for quantization-sensitive layers.

Smaller KV blocks take up less space in lower tiers and leave less data to transfer when they return to the GPU. Longer contexts and greater concurrency can still exhaust the space it frees, leaving NVMe to accommodate the remaining overflow.

Compression addresses a related but distinct part of the path. SplitZip targets lossless compression of KV blocks in transit during disaggregated serving. Unlike a larger storage tier, this approach acts on the bytes crossing the link. Offloaded execution can spend most of its time transferring those bytes.

What head-wise and disaggregated offload patterns add beyond simple tiering

HeadInfer changes the unit of offload from a layer to an attention head. During attention computation, it loads only the KV data for the head being computed from CPU DRAM, leaving the other heads offloaded. Heads are computed independently within a layer. That permits the on-GPU cache to shrink to a single head’s worth rather than an entire layer’s, although its size still grows with context length. The result enables single-GPU inference at context lengths that would otherwise require multi-GPU tensor parallelism.

Disaggregated serving changes where computation happens, with Mooncake’s cache-centric architecture separating prefill and decode into different node pools. Prefill workers generate cache and send it across the fabric to decode workers, making the cache an explicit network object. Routing thus depends on cache locality as well as which GPU is available.

This separation also changes hardware specialization across the fleet. Prefill nodes can be optimized for compute throughput, while decode nodes can be optimized for memory bandwidth and cache capacity. Their tier configurations need not be identical. NIXL and similar transport layers make block movement between those pools a first-class datacenter operation.

What production systems actually show — benchmarked outcomes from real deployments

Red Hat’s November 2026 benchmark on OpenShift AI 3.5 used H100 GPUs serving Nemotron-Ultra. Throughput roughly doubled with CPU DRAM offload and roughly tripled when NVMe was added as a second offload tier.

LMCache reports time-to-first-token reductions of several multiples on cache-hit-heavy workloads by persisting KV blocks across CPU DRAM, disk, and object storage. Those gains concentrate in repeated-prefix workloads, including multi-turn chat, RAG with shared system prompts, and agentic loops. Reuse is the source of the avoided prefill work. MLPerf Storage v3.0 added KV cache and vector database workload tests in September 2026, together with a new S3 data layer, formalizing cache management as a storage measurement category.

The results depend on what requests actually reuse. Meng et al. identify high NVMe prefix-cache hit rates as a condition for configurations combining FP8 and NVMe offload to work well. With unique prefixes, including creative generation and diverse agentic tasks with non-repeating context, gains collapse and I/O overhead dominates. Workload profiling must therefore precede tier sizing. An NVMe specification that works for a reuse-heavy RAG deployment may be a poor fit for creative generation.

What infrastructure is required to specify the storage hierarchy correctly?

Planning starts with measuring prefix reuse, then setting out how much each tier must hold and how data must move. One to two terabytes of system memory per eight-GPU node is defensible for the warm cache tier. This reverses the assumption that inference nodes should be RAM-light. DRAM adds to the cache capacity a GPU can use at low cost, but how much it helps depends on the host-to-GPU path described earlier.

NVMe requires capacity, endurance, and parallelism. KV traffic is write-heavy and runs all the time, so it needs datacenter-class drives with high drive writes per day ratings. Drives that work in parallel have more write bandwidth between them than a single large consumer SSD can provide. For shared cache and disaggregated serving, fabric planning likewise needs training-cluster attention, including RDMA-capable NICs over InfiniBand or RoCE.

GDS must also be explicitly configured with compatible drivers. Local NVMe paths require compatible controllers and PCIe P2PDMA support, while remote paths require RDMA-capable NICs. If this is left until deployment, the planned bypass may go unused. Emerging memory-interconnect approaches and high-bandwidth flash belong in forward-looking architecture discussions alongside provisioning the available hierarchy.

The final specification must therefore connect endurance, write bandwidth, driver compatibility, NIC class, and serving-engine behavior. Having enough storage space does not ensure that blocks can reach the GPU on time.

Sources

  1. Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  2. KV Cache Optimization Strategies for Scalableand Efficient LLM Inference
  3. HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
  4. Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5

More in Long-Context Serving