KV Cache Memory Scaling From 32K to 1M Token Contexts

Tiered memory strategies keep long-context LLM inference tractable as token windows scale.

· 12 min read · Updated
Cover illustration for “KV Cache Memory Scaling From 32K to 1M Token Contexts”
Long-Context Serving · October 7, 2026 · 12 min read · 2,706 words

KV cache memory is what decides whether a long-context serving system works at all, not a detail to tune after the fact. As context windows move from modest sizes to the million-token range now showing up in production, the memory needed to hold past key-value pairs grows in lockstep, and at some point it becomes the single largest line item in the system's resource budget, ahead of the model weights themselves.

GPU memory management for long-context LLM inference centers on the KV cache, which grows linearly with context length and becomes the dominant memory cost ahead of model weights themselves at long enough sequences. The practical strategy follows the hardware hierarchy: keep hot KV blocks in GPU HBM for active requests, offload less-accessed blocks to CPU DRAM, and push cold prefix cache entries to NVMe SSD, accepting bandwidth penalties of roughly 50:1 and 480:1 at each step down. Quantizing the KV cache to FP8 halves per-token memory and bandwidth pressure with manageable accuracy costs if kernels are built for it, while eviction and layer-aware allocation reduce how many tokens are kept at full precision at all. At the longest contexts, prefill-decode disaggregation across separate worker pools, with fast RDMA-based KV transfer between them, is the architecture that makes the remaining gains tractable.

Why is GPU memory so hard to manage for long-context LLM inference?

Every token a model generates needs the key and value vectors of every token that came before it, held in memory so the model doesn't have to recompute them from scratch at each step. There's no way around storing that data short of evicting some of it or compressing it, and both of those moves come with costs the rest of this piece will work through. What matters first is the shape of the growth: KV cache size rises in strict lockstep with context length. Doubling the number of tokens in context doubles the memory needed to hold their keys and values, for a fixed batch size and a fixed model. There's no sublinear path available here by default.

Attention computation itself scales quadratically with sequence length, but Flash Attention can bring the memory footprint down to something closer to linear. But even once you have that optimization in place, linear growth at a million tokens still means a steep, compounding load spread across every layer of the model. Nothing cancels it out; it just becomes manageable instead of catastrophic.

At common precision levels, the practical result is stark. At BF16, memory demand climbs so steeply with context length that a single request running at maximum context can use up more HBM than one H100 GPU has, before you even load the model weights into memory. And that's for one request. Batch size multiplies the pressure on top of that: serving several concurrent users at long context can exhaust the memory of a multi-GPU setup in short order, because the per-request KV cost simply gets multiplied by however many requests are running at once.

This isn't a problem waiting in the wings for some future generation of models. Open-source models already ship with very long context windows, and proprietary models are pushing those windows further still. Teams running Llama, Qwen, or Mixtral at production scale are dealing with this today, in whatever serving stack they've already built, with whatever hardware they already have.

GPU memory hierarchy and context scale limits

You have to map the numbers above onto actual hardware before they mean anything, and that hardware comes in tiers with wildly different bandwidth. GPU HBM, the fastest tier, runs at around 3.35 TB/s on an H100 SXM5, and it's where the hot KV blocks for active requests need to live if the model is going to generate tokens quickly. Below that sits CPU DRAM, reachable over PCIe, and below that sits NVMe SSD storage, running at roughly 7 GB/s over PCIe 4.0, useful for cold historical context and prefix cache entries that aren't needed on every step.

The drop between these tiers isn't a mild slowdown; it's a cliff. Going from HBM to CPU-GPU transfer over PCIe 5.0 costs roughly a 50:1 drop in bandwidth. Going from HBM all the way to NVMe costs roughly 480:1. Those ratios turn directly into latency a user can feel: moving a large KV cache out of HBM takes around 15 milliseconds, while pulling the same data back in from CPU DRAM takes far longer, and under load that gap appears as inter-token latency or as a slower time-to-first-token.

There's a second problem layered on top of the raw bandwidth gap. KV blocks on the GPU are often stored in a paged, non-contiguous layout for efficiency, and when those blocks get evicted to SSD, that same layout turns into severe I/O fragmentation on the storage side. The mismatch between how paged KV data sits in GPU memory and how SSDs want to be read from is a distinct engineering problem in its own right, not just a slower version of the same bandwidth issue.

Context parallelism shards the KV cache across multiple GPUs, so no single device has to hold the whole thing, and it can reach high efficiency for very large models. It needs scale and interconnect that not every team has lying around, making it an option mainly for organizations already running sizable GPU clusters.

So these tiers roughly set the architectural choices you make as context length grows. At shorter contexts, staying entirely within GPU HBM is workable with some discipline about batch size and model choice. At medium contexts, you need CPU offloading or aggressive quantization to keep the active working set within reach. At the longest contexts, NVMe offload, disaggregated serving, or sequence parallelism stop being optional. These thresholds aren't fixed: batch size, model size, and the generation of hardware in use all shift the exact numbers where one tier gives way to the next. But the underlying logic holds no matter the configuration, so infrastructure teams should treat each tier transition as a decision made ahead of time.

Diagram: The Memory Bandwidth Cliff: HBM to NVMe. Visualizes: Visualize the three-tier GPU memory hierarchy and the catastrophic bandwidth drops between them.

Prefill-decode disaggregation and RDMA interconnects

Serving an LLM request involves two phases, and they have very different resource profiles. Prefill, which processes the incoming prompt, is compute-bound and saturates the GPU's processing capacity. Decode, which generates tokens one at a time, is memory-bandwidth-bound and saturates HBM bandwidth instead. Running both phases on the same GPU pool causes them to compete for the same resources, and that interference produces spikes in p99 latency under load. A system can look fine on paper and still stall badly for a fraction of users at the worst possible moments.

Disaggregating the two phases into separate, dedicated worker pools addresses this directly. Prefill workers handle only prefill, decode workers handle only decode, and each pool can be sized and tuned for its own resource profile. Done well, this architecture produces realistic gains of 2 to 3 times the throughput at equal hardware cost for long-prompt, high-concurrency workloads. Once prefill and decode run on separate machines, the KV cache computed during prefill has to move to the decode worker before generation can start, and solving that transfer is harder than the disaggregation itself. If that transfer is slow, it eats the entire throughput gain disaggregation was supposed to deliver. Disaggregation itself isn't the hard part; you have to get the KV cache across the wire fast enough that splitting the work was worth it.

Moonshot AI's Mooncake system, which won the Best Paper award at FAST 2025, is the clearest production proof that this can be solved at scale. Mooncake runs in production at Kimi, where it processes over 100 billion tokens daily across thousands of nodes, and the team reports substantial throughput gains in long-context scenarios as a result. It treats the KV cache as a resource that lives across GPU memory, CPU DRAM, and SSD as a deliberately managed hierarchy, rather than treating anything outside GPU memory as a fallback. The Mooncake paper describes this as a disaggregated cache that uses the underutilized CPU, DRAM, and SSD resources of the GPU cluster, with KV entries placed across VRAM, DRAM, and SSD according to how often they're accessed.

Moonshot open-sourced the high-speed networking-based Transfer Engine that makes this work, and because its topology-aware path selection picks the fastest available route between nodes rather than a fixed one, it cut mean time-to-first-token by a quarter compared to plain TCP transport. That quarter reduction is the kind of number that separates disaggregation as a theoretical win from disaggregation as something a team can actually ship.

NVIDIA's answer to the same problem is Dynamo, an orchestration layer that sits above vLLM and routes prefill and decode work to separate, dedicated pools. Dynamo relies on a set of transport and serving libraries to handle the actual KV transport over high-speed node interconnects, giving teams already standardized on NVIDIA's stack a packaged path to the same disaggregated architecture Mooncake proved out independently. Support for equivalent disaggregated KV transfer on other hardware platforms is less standardized at this point, which matters for any team choosing a hardware platform with long-context serving as a requirement.

NVIDIA ICMSP and NVMe-resident KV cache standardization

NVIDIA's Inference Context Memory Storage Platform, ICMSP, takes the pattern Mooncake proved out and turns it into a standardized platform offering, so each team no longer has to build it for itself. It's the first major attempt to make NVMe-resident KV cache a formal part of the inference memory address space, instead of a workaround bolted on by whichever engineering team happened to hit the wall first.

The platform relies on specialized data processing units to handle the mechanics of that offload. BlueField-4 eliminates metadata overhead associated with managing KV cache placement, reduces the amount of data that has to move between nodes, and gives GPU nodes secure, isolated access to cache data that may be shared across a rack. On top of that hardware sits NVIDIA's software stack, built around the DOCA framework, the Dynamo KV cache offload engine, and NIXL, which together handle the work of sharing KV cache across AI nodes without requiring each application to manage that logic itself.

NVIDIA states that the platform increases KV cache capacity available to a cluster and speeds up sharing of that cache across rack-scale AI systems, which has the most practical value in multi-turn agent workloads, where keeping persistent context resident and shareable across many exchanges improves responsiveness and supports scaling long-context, multi-agent inference without forcing every agent session to rebuild its context from scratch.

ICMSP's significance isn't the specific capacity number NVIDIA quotes; NVMe is being formally absorbed into the managed KV address space as a standing architectural tier, the way Tier 1 HBM and Tier 2 CPU DRAM already are, rather than treated as an emergency overflow buffer a team reaches for only when everything else is full. Once a cluster is managing KV cache across three tiers as a matter of course, how aggressively each block gets compressed before it moves between tiers becomes one of the most consequential decisions in the whole system, since it directly determines how much context fits in a given footprint and how fast it can move when needed<sup>1</sup><sup>2</sup><sup>3</sup>.

KV cache quantization accuracy costs at FP8 and 4-bit precision

Quantization is the fastest compression technique to deploy, since it doesn't require restructuring the serving architecture the way disaggregation or NVMe offload do. It comes with accuracy costs that get steeper as precision drops, and those costs need to be measured against the specific workload in question.

If you move KV cache from BF16 down to FP8, you cut memory per cached token in half and cut memory traffic per attention step in half too, and that reduction in traffic translates directly into a flatter inter-token latency curve as context grows. That's a real, measurable win, and it's available at a precision level that's relatively easy to support in existing kernels.

But if you apply naive FP8 attention without care, it can cause severe accuracy regressions at long context. If you test this on a long-context needle-in-a-haystack task, where a model has to retrieve one specific fact buried deep in a long input, retrieval accuracy drops dramatically under naive FP8 without careful kernel-level corrections. FP8 attention needs kernels purpose-built to handle the numerical range and precision loss correctly, and teams that drop in FP8 without that engineering work are likely to see failures concentrated exactly in the long-range retrieval cases that long-context serving exists to support.

At 4-bit precision, benchmarking of one widely-used low-precision format gives the clearest publicly available data on where the accuracy floor sits at that level. These figures represent the best empirical evidence currently available on the trade-off between 4-bit KV compression and task accuracy, not an endorsement of that format as a default choice<sup>1</sup><sup>2</sup><sup>3</sup>. Whether a given workload can tolerate that level of compression depends on how sensitive its tasks are to the kind of precision loss that format introduces, and that has to be tested case by case.

KV cache compression reduces memory bandwidth use more than weight quantization does once context length gets large, as shown by empirical benchmarking at very long context. In tested configurations where the KV cache stayed at FP16 while weights were quantized down, the KV cache still dominated memory bandwidth. Quantizing the weights further did little to solve the actual bottleneck. The practical implication is direct: teams that have already quantized their model weights but left the KV cache at full precision are very likely leaving the larger available gain untouched. Weight quantization is table stakes at this point; KV cache precision is where the remaining headroom lives.

None of this works by reusing compression methods built for weight tensors. BF16 and other AI-native numeric formats need compression strategies built specifically for how KV tensors get accessed and how much precision they need under long-context attention, because general-purpose approaches built around weight tensors don't transfer cleanly to how KV data gets read and written during generation.

Token eviction and layer-aware compression for KV footprint reduction

Quantization reduces how much space each token's KV data takes up. Eviction and layer-aware compression take a different approach: they reduce how many tokens get kept in full at all, by exploiting the fact that not every token matters equally to every layer of the model.

Research into how attention actually behaves across a transformer's layers has found a consistent pattern, sometimes described as pyramidal information funneling: attention in the lower layers of a model is broad and scattered across many tokens, while attention in the higher layers narrows and concentrates on a smaller set of critical tokens. So if you apply a uniform KV cache budget evenly across every layer, you waste capacity in the upper layers and risk under-allocating it in the lower ones. Allocating more cache to lower layers and less to higher ones, following the shape of that attention pattern rather than ignoring it, is the core idea behind pyramidal compression approaches built on this observation.

Eviction carries its own risk, though, and it's a serious one. If you discard a token's KV entry, you permanently lose whatever information that token contributed to the model's understanding of the sequence, and heuristic eviction policies, the kind that drop tokens based on simple rules like age or raw attention score, risk dropping something the model will need later, which is the failure mode motivating recent work on learned eviction, where a trained indexer predicts which tokens actually matter to keep rather than relying on a fixed heuristic, paired with a latent memory module that compresses the information from evicted tokens into a compact, continuously updated state. That compensation mechanism provides a residual readout the model can still draw on, recovering some of the attention contribution that would otherwise be lost for good.

The throughline across quantization, eviction, and layer-aware allocation is the same one that runs through the memory hierarchy and the disaggregation architecture covered earlier: KV cache at long context isn't a side effect of serving large models, and managing it well requires treating it as its own system, with its own tiers, its own transfer paths, and its own compression techniques built for the particular way attention actually uses it.

Sources

  1. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
  2. IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
  3. KV CACHE OPTIMIZATION STRATEGIES FOR SCALABLE AND EFFICIENT LLM INFERENCE

More in Long-Context Serving