How Hyperscale LLM API Teams Architect KV Cache Management Under Real Throughput Pressure

Tiered KV caches only pay off when routing, I/O, and topology cooperate.

· 8 min read
Cover illustration for “How Hyperscale LLM API Teams Architect KV Cache Management Under Real Throughput Pressure”
Long-Context Serving · November 9, 2026 · 8 min read · 1,832 words

Production LLM serving requires a KV cache architecture that spans GPU HBM, CPU DRAM, and NVMe SSD, with transfers governed by workload-aware routing and scheduling. Once the cache exceeds HBM, throughput and cost depend on how much data moves, where it moves, and whether it arrives before compute needs it.

Why KV cache overflows GPU HBM at hyperscale and what that forces

KV cache size grows linearly with context length, batch size, model width, and number of layers. That growth is structural. HeadInfer quantifies the pressure at very long context lengths as tens to hundreds of gigabytes for KV cache alone, beyond a single 80 GB H100. Multi-turn, web-use, and coding workloads have much longer context than single-turn requests, so overflow is common in production.

OasisKV identifies a second constraint beyond storage capacity. Every request keeps the KV cache it has built up, which limits batch size, while dense attention reads that cache again and again, so HBM bandwidth limits decode speed. Capacity and bandwidth therefore constrain serving together. Even with more GPUs, the system still needs rules for how to keep and access the state it has built up. Overflow forces an architectural choice beyond the all-in-HBM model.

The three-tier storage hierarchy that absorbs KV overflow

The standard response separates KV blocks by their expected use. Hot blocks needed for active generation remain in GPU HBM, warm blocks likely to return in multi-turn requests move to CPU DRAM, and cold historical context or prefix cache moves to NVMe SSD. Each transition adds latency to later access. In the staged hierarchy, a block returning from NVMe crosses both the storage and host-to-GPU interfaces, while a miss beyond NVMe requires full prefill recomputation.

Offloading helps when GPU KV capacity is genuinely the bottleneck, but short-context or low-concurrency workloads can incur I/O latency without gaining useful capacity. py-kvcache finds that short-prefix cache hits on a strong GPU such as an H100 can be slower than recomputation. NVIDIA's ICMSP hardware, announced at CES 2026, formalizes tiered movement through BlueField-4 DPUs, keeping that work off GPU streaming multiprocessors. Even with hardware built for the task, the hierarchy only provides places to store data. Those places help serving when their management rules support it.

Offload granularity and how the unit of transfer shapes latency and hit rate

HeadInfer demonstrates how sharply the transfer unit changes memory requirements. Its head-wise offloading reduces the KV footprint in a long-context inference from 128 GB to 1 GB and total GPU memory from 207 GB to 17 GB. Attention heads are mathematically independent, so attention output can be computed as each head's cache is loaded. With conventional layer-wise offloading, the cache must hold at least all the heads in one layer.

That finer unit requires more careful coordination. HeadInfer combines adaptive head grouping, chunked prefill, and asynchronous GPU–CPU transfers to control residency and overlap movement with compute. At the token-block level, py-kvcache likewise finds that performance depends on transfer granularity and intermediate memory use as well as device bandwidth. Coarse transfers simplify implementation but move unused data, while fine transfers create more small I/O operations and scheduling work. Neither choice alone establishes hit rate, because reuse also depends on where requests and their blocks are routed.

Prefix-hash routing as the mechanism that turns tiering into hit rate

Prefix-hash routing assigns KV blocks to storage locations deterministically from their token prefix. Consistent hashing governs placement. A request routed to a node without its relevant blocks experiences the equivalent of a miss even if the data exists elsewhere in the cluster. That miss triggers the quadratically expensive prefill fallback.

Admission scheduling decides whether to fetch those blocks and when fetching them will help. A hit arriving after the GPU starts recomputing the prefix contributes nothing and consumes I/O bandwidth. py-kvcache starts disk reads while requests are still queued, making scheduling delay available for preloading. At sufficient context lengths, it loads from disk at twice the speed of GPU recomputation. In disaggregated serving, placement also spans the network because prefill workers must transfer generated KV blocks to separate decode workers through RDMA.

Why I/O scheduling and datapath selection determine whether offloading pays off

Serving is still slow when placement is correct but the datapath is slow. For inference's small random reads, io_uring provides better latency and competitive NVMe IOPS, while GPUDirect Storage benefits the large sequential transfers used in pre-training and fine-tuning by reducing model load time and CPU usage. The pipeline phase determines the useful mechanism. py-kvcache combines asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading for vLLM. Its implementation performs within a few percent of native vLLM KV Offload and outperforms LMCache at long contexts.

NVIDIA NIXL unifies RDMA and GPUDirect movement across GPU, CPU, NVMe, and cloud object stores, with integrations into vLLM, SGLang, TensorRT-LLM, and Anyscale Ray. NVIDIA's Dynamo uses NIXL to transport KV between prefill and decode workers. NVIDIA cuObject, released January 2026, provides a public API for the RDMA fast path between S3-compatible object storage and GPU memory. These mechanisms still depend on transfer scheduling, and their available throughput depends on the PCIe topology beneath them.

PCIe topology and RDMA placement as hidden multipliers on transfer throughput

Traditional Kubernetes device plugins can assign a GPU and network adapter across NUMA boundaries, allowing the pod to start without errors while GPUDirect RDMA traffic traverses suboptimal buses. The fault appears under load. Red Hat's vLLM benchmarks show that eliminating cross-bus bottlenecks cuts KV transfer latency in half and doubles throughput.

Dynamic Resource Allocation gives operators a way to avoid that mismatch. It lets workloads request devices by hardware attributes such as PCIe topology, so scheduling can match devices by how they connect as well as by their counts. Device counts alone cannot express that requirement. At 400 GbE and above, the notes' training case also makes kernel-bypass RDMA a production prerequisite because TCP interrupt handling and packet copies can starve GPUs. Moving transport off the host CPU addresses that burden, but it does not excuse an inefficient GPU–NIC pairing.

Sparse prefetching as a strategy for hiding offload latency during decode

Decode imposes a tighter timing constraint than queued-request preloading. Dense attention reads the full KV cache for every generated token, making HBM bandwidth a limit on token throughput even when compute remains available. Sparse attention cuts down on those reads by choosing which previous tokens to read. On its own, however, it does not reduce the capacity needed to retain the full cache. OasisKV addresses both constraints by offloading the full cache and keeping only a small active working set in HBM.

Its lookahead mechanism uses speculative decoding to predict which tokens will be important in the next step. Asynchronous prefetching stages their KV entries before attention needs them, hiding transfer latency behind ongoing compute. Prediction accuracy is the central practical condition. OasisKV reports average accuracy within a negligible margin of full attention on long-context benchmarks, throughput gains over vLLM's dense baseline, and greater concurrency under the same host-memory budget in disaggregated serving. The predicted set must still fit the available PCIe or network bandwidth, because prefetched blocks that go unused take up transfer capacity.

CoT reasoning as a structurally different KV management problem

The preceding mechanisms determine how blocks move, but not whether every block deserves retention. Chain-of-Thought reasoning exposes that distinction because think-stage tokens are large in volume while the user's objective is the final answer. Tokens that get attention during reasoning may have no bearing on the answer. DeepSeek-R1, Qwen3, and similar reasoning models can generate thousands of think-stage tokens per request, filling both HBM and offload tiers. A policy based only on recency or attention can keep state after it has stopped helping reasoning or the answer.

Crystal-KV separates SlipKV, which maintains reasoning flow but may introduce misleading context, from CrystalKV, which contributes to answer correctness. Its answer-first principle maps answer preferences back into think-stage attention. Eviction then follows the generation objective. An attention-based Least Recently Frequently Used algorithm identifies expired SlipKV utility, while adaptive budgets adjust by layer and head according to the changing proportion of CrystalKV. Because of this distinction, a single global LRU policy is a poor fit for dialogue, RAG, and reasoning taken together.

Lossless compression as a throughput multiplier for KV data in transit and at rest

Lossless compression of BF16 KV tensors can increase effective bandwidth across PCIe, NVMe, and the network without changing those interfaces. The gain comes from moving less data. It also reduces the space occupied by cached state, improving the economics of retention as well as transfer. Those benefits depend on achievable compression and processing throughput.

IBM Research's incremental tensor snapshot work supplies a concrete, though distinct, checkpointing example. That result is not a KV compression guarantee. Inference-time KV compression can achieve meaningful size reductions in some FP8 and BF16 cases, but the lossless requirement remains explicit. Lossy reconstruction can change the inputs to attention in ways that the storage layer cannot spot or fix at serving time.

What MLPerf Storage v3.0 reveals about production KV cache I/O requirements

MLPerf Storage v3.0's KV Cache test measures storage performance for LLM inference cache reads and writes. This supplies a reproducible profile for evaluating the storage side of offloading rather than inferring suitability from peak sequential bandwidth. The workload is the relevant reference point. MLCommons announced the round in September 2026, adding S3 object storage access beyond the existing POSIX tests and introducing both KV Cache and Vector Database workloads. The dedicated cache test makes inference I/O externally comparable.

The broader round also establishes useful context for storage evaluation. The round's checkpointing test shows write bandwidths of hundreds of GiB/s across dozens of data nodes, which scale predictably as the number of nodes grows. Roughly one in six total submissions used S3 access, and first-time entrants included Azure, NVIDIA, and Nebius. Participation against a public workload profile gives vendor claims a firmer basis than internal tests alone, without making every submitted workload interchangeable.

How should inference platform teams manage KV cache memory efficiently at scale?

Tiering establishes capacity, granularity controls how efficiently it is used, and routing enables hits on stored state. I/O scheduling and topology control the timing and usefulness of that hit. Prefetching addresses decode timing, objective-aware eviction governs retention, and lossless compression reduces transfer volume throughout the chain.

Teams should establish whether capacity is actually limiting throughput before choosing offload granularity and measuring cache admission against GPU recomputation. That comparison must remain dynamic at runtime. py-kvcache shows why GPU generation and workload mix change the break-even point, making a static deployment setting insufficient. Storage evaluation should then cover io_uring for random inference reads, GPUDirect paths where appropriate, topology alignment, and lossless BF16 handling against the relevant benchmark profile.

Readers moving from evaluation to deployment can use code-hosting and community chat platforms to look at implementations and discuss benchmark or integration work. The governing criterion remains whether the complete path delivers useful KV state before compute needs it, at less cost than the fallback.

Sources

  1. HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
  2. Crystal-KV: Efficient KV Cache Management for Chain-of-Thought LLMs via Answer-First Principle
  3. OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
  4. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

More in Long-Context Serving