Why Long-Context LLM Serving Is a Storage Architecture Problem Before a Model Problem
KV cache storage, not model size, determines long-context serving capacity.

Serving LLMs with long context windows at scale requires a storage architecture that can place, move, and schedule KV cache across GPU HBM, host memory, SSDs, and network links. Long-context serving is a storage problem before a model problem because per-request KV state can exhaust HBM and limit concurrency. Better models reduce that burden, but the data path decides whether the system can serve the state that is left within its latency and cost targets.
Why autoregressive decoding makes KV cache size a function of context length, not model size
During autoregressive decoding, each new token must attend to the key-value pairs of the tokens that came before it. Those pairs must stay in fast memory for the whole request, including after the computation that produced them is done. Each additional token extends the retained state. The py-kvcache study traces KV cache growth to context length, batch size, model width, and layer count, so parameter count alone cannot describe the memory requirement.
The distinction from model weights is structural. Weights stay the same across requests and can be shared, while KV cache changes and grows with the context of each request. A smaller model with a long context can therefore need more storage than a larger model with a short context. On a large model in BF16, very long contexts can use tens of gigabytes of KV cache per request, more than the HBM on many production accelerators can hold, and cut concurrent batch sizes to single digits.
How GPU HBM becomes the hard ceiling that caps serving throughput before any model optimization applies
GPU HBM is the fastest available tier, but also the smallest. At long context, KV cache can use almost all of it, so there is little room for model weights or larger batches without offloading. Throughput is constrained before additional compute can help. Latency and cost per token then depend on how much memory is still usable and how well the system handles state that no longer fits.
The same study finds that once the reusable working set exceeds VRAM capacity, the system must evict useful KV blocks or move them to slower tiers. Without offloading, concurrency is bounded by HBM rather than the node’s total NVMe capacity, regardless of attention architecture or quantization choice. CPU DRAM, interconnect-attached memory, and SSDs add space only if the serving engine has a transfer path it can use to reach them.
Larger-HBM accelerators postpone this capacity limit. Context length and concurrent request counts still drive linear growth in retained state. Better hardware changes where the constraint appears. Model-level optimizations work within that physical budget, while storage architecture decides what happens when it runs out.
What infrastructure does a serving node need for long contexts, and what does each tier cost in latency?
GPU HBM provides the highest bandwidth and sub-microsecond latency, making it the fast tier against which every offload path is measured. CPU DRAM offers greater capacity but substantially lower bandwidth and microsecond-range GPU-observed latency. Its capacity is still bounded by memory channels and DIMM slots. The Tutti paper estimates that about two terabytes of DRAM retains only around five minutes of KV cache at scale. Host memory is therefore a capacity-limited intermediate tier.
CXL-attached memory offers a byte-addressable interface that eliminates block-I/O staging costs. That interface alone does not deliver DRAM-like performance. The LM-CXD paper finds that a stock CXL-SSD is roughly three times slower than local DRAM and no faster than an NVMe SSD, while generic device prefetching leaves average time-to-first-token, or TTFT, nearly unchanged. NVMe SSDs offer large capacity at low cost per gigabyte, but their block path adds CPU cache contention, staging, and NAND latency. RDMA fabrics and parallel filesystems spread the hierarchy across nodes, so transfers have another boundary to cross.
These gaps make interface overhead a central engineering constraint in realizing advertised bandwidth. The same paper finds that isolating CPU cache contention from inference threads lowers P99 TTFT by up to 44%. Device speed cannot explain that improvement alone. Block-I/O handling, intermediate buffers, and the mismatch between KV layout and storage transfer units consume time even when the underlying medium is fast. Naive offloading can therefore damage TTFT unless scheduling accounts for the entire path.
Why the fragmented GPU memory layout turns NVMe offloading into a CPU bottleneck
Serving engines such as vLLM manage KV cache in small, non-contiguous paged blocks. With that layout, loading a long prefix back from SSDs takes many tiny random I/Os, which overwhelm the CPU even when the drives have bandwidth to spare. Fragmentation becomes a control-path problem as well as a data-transfer problem. Tutti reports 70–80% GPU stalls due to KV cache transfers from SSDs and identifies scattered I/Os as the primary cause.
GPUDirect Storage, or GDS, moves data directly between SSD and GPU, but still requires CPU intervention to initiate each I/O. The CPU therefore remains a scheduling bottleneck. LM-CXD also finds that GDS-based approaches retain block-interface costs because data first enters a VRAM staging buffer, then passes through a reshape and copy into the final KV layout. The final placement of the data remains part of the retrieval cost.
Pipelining transfers with computation does not automatically solve this failure mode either. Tutti finds that applying the standard DRAM-offload mitigation to SSDs fragments transfers further, reduces effective bandwidth, and adds CPU scheduling overhead. More overlap can produce less useful I/O. Fixing this therefore requires changes to transfer granularity and control, beyond what faster drives or more transfer stages can do.
Three system designs that rearchitect the KV I/O path rather than work around it
Tutti, from Xiamen University, SJTU, and HKUST, published in May 2026, takes the GPU-native I/O approach. It treats KV cache as objects, using bulk transfers, GPU io_uring for asynchronous direct object I/O, and slack-aware scheduling to avoid GPU resource contention. The CPU loads I/O kernels asynchronously once per layer and then is out of the critical data and I/O control paths. The paper reports 78.3% lower TTFT and roughly double the achievable request rate compared with GDS-enabled LMCache under strict SLO constraints, while reducing serving cost.
LM-CXD, from Sogang University, Samsung Electronics, and ETRI, published in September 2026, instead specializes the device interface. It makes KV chunks device-visible I/O units, exposes NAND-to-DRAM progress, and uses device DRAM as a GPU-accessible buffer. The serving engine and device share consumption and placement information. Windowed prefetching and layerwise movement then coordinate NAND access with GPU computation while avoiding blind fragmentation of SSD transfers. The paper reports average TTFT within 1.5× of local DRAM across five LLM models, with layerwise prefetching improving average TTFT over a stock CXL-SSD by up to 4.03×.
The py-kvcache connector, from Vrije Universiteit Amsterdam and IBM Research Zurich, published in September 2026, focuses on scheduler integration. Its asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading start disk reads while requests are still queued. Transfer timing becomes part of request scheduling. The study also finds that short prefixes or fast GPUs can make recomputation faster than an external cache hit, making admission policy as consequential as transfer throughput. These are complementary architectural directions, with GPU-native control, device specialization, and scheduling each addressing a different part of the KV I/O path.
How disaggregated prefill/decode architectures expose the network as the next storage bottleneck
Separating prefill and decode across GPU pools requires moving the KV state produced by prefill to the nodes that will decode. A naive path copies data from GPU HBM into pinned host DRAM through cudaMemcpy, invokes an RDMA verb through the NIC, and reverses those steps at the destination. Each hop adds latency, while CPU staging reintroduces work that single-node designs seek to remove. The interconnect can then set the throughput limit.
Mooncake, from Moonshot AI / Kimi, treats KV cache as a disaggregated resource spread across cluster-wide CPU DRAM and SSDs. Its global cache and scheduler coordinate separate prefill and decode clusters to maximize throughput within latency SLOs, trading more storage for less computation. Messenger handles KV block transfers across CPUs and GPUs through GPUDirect RDMA. Yet each NIC has a limit on its bandwidth. Mooncake remains bounded by that RDMA capacity, even when the underlying storage can supply more data.
NVIDIA Dynamo and the Inference Context Memory Storage Platform, or ICMSP, address the same boundary through compute and transfer separation. Dynamo disaggregates prefill and decode across GPUs, while ICMSP, announced at CES 2026, uses BlueField-4 DPUs to manage KV movement off the GPU compute path. Streaming multiprocessors can focus on the forward pass, freed from PCIe I/O stalls. This removes transfer management from scarce compute resources. The KV bytes still have to move from where they are stored to where they are used for execution.
What KV cache compression actually does to the storage problem — and what it cannot do
Switching KV storage from BF16 to FP8 halves cache memory with negligible quality loss on most production workloads. That is a substantial gain in usable capacity. It does not, however, determine end-to-end performance by itself. Nikita Agrawal and Ruben Mayer’s KV cache benchmark finds that KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long-context throughput, and CaM shows substantial workload sensitivity.
Even after quantization, long contexts under sufficient concurrency can still saturate HBM, leaving offloading necessary. Multi-Head Latent Attention reduces per-request KV state more aggressively than quantization, but growing contexts and concurrent sessions continue to increase the total footprint. Agentic workloads also accumulate state across multi-turn sessions spanning hours of inference. Frameworks that assume standard Multi-Head Attention can further waste MLA’s savings by over-provisioning KV memory, turning a model improvement into a systems integration problem.
ExANS, introduced in August 2026, exploits the low entropy of the BF16 exponent field within KV tensors, even though BF16 as a whole is generally high-entropy and lacks repeating bytes. It aligns exponents to a local center and entropy-codes them with ANS encoding, with high-throughput GPU-native decoding. Compression built for this data can therefore cut the storage space and network traffic it needs without accuracy loss, but the compressed state still has to be placed, moved, and scheduled across tiers.
Why MLPerf Storage v3.0 adding KV cache workloads signals that the industry has accepted the storage-first framing
MLPerf Storage v3.0, announced September 1, 2026, makes KV cache serving a canonical storage workload for the first time in a major storage benchmark. The benchmark adds tests for KV cache and vector database use in inference, along with an S3 object storage access layer beside the POSIX layer it already has. Its KV workloads exercise mixed reads and writes under concurrent I/O pressure. Storage is tested on simultaneous prefetch and eviction, rather than sequential access that avoids the serving system’s central contention.
The v3.0 round has 143 results from 19 organizations that submitted them. Samsung’s PM1763 PCIe Gen6 NVMe SSD demonstrated concurrent read and write bandwidth while serving a LLaMA 3.1 workload, showing that production-grade NVMe can support KV serving when the I/O path is designed appropriately. Other configurations expose the matching network requirement. Some systems use dual 400 Gb/s NDR InfiniBand/RDMA links per node for payload traffic, illustrating why SSD throughput cannot be evaluated apart from interconnect capacity.
For infrastructure teams, the practical implication is a reproducible, architecture-neutral way to evaluate whether storage can sustain production KV serving loads. Long-context serving throughput is bounded by the storage layer’s ability to retain and deliver KV state, and the industry now has a benchmark that measures that constraint directly.
Sources
- Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving
- Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
- Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
- Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
- Predictive Multi-Tier Memory Management for KV Cachein Large-Scale GPU Inference


