Why Storage Latency Becomes the Binding Constraint in Production LLM Inference Pipelines

Long-context KV caches outgrow HBM, so I/O speed decides latency.

· 8 min read
Cover illustration for “Why Storage Latency Becomes the Binding Constraint in Production LLM Inference Pipelines”
Distributed Serving · November 11, 2026 · 8 min read · 1,888 words

Production LLM inference becomes storage-bound when KV caches outgrow GPU memory and retrieval cannot keep pace with computation. NVMe and tiered storage improve inference throughput only when the I/O path delivers reused KV blocks faster than recomputation would produce them. The architecture that reduces latency therefore combines added storage capacity with sufficient bandwidth, direct I/O, scheduler-aware preloading, and transfer–compute overlap.

Why KV cache size grows faster than available GPU memory at production context lengths

KV state grows linearly with context length and batch size, while GPU HBM provides a fixed memory budget. For a large-class model, each token requires roughly 300–350 KB of KV state at FP16. A single long-context session can therefore approach tens of gigabytes before concurrency enters the calculation. Fifty concurrent long-context sessions exceed any single GPU’s memory by an order of magnitude. Weight quantization alone cannot resolve this mismatch because it does not remove the separate growth of the KV cache.

Agentic workloads make that capacity pressure persist beyond active generation. A session suspended during a tool call must either preserve its KV state or pay the full prefill cost again when it resumes. The cache becomes part of session continuity. Persistence is consequently more than a way to accelerate repeated prompts, although recomputation remains an alternative to retaining the state. Once active and suspended sessions exceed HBM capacity, a serving system that preserves their caches must place some of that state in slower memory or storage.

The memory hierarchy’s bandwidth gaps and what they mean for a stalled GPU

Moving KV state out of HBM makes room but leaves the system with data to deliver. HBM supplies TB/s, while the CPU–GPU PCIe 5.0 link supplies roughly 128 GB/s. That link is already a substantial restriction before an NVMe device enters the path. The per-tier access-latency ladder widens from under 1 µs for HBM to 10–100 µs for CPU DRAM and 100 µs–1 ms for NVMe SSDs.

The transfer can take longer than the computation it enables. For a long-context document served by a large model, moving the KV state over PCIe can dominate end-to-end latency by an order of magnitude or more. Faster GPU computation does not make that transfer disappear. Unless useful computation overlaps the movement, the GPU waits for the next data it needs. The resulting idle intervals reduce utilization and throughput even though the accelerator has ample compute capacity.

NVMe adds another boundary alongside the existing PCIe restriction. DUAL-BLADE, by Jeong et al., 2026, documents how conventional filesystem paths prevent full use of device bandwidth, with page-cache thrashing during decode reducing effective I/O performance further. What reaches the GPU depends on both the physical links and the path used to traverse them.

How the software stack adds latency on top of the hardware limits

Software overhead compounds these hardware limits. A storage request usually goes through VFS, the filesystem, the block layer, and the device driver. DUAL-BLADE identifies excessive stack overhead, page-cache thrashing, and lost sequential locality as distinct failure modes. In particular, blk-mq can fragment contiguous KV streams across NVMe queues.

The penalties also differ between prefill and decode. An mmap/page-cache approach can trigger synchronous write-back during heavy prefill writes, producing write stalls. During decode, sequential rereads can evict pages before reuse and collapse the hit ratio. More device bandwidth on paper does not fix either failure in the access path. The software must keep locality intact and keep work off the critical path when it need not be there.

py-kvcache, by Kanichai et al., 2026, finds that transfer granularity, intermediate staging memory, and scheduling time determine observed latency in vLLM. Tutti, an arXiv 2026 paper, addresses a related scheduling problem with GPU io_uring, or gio_uring. It removes I/O submission and completion from the GPU computation critical path, using NVIDIA green contexts to separate a Compute Domain from an I/O Control Domain. Latency-sensitive kernels therefore need not compete with I/O operations in the same way. Direct-I/O and GPUDirect Storage architectures address other parts of this path, but their benefit still depends on when the serving scheduler requests the data.

Where KV offload actually breaks the recompute-vs-retrieve tradeoff

py-kvcache finds that short prefixes or fast GPUs such as H100 can put average requests below the break-even point for external caching. Recomputing the prefix can cost less than a cache lookup, a disk read, and a CPU–GPU transfer. External KV caching should therefore be a setup-specific admission decision, governed by prefix length, GPU speed, hit rate, and I/O efficiency.

At long context, that comparison reverses sharply. Recomputing a long prefix from scratch takes roughly 11 seconds on H100, while loading the cached state from NVMe can be much faster even with software overhead. Reuse determines how often that advantage applies. Multi-turn assistants, RAG with repeated system prompts, agentic loops with tool-call suspensions, and code assistants sharing repository context all offer repeated prefixes. One-shot unique prompts and short-context single-user workloads offer much less opportunity to recover the cost of retrieval.

This distinction also governs the cost-per-token argument. NVMe tiering can avoid expensive repeated prefill, but an I/O path that stalls the GPU can erode that saving. Capacity alone is not the economic test. At high hit rates, KV offloading shifts prefill from compute-bound toward memory-bound execution, changing the binding resource from FLOPS to storage bandwidth. GPU utilization by itself cannot identify why that shift has reduced performance.

How storage tier choices propagate into TTFT, throughput, and GPU utilization

Time to first token, or TTFT, absorbs both sides of the cache decision. A miss needs prefill, while a hit still takes time to retrieve unless the transfer happens during other work. The storage tier therefore affects TTFT through both its exposed latency and its hit rate. py-kvcache shows that scheduler-aware preloading contributes materially to lower TTFT at long context lengths. Starting disk reads while requests remain queued gives the system a chance to overlap retrieval with computation.

Decode throughput depends on a similar distinction between serial and overlapping work. If each required NVMe read waits on the token-generation path, storage delays accumulate directly in generation time. DUAL-BLADE reduces decode latency by overlapping storage I/O with GPU DMA through adaptive pipeline parallelism. Utilization is a lagging indicator of these delays. It can appear healthy until the pipeline stalls, leaving teams to misattribute storage-path delays in TTFT to model size or quantization.

ObjectCache, by Zhu et al., 2026, shows how overlap can hide some of the time spent reaching a more distant tier. On a high-bandwidth RoCE cluster, layerwise retrieval from object storage produces only modest TTFT overhead over local DRAM for long-context KV reuse. Shorter contexts provide less computation behind which to hide transfer, so the overhead grows. Network fabric bandwidth becomes the binding constraint at that tier.

What engineers must characterize to find their actual binding constraint

The practical response is to characterize the complete I/O path under inference conditions. Tests should use the block sizes and concurrency that the serving system works with, because peak device throughput alone leaves out those conditions. Transfer granularity is part of the workload. Intermediate staging and scheduler timing must also be included because either can dominate the latency attributed to storage. A faster device cannot settle a comparison that excludes the rest of the request path.

Per-tier p99 latency under production concurrency reveals delays that averages conceal. Cache hit rate should be separated by workload class so repeated-prefix traffic does not obscure unique-prompt behavior. The break-even prefix length must be measured for the specific GPU and I/O path, establishing where retrieval starts to beat recomputation. GPU utilization over time then helps locate stall events. Together, these measurements distinguish insufficient bandwidth from poor reuse, late scheduling, and transfer overhead.

The KV cache optimization survey by Xu et al., Dell Technologies, 2026, frames this as a problem to address during storage provisioning, before deployment. An inference server needs a storage specification alongside a GPU count. Protocol overhead belongs in that specification too. Standard S3 exposes object-level operations, while runtime KV reuse needs prefix-selected blocks delivered in layer order. ObjectCache tackles that mismatch through protocol-scheduling co-design, showing why transport bandwidth alone tells only part of the story for the object tier.

Engineering approaches that break the latency constraint at each tier

At the CPU–GPU boundary, cutting out intermediate copies that are not needed fixes a different failure mode from adding storage bandwidth. GPUDirect Storage targets that copy overhead, while scheduler-aware preloading moves retrieval earlier in the request lifecycle. Neither makes PCIe bandwidth unlimited.

At the NVMe boundary, py-kvcache combines asynchronous direct I/O, bounded shared staging, and preloading while requests wait. DUAL-BLADE takes a complementary approach to residency and locality. It assigns KV tensors to a page-cache path or an NVMe-direct path according to runtime memory availability. The direct path bypasses the filesystem by mapping tensors to contiguous logical block address regions, while adaptive pipelining overlaps storage I/O with GPU DMA. These mechanisms address the specific failures of page-cache thrashing, fragmented streams, and serialized transfers.

At the object-storage boundary, ObjectCache co-designs the protocol and transfer schedule to deliver KV data in layer-consumption order. Its prototype uses NIXL, Ceph RGW, and DAOS on a high-bandwidth RoCE cluster. Layerwise ordering lets transfer overlap with per-layer GPU computation on RDMA-capable infrastructure. That approach addresses the mismatch between object operations and the sequence in which inference consumes cached state.

Across these tiers, LMCache manages KV block placement and movement for vLLM, SGLang, and NVIDIA Dynamo. Google Cloud’s GKE Inference adoption and CoreWeave/Cohere benchmarks provide concrete deployment and performance anchors, with reported latency reductions on cache-hit-heavy workloads. Placement management complements lower-level I/O engineering. Comparisons among these systems should weigh the measured failure mode each fixes and the workload it serves, with storage pool size as only part of the picture.

What MLPerf Storage v3.0 adds and why KV cache benchmarks change the provisioning conversation

MLCommons released MLPerf Storage v3.0 in September 2026, adding two inference-focused tests and opening the suite to S3 object storage for the first time. Its KV cache test measures read/write operations directly behind LLM inference. That changes the basis of comparison. A suite previously limited to training and checkpointing can now test storage against an inference-specific access pattern, supplementing proxies such as peak device throughput.

Working group co-chair Curtis Anderson said that, as AI context scales reach into the trillions, object-based storage may work as an alternative to filesystem-based storage and may even be preferred. Working group co-chair Brian Belgodere stated that tests separating specific storage uses, including checkpointing, KV caching, and vector databases, “gives stakeholders a much clearer idea of how to engineer and provision AI systems to minimize storage performance bottlenecks”. The emphasis is on decomposing the system. That is the same discipline required to distinguish a PCIe restriction from NVMe software overhead or object-tier scheduling delays.

The earlier MLPerf Storage v2.0 round in August 2025 added checkpointing tests, and the systems tested served roughly twice as many accelerators as those in the v1.0 round. Benchmark-driven provisioning is therefore an iterative specification process. For inference teams, storage selection should now include a reproducible benchmark citation alongside vendor claims. Public benchmark work on code-hosting platforms and technical discussion on community chat platforms give teams places to check and question those results. The durable standard is whether a storage architecture proves its performance on the KV workload it will actually serve.

Sources

  1. DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
  2. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
  3. ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
  4. KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading

More in Distributed Serving