Storage Bandwidth Saturation Points That Collapse LLM Inference SLAs at Scale

Memory bandwidth becomes the binding constraint for long-context LLM inference at scale.

· 9 min read
Cover illustration for “Storage Bandwidth Saturation Points That Collapse LLM Inference SLAs at Scale”
Distributed Serving · October 26, 2026 · 9 min read · 1,955 words

NVMe and tiered storage improve LLM inference throughput only when KV cache movement stays within the bandwidth budget of every tier. Storage bandwidth saturation is a threshold event. Once demand exceeds HBM, PCIe, NVMe, or network bandwidth, serialized decode work faces non-linear latency growth, so capacity expansion alone cannot protect inference SLAs.

Why decode-phase attention is memory-bandwidth-bound, not compute-bound

During autoregressive decode, attention must read the entire KV cache for every token generated. Each step therefore depends on both how quickly memory supplies cached keys and values and how quickly the GPU performs arithmetic. Arithmetic intensity stays nearly the same as batch size or context length grows, while KV cache traffic gets closer to the DRAM-bandwidth limit. Compute can remain underutilized at that limit.

An H100 delivers roughly 3 TB/s of HBM bandwidth, yet reading the cache produced by a single long-context session already consumes a significant fraction of a decode step’s budget. Other memory traffic competes for that same bandwidth. Longer contexts increase the cache footprint linearly, making every subsequent token more expensive to serve from memory. Faster arithmetic does not remove those reads. Because peak compute has historically scaled faster than DRAM and interconnect bandwidth, future accelerators do not inherently resolve this imbalance without substantially higher HBM bandwidth or reduced KV cache traffic.

How context length and concurrency turn a bandwidth pressure into a memory-wall collision

As context length grows, each session keeps more state, while concurrency multiplies the number of states kept in memory. The memory formula makes that compounding visible as 2 × batch size × sequence length × number of layers × hidden size × precision bytes. In the long-context example, eight concurrent users require roughly 320 GB, four times an H100’s entire HBM. Capacity runs out before serving demand does. Cold blocks must leave the GPU even if its arithmetic units still have room for more work.

Fragmentation makes the raw capacity arithmetic worse. Traditional allocation wastes a large majority of KV cache memory, and PagedAttention reduces that waste to under 4%. But even with more efficient allocation, the state still takes up memory. A large model at scale can still consume more memory for KV cache than for model weights, so buying a larger GPU only postpones the collision.

Agentic workloads bring it forward through repeated reuse of long histories. The DualPath paper documents DeepSeek-V3.2 coding workloads with mean contexts of tens of thousands of tokens but individual appends of only hundreds. With high cache hit rates, the workload spends more of its time on I/O than on compute. An agent that resumes after a tool call must also pay the full prefill cost again unless its cache survives. Once HBM cannot retain that state and recomputation becomes untenable, the engineering response is a tiered storage hierarchy.

How NVMe tiering affects LLM inference latency at each storage boundary

Diagram: Storage Tier Bandwidth and Latency: A Four-Level Hierarchy. Visualizes: Show the four tiers of KV cache storage as a ranked hierarchy contrasting bandwidth and latency.

GPU HBM offers roughly 3.35 TB/s with sub-microsecond latency, while CPU DRAM reached through PCIe offers roughly 63 GB/s with 10–100 µs latency. Local NVMe provides roughly 7 GB/s and 100 µs–1 ms per block access. Network storage adds fabric traversal to device latency, with performance dependent on the fabric. These boundaries are not interchangeable extensions of GPU memory.

NVMe is particularly hazardous under a tight p99 SLA. Its access latency alone is in the hundreds of microseconds per miss, so fetching one cold block can use up a tight enough latency budget before decode begins. Decode steps are serialized. Consequently, a miss on the critical path exposes the tier’s access penalty directly, while bandwidth saturation adds contention that turns a capacity solution into non-linear tail-latency growth.

Disaggregation exposes the same trade-off through time to first token, or TTFT. The DualPath paper notes that NIXL KV transfer latency for a short prompt on a large FP8 model can reach tens of milliseconds, exceeding the compute savings from separating prefill and decode. NVIDIA Dynamo disaggregated serving wins above roughly a few dozen GPUs at long context or high concurrency, but loses below roughly 16 GPUs. The Multi-Tier Dynamic Storage framework uses dynamic storage access control and adaptive hierarchical eviction to handle these boundaries, because static eviction cannot deal with both bandwidth contention and capacity overhead.

Where saturation concentrates in disaggregated architectures and why it is asymmetric

In disaggregated prefill/decode serving, storage pressure does not spread evenly across the system. Prefill engines load large KV cache volumes from remote storage layer by layer, saturating their storage NICs. Decode engines receive that state from prefill through RDMA, leaving their own storage NICs largely idle. Aggregate bandwidth therefore conceals the actual bottleneck. The DualPath paper documents a cache-compute ratio of approximately 22 GB per PFLOP for DeepSeek-V3.2, making storage bandwidth the binding resource in these agentic workloads.

Transfer granularity compounds this imbalance before nominal link bandwidth is exhausted. Popular inference engines use cache pages of roughly 20 KB–63 KB for models including Llama, Qwen, and GPT-OSS. On a two-node setup with 400 Gbps NICs, single WRITE operations need transfers of at least 16 MB to saturate available bandwidth, although paged WRITE operations can do so at 32–64 KB. Operation choice changes the usable bandwidth. So adding bandwidth on the prefill side leaves some of these problems in place and is costly in general-purpose clusters.

Mooncake addresses cache availability through a distributed DRAM pool and an affinity-aware scheduler that raises hit rates. That approach has a capacity and cost boundary of its own. It cannot operate in memory-constrained scenarios such as RL rollout when offloaded training state occupies DRAM, and DRAM’s premium over SSD makes very large working sets uneconomical. DualPath instead targets the unused storage bandwidth on the decode side, treating the imbalance as an architectural defect rather than simply a shortage of provisioned resources.

What DualPath and NIXL reveal about bandwidth utilization as a design variable

DualPath, developed by Peking University, Tsinghua University, and DeepSeek-AI in 2026, adds a second route for loading KV cache. Beyond storage-to-prefill transfers, it loads cache into decode engines and forwards it to prefill through RDMA over the compute network. Previously idle storage NICs become useful resources. Its optimized path avoids network congestion and interference with latency-critical model execution communications, while a global scheduler balances load across prefill and decode engines.

NIXL transfer behavior demonstrates the complementary constraint that moving state has a cost even when the destination has spare capacity. Because Dynamo’s disaggregation trade-off changes with scale, designs must account for that cost even when aggregate throughput hides it. An architecture that gains throughput at long context or high concurrency does not automatically improve short-prompt TTFT.

LMCache demonstrates another route to better utilization through chunk-size optimization and zero-copy operations, which can multiply effective loading bandwidth without an architectural redesign. These mechanisms address different sources of waste. DualPath redistributes traffic across engines, while chunk sizing and zero-copy operations improve the efficiency of transfers themselves.

Lossless compression as a bandwidth multiplier for BF16 and FP8 KV caches

Compression addresses the other side of the bandwidth equation by reducing the bytes that must move. BF16 and FP8 KV cache tensors contain statistical structure in their exponent bits that lossless coding can exploit. Layer-wise compression separates exponent and mantissa bits, then codes the exponent stream. The opportunity is specific to the data format. FP8 exponents remain skewed across time steps, especially in early tokens, while BF16’s wider exponent permits greater compressibility.

ZipNN demonstrates that lossless compression of BF16 model families can reduce transferred data by roughly a third in common cases and by more than half in favorable ones, with bit-exact numerical precision. Those results are for model data and leave open how much compression a given KV cache can achieve. ZipServ applies the principle to GDDR memory-based GPUs where memory bandwidth is the primary bottleneck. Its ASPLOS 2026 results include up to 30% model size reduction, up to 2.21× kernel-level speedup over cuBLAS, and average end-to-end speedup over vLLM.

FP8 leaves less absolute redundancy within its 8-bit representation. Lossless compression is therefore an incremental bandwidth gain on top of quantization. NVIDIA Blackwell and Hopper support FP8 natively in tensor cores, making it a practical production choice for most serving workloads. BF16 is still the standard for agentic applications and fine-tuned domain-specific models where errors build on one another across reasoning steps. For those workloads, lossless compression offers the larger bandwidth multiplier without changing numerical precision.

What MLPerf Storage v3.0 benchmarking reveals about production KV cache I/O requirements

MLCommons released MLPerf Storage v3.0 in September 2026 with KV cache and vector database workload tests, plus an S3 data layer in addition to the existing POSIX layer. KV cache I/O is now a distinct benchmark category. Capacity and read-heavy specifications alone do not characterize a serving system that continuously creates, preserves, and retrieves session state.

The benchmark’s top submission in the 1.25T parameter model KV cache category delivered 877.52 GiB/s write bandwidth and 588.28 GiB/s read bandwidth using 30 data nodes for a large simulated accelerator fleet. These figures provide a calibration point for sustained throughput under the benchmark workload. The write-read asymmetry is operationally important. KV cache workloads are write-heavy compared with model-weight workloads, so a specification focused on reads leaves out a large part of what serving needs.

The lesson for operators is to benchmark KV-cache-specific access patterns, including sequential large writes followed by random reads at page granularity. Peak burst bandwidth is not enough. Testing how well a tier keeps up with that mix shows where it reaches saturation as cache demand grows. The S3 layer also treats network storage as a production cache tier, so its performance needs to be tested as closely as local NVMe before remote capacity can be treated as ready to use.

What operators must instrument and architect to detect saturation before SLAs collapse

Operators must measure bandwidth at each tier boundary rather than infer storage health from GPU compute utilization. HBM bandwidth utilization belongs at decode-step granularity, because sustained operation near its limit leaves little room for longer contexts or more sessions. Operators need to see PCIe utilization for each transfer event as well as its average over time. Short bursts can disappear inside an average. For NVMe, sustained write bandwidth and write amplification show how continuous cache traffic puts pressure on bandwidth and risks wearing out the drives.

Cache hit rate must also be separated by tier. When the DRAM hit rate falls without a compute spike, cold blocks are falling through to NVMe, which gives an early warning that p99 is getting worse. Disaggregated systems additionally need storage NIC utilization by engine type. System-wide totals hide the prefill/decode imbalance. DualPath reports up to 1.87× offline inference throughput improvement and an average 1.96× online serving throughput improvement without SLO violation across three models on production agentic workloads, quantifying the value of recovering that stranded bandwidth.

Provisioning 1–2 TB of system memory on an 8-GPU inference node is defensible in 2026, with CPU DRAM extending effective VRAM at lower cost per byte. For NVMe, operators should choose several datacenter-class drives with high drive writes per day over a single high-capacity consumer drive. Endurance can fail before bandwidth does. This makes cost per token a tiering question, although the performance evidence does not establish a numerical cost-per-token saving.

At network boundaries, software aggregation or prefetching should reconcile inference-page sizes with efficient transfer units. Lossless BF16 compression belongs at the constrained boundary, whether GPU-to-DRAM for hot offload or DRAM-to-NVMe for warm cache, as well as at rest. Sufficiently scaled disaggregated deployments should use decode-side storage bandwidth as well as prefill-side bandwidth. Eviction must respond to bandwidth pressure as well as capacity. The MTDS framework lets eviction adapt in this way, as part of a design guided by KV access patterns, high-endurance NVMe, and RDMA-capable fabric.

Sources

  1. KV CACHE OPTIMIZATION STRATEGIES FOR SCALABLE AND EFFICIENT LLM INFERENCE
  2. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
  3. Multi-tier dynamic storage of KV cache for LLM inference under resource-constrained conditions
  4. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  5. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

More in Distributed Serving