Tiered Storage Layout Decisions That Determine Sustainable Throughput Ceilings in LLM Inference
KV cache placement and eviction policy determine LLM inference throughput at scale.

NVMe and tiered storage determine sustainable LLM inference throughput by governing how KV cache moves across HBM, DRAM, local flash, and remote storage. Once the working set exceeds fast memory, placement policy, eviction granularity, prefetch scheduling, and bandwidth matching join GPU compute in setting the throughput limit. The architecture that reduces latency at scale preserves useful cache hits without letting transfers and writes consume the benefit of avoiding recomputation.
Why the throughput ceiling of an LLM inference system lives in the storage layer, not the GPU
Prefill is compute-bound and writes KV blocks, while decode is memory-bandwidth-bound and reads those blocks repeatedly. A layout that quickly clears GPU memory after prefill can impose repeated transfer costs during decode.
At low concurrency, HBM holds the working set, and serving strategies look equivalent. The distinction emerges as concurrency rises. HBM fills first, and throughput drops sharply without offloading. CPU DRAM eventually saturates too, leaving NVMe or remote storage to preserve cache hits that the nearer tiers cannot retain. At that scale, storage-backed hits can more than double throughput compared with alternatives.
Context length intensifies this pressure because KV cache size grows linearly with it. Longer contexts widen the gap between HBM capacity and the working set, even before concurrency increases. Consequently, peak GPU compute does not describe the sustainable serving rate once data movement becomes the binding constraint. The relevant limit is how much useful KV traffic the hierarchy can sustain while meeting latency requirements.
What the four-tier hierarchy actually looks like in bandwidth and latency terms
GPU HBM provides the highest bandwidth and smallest capacity. H100 SXM5 bandwidth is measured at roughly 3.35 TB/s, while HBM capacity ranges from 40–192 GB depending on SKU, with sub-microsecond access. CPU DRAM has multiple TB of capacity and roughly 200–600 GB/s of native server DDR5 bandwidth. That native rate is not the GPU-accessible rate. Access over PCIe is constrained to roughly 63–128 GB/s, with latency in the tens-to-hundreds of microseconds range.
The PCIe distinction is central to interpreting DRAM specifications. PCIe 5.0 x16 pinned-memory bandwidth is roughly 63 GB/s unidirectionally and roughly 126 GB/s bidirectionally. Neither figure makes host memory equivalent to HBM. The available rate depends on both the bandwidth of the memory holding KV blocks and the path those blocks actually traverse.
Local NVMe SSDs provide roughly 7 GB/s read bandwidth under ideal PCIe 4.0 conditions, sub-millisecond to low-millisecond latency, and tens of TB of capacity. Remote storage, including object stores and distributed KV pools, provides one to tens of GB/s of effective bandwidth depending on the fabric, with effectively unbounded capacity. These capacity extensions have distinct operating envelopes. Each boundary exchanges faster access for more room, making a cache hit’s location inseparable from its serving cost.
How placement policy determines which blocks cross which tier boundary
Placement policy sets where a newly written block goes and which boundary a later read must cross. Recency is the dominant heuristic. Active-generation blocks remain in HBM, completed conversation blocks likely to recur move to DRAM, and cold prefixes move to NVMe. vLLM treats its host primary tier as a proper LRU/ARC cache rather than a staging buffer, so chunks serve hits directly from DRAM until capacity forces eviction.
That distinction separates retained cache capacity from temporary transfer space. vLLM routes KV data through CPU DRAM first, releases GPU memory after the local PCIe copy completes, and writes to secondary tiers asynchronously. Its just-in-time allocation pattern holds accelerator memory only while actively needed. Content-addressed naming then maps identical token sequences to the same key, so matching prefixes share cached data automatically.
Recency alone says little about whether a block will be reused enough to justify a far-tier write. Reusable and transient KV need different treatment. Shared system prompts, common prefixes, and agent workflow history benefit from durable, shared placement because another node may fetch them. Single-turn, non-reusable KV incurs write cost without a corresponding read benefit.
Placement therefore calls for classifying reuse probability at admission. TTKV extends that reasoning by assigning blocks according to temporal proximity and inferred importance. Its temporal-tier approach shows that some KV states can be easier to access than others, but temporal ranking still has to account for how the workload actually reuses them.
Why eviction granularity sets the write-amplification floor and constrains NVMe endurance
The HBF characterization paper finds a counterintuitive result. Replacing SSD storage with faster High-Bandwidth Flash under a standard Mooncake-style KV-offloading runtime degrades end-to-end latency and reduces maximum SLO goodput. Writes outnumber reads on every production trace examined. The paper’s thermal model also shows that the flash stack reaches its thermal limit well below peak bandwidth, so a faster read path alone cannot handle the demands of a write-heavy workload.
Eviction granularity helps determine how that write pressure develops. Coarse eviction moves blocks together, including data that will never be reused. Fine eviction avoids some unnecessary movement but increases I/O operations and metadata overhead. The trade-off between small and large requests involves both how much data gets reused and how efficiently it moves.
vLLM exposes this trade-off through its configurable blocks_per_chunk parameter. Larger chunks produce fewer, larger operations to host and secondary tiers, improving storage throughput. Smaller chunks preserve finer-grained reuse and reduce over-fetch. A headline read-bandwidth figure says little about which choice keeps writes at a rate the storage can sustain.
The engineering response requires reuse-aware placement with write budgeting and thermal coordination. DASH combines workload-aware placement, early expert determination, and prefill/decode-aware KV write scheduling to manage write volume within the serving schedule. For NVMe deployments, plans for endurance and replacement must account for this write load because the write path or endurance budget can reach its limit before rated read bandwidth is used up.
How prefetch scheduling determines whether tier latency is hidden or exposed on the critical path
ObjectCache co-designs its storage protocol and transfer schedule so that storage delivers KV data in the order the GPU consumes it. Delivery proceeds layer by layer, overlapping transfers with per-layer computation. Arrival order is part of the performance mechanism. A correctly placed block still imposes a throughput tax if it arrives after computation needs it.
The same paper shows that long contexts allow layerwise overlap to add only a small latency overhead over local DRAM. Short contexts have less computation to hide transfers behind, so their overhead rises well above the optimal local layerwise baseline. The contexts that are hardest to fit in memory also provide more computation with which to mask retrieval.
Scheduling also determines how shared bandwidth translates into time to first token, or TTFT. ObjectCache reduces added TTFT by 1.2–1.8× compared with equal bandwidth sharing. That gain comes from how bandwidth is shared and when transfers run, with no increase in the raw link rate. Prefetch lead time must also reflect the source tier, since object storage and local NVMe do not present the same retrieval delay.
Disaggregated prefill/decode serving adds storage-network load imbalance to this timing problem. Explicit dual-path or workload-aware scheduling addresses that deployment condition.
Why bandwidth matching across tier boundaries is the condition that makes the other decisions coherent
Placement, eviction, and prefetch all change the traffic presented to each boundary. A placement policy may keep more reusable blocks but need more remote transfers, while a larger eviction chunk may make operations more efficient but move more unused data. Throughput can hold up only if that traffic fits through the narrowest boundary, which can shift as those choices change.
TopKV makes transport selection depend on the interconnect graph at startup. It uses NVLink for same-domain transfers, PCIe for same-node cross-domain transfers, RDMA across nodes, and TCP as a fallback. At roughly 900 GB/s, NVLink completes the cited transfer in 2.9 ms, an 18× improvement over TCP. A topology-blind scheduler misses that speed gain even when it has a faster path available.
FP8 KV cache halves the volume relative to BF16, reducing demand on HBM, PCIe, and NVMe. AMD’s MLPerf v6.1 submission used FP8 KV cache with MXFP4 expert weights on MI355X GPUs. This establishes mixed-precision KV quantization as both a production benchmark strategy and a storage-capacity technique.
Inter-node transport also constrains reuse. RDMA and GPUDirect RDMA are point-to-point, so reuse across workers requires retransmission and schedulers must track per-node KV locality.
What routing and gateway-layer coordination add when KV cache spans multiple nodes
At cluster scale, routing a request to reusable KV avoids the full prefill recomputation required otherwise. Cache location becomes a scheduling input. A routing decision that ignores it can discard the benefit of careful placement, while routing toward a cache still has to respect the bandwidth needed to retrieve it. This is the cluster-scale expression of the same storage constraint.
ObjectCache changes that relationship by placing immutable prefix blocks in a shared S3-compatible tier. The blocks are content-addressed by prefix hash and reusable across users, sessions, and compute nodes. Prefill and decode workers become largely stateless with respect to reusable prefixes. Routing therefore no longer needs per-node KV locality tracking for those prefixes.
The Unified AI Gateway framework extends coordination to model routing, KV storage, prefill, and cache transformation. It treats reuse, compression, cross-model mapping, and distributed storage as coordinated capability groups so that a model or provider switch does not silently discard useful state. The framework reports modeled TTFT speedups of 1.25×–13.28× and input-cost benefits of 1.20×–6.16× across eight workload profiles. Those ranges vary with model-switching probability and reuse structure, so the gains differ across serving workloads.
RDMA retains the need for per-node locality indices, whereas CXL’s write-once-read-many model removes that requirement but introduces different deployment constraints at scale.
A decision framework for mapping workload profile to layout choices at each tier boundary
The decision map begins with context length distribution, prefix reuse rate, concurrency, and the latency SLO. These variables identify which boundary needs relief. For short-context, low-reuse workloads at low concurrency, HBM alone or a shallow DRAM tier can be sufficient. NVMe adds I/O latency without a capacity benefit when GPU KV capacity is not the bottleneck, leaving eviction and prefetch choices with little throughput impact.
Long-context, high-reuse workloads, including RAG, agentic, and multi-turn serving, require the full hierarchy. Placement then depends on reuse probability at admission, while immutable, content-addressed prefixes make object storage viable. Layerwise prefetch determines how much retrieval latency becomes visible. Precision quantization becomes necessary to keep block volume manageable across the hierarchy, rather than serving as an isolated device optimization.
High-concurrency, mixed workloads shift attention toward PCIe bandwidth matching. Topology-aware transport selection becomes the most consequential scheduling choice, while high eviction rates make write amplification and flash endurance part of capacity planning. CXL becomes worth evaluating when per-node locality tracking constrains the scheduler. In disaggregated deployments, gateway coordination and dual-path or workload-aware scheduling address the additional routing and storage-network demands.
NVMe’s effect on cost per token depends on these conditions as well as on its capacity. Useful cache hits avoid recomputation, but unnecessary writes, exposed retrieval latency, and replacement costs work against that benefit. Whether selective KV retention or offloading is the primary path for extreme-length contexts remains unresolved. TTKV and OasisKV point toward hybrid approaches when most evicted blocks are never read again, leaving the workload’s reuse distribution as the decisive input.
Sources
- HBF Sucks? A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving
- Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management
- ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
- TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
- OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
- Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
- An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age


