NVMe Tier Bandwidth Accounting as the Primary Driver of Cost per Token in LLM Serving

NVMe bandwidth per token generated becomes the measure that governs LLM serving costs.

· 9 min read
Cover illustration for “NVMe Tier Bandwidth Accounting as the Primary Driver of Cost per Token in LLM Serving”
Distributed Serving · November 5, 2026 · 9 min read · 1,921 words

When KV cache spills to NVMe, LLM inference throughput and cost per token depend on the bytes moved through the storage path for each token generated. A three-tier HBM, DRAM and NVMe architecture expands usable cache capacity, but its economics depend on bandwidth, reuse and compression. Treating NVMe bytes moved per token as a first-class accounting variable makes those costs measurable and exposes where optimization will pay off.

Why decode-phase memory access determines cost per token before any other variable

Transformer decode is memory-bandwidth-bound rather than compute-bound in the serving regime considered here. The GPU waits for KV data instead of spending its time executing matrix multiplications. The KV cache creates this trade-off. It stores key and value tensors for previous tokens so the model avoids recomputation, exchanging compute work for memory traffic on each decode step.

The cache footprint depends on the model and how it is served. Its size is 2 × num_layers × num_kv_heads × head_dim × bytes_per_element × sequence length × batch size. The model architecture sets the layer count, KV head count and head dimension, while the choices made for serving set the precision, sequence length and batch size. Where that data is kept sets how quickly it reaches the GPU. HBM delivers TB/s, CPU DRAM accessed over PCIe 5.0 x16 delivers roughly 63 GB/s, and a single PCIe Gen4/Gen5 NVMe drive tops out at 7–14 GB/s.

How context length and concurrency turn the bandwidth gap into a cost equation

KV cache demand scales linearly with context length and concurrent users. Doubling either doubles the cache bytes involved in a decode step, although tier residency determines how much traffic reaches NVMe. Batch size therefore affects more than compute utilization. It also sets the aggregate cache footprint that the memory hierarchy must accommodate, while longer contexts increase the footprint of each sequence.

A single user’s BF16 KV cache can reach hundreds of gigabytes, exceeding the full HBM of any current GPU by a substantial multiple. KV memory can also outgrow parameter memory and dominate GPU VRAM. Once required cache blocks reside on NVMe, their loading rate can limit tokens per second before GPU compute does.

The cache-compute ratio expresses this pressure against a resource serving teams already track. DeepSeek-V3.2 reaches approximately 22 GB/PFLOP in agentic workloads, quantifying the cache traffic associated with compute demand. Agentic inference has shown severe GPU underutilization traced to KV-cache loading speed. More compute does not resolve that bottleneck. When serving costs stay the same over time, completing fewer tokens raises cost per token, so cache bandwidth is part of the cost equation.

Modeling bytes-moved per token at the NVMe tier: the accounting unit engineers need

The base unit is the KV data stored for one cached token. Bytes per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element. Multiplying that quantity by the number of tokens in the cold tier gives the uncompressed cold-cache volume associated with a fetch. Dividing by tokens generated in the step gives the normalized accounting expression, NVMe bytes moved per generated token = bytes per cached token × cold-tier token count / tokens generated this step. The denominator must remain explicit.

That expression describes the cold data loaded in the accounting interval. With a fully cold prefix, the entire prefix KV block must be loaded before the first decode token is emitted. This initial fetch can dominate I/O. The prefill-side load is a large sequential read governed mainly by sustained bandwidth, while decode adds incremental write-back of newly computed KV blocks as the context grows.

The accounting must count traffic in both directions. PagedAttention writes KV blocks in page-sized units, producing small-block random writes mixed with large sequential reads. Write amplification and queue depth affect how much of the write budget is used, apart from how well sequential reads perform. Across a measurement window, total NVMe bytes moved divided by generated tokens captures both costs.

Loading a large cache from NVMe can be faster than recomputing a long prefix on an H100. The crossover comes when the storage path can transfer the required data in less time than recomputation takes to do the same work. Cache capacity alone cannot establish that advantage.

How the three-tier architecture sets the bandwidth budget available to serve the accounting model

The emerging production architecture assigns hot KV blocks to GPU HBM, warm blocks to CPU DRAM and cold blocks to NVMe SSDs. Its representative rates are TB/s for HBM, approximately 63 GB/s through PCIe 5.0 for DRAM access, and approximately 7 GB/s with approximately 100 µs latency for NVMe. Each promotion path has a finite budget. Tiering reduces pressure on expensive GPU memory only if required blocks can be promoted within the serving latency budget.

NVIDIA’s Inference Context Memory Storage Platform, branded CMX and announced at CES 2026, standardizes KV movement across these tiers by access recency. ICMSP offloads I/O scheduling to BlueField-4 DPUs with 64 Arm cores and 800G ConnectX-9 networking, avoiding host CPU interruptions from NVMe completions at high concurrency. NIXL handles cluster KV movement across NVLink, InfiniBand, RoCE, Ethernet/TCP and storage backends including NVMe-oF and S3. NVIDIA’s GTC 2026 configuration-level claim rates the STX reference design for up to 5× token throughput and 4× energy efficiency versus traditional storage configurations.

LMCache provides software-managed KV tiering across CPU memory, local storage and remote backends without requiring a DPU. Its accounting includes host-side overhead as CPU interrupts accumulate at high concurrency. DualPath, described in an arXiv 2026 paper, addresses a different limit in disaggregated inference. It redistributes storage network load that otherwise saturates prefill engines while leaving decode-engine storage bandwidth idle, achieving up to 1.87× offline inference throughput.

MiMo-V2.5’s GCache architecture, introduced in 2026, instead repurposes memory and built-in NVMe SSDs on Prefill and Decode GPU machines at zero additional storage cost. GCache reports system-path measurements of single-process RDMA reads of 170 GB/s at 280 µs latency with 1 MB I/O sizes, and approximately 350 GB/s under GDR scenarios. They show how much data the hardware already in place can transfer within the budget.

Compression as a multiplier on effective NVMe bandwidth: how quantization and lossless encoding reduce bytes-moved per token

Compression changes the demand side of the equation. In vLLM, --kv-cache-dtype fp8 reduces KV block size by approximately half before storage. vLLM’s benchmarks report sizes as low as 54% of BF16 in the best cases, roughly doubling effective KV storage capacity and halving I/O bandwidth requirements. The drive has not become faster. Each transfer now represents more cached tokens.

FP8 KV cache plus FP8 attention produces sub-point aggregate score differences for some models, including Qwen3 reasoning tasks, while other models show 1–2 point degradation. Long-context retrieval exposes a sharper failure mode. Needle-in-a-haystack scores fell markedly because of imprecise accumulation in Hopper FP8 Tensor Cores past a large contraction dimension. Two-level accumulation brought results back to near-BF16 levels, while the B200 works without that fix.

Lossless encoding can reduce the remaining bytes further. Speculative KV coding, introduced in 2026, identifies bytewise entropy below the raw BF16 representation, leaving room for compression without discarding information. Lossless compression can achieve up to roughly 4× compression on top of FP8 quantization, yielding 6–8× total compression relative to the original BF16 cache. The lossless layer adds no accuracy trade-off, but it does not remove any introduced by quantization.

Prefix reuse reduces how often those compressed bytes need to move. SGLang RadixAttention achieves very high cache reuse on few-shot workloads and substantial throughput gains on prefix-heavy traffic. When reuse avoids a cold fetch, it lowers NVMe demand.

What the resulting cost structure looks like in a production serving system

ICMSP-enabled NVMe tiering on an H100 SXM5 serving a large model at long context lowers cost per million tokens when storage offload enables a move from one user to many concurrent users per GPU with unchanged compute capability. Cache offload expands feasible concurrency, while sufficient bandwidth lets that concurrency translate into completed tokens.

Capacity and throughput must be assessed together. An offload configuration that accommodates more contexts but cannot load them quickly enough does not deliver the same economic result. Bytes moved per token links the two by showing how much storage traffic is needed for each unit of output. Together with the bandwidth the path can sustain, it sets a limit on the token rate that storage can support.

Device selection changes the cost of meeting that budget. The D7-PS1010 generally outperforms the D5-P5336 with fewer devices because of its PCIe Gen5 bandwidth advantage. The latter requires more devices to match performance. This comparison ties drive bandwidth to the hardware count needed to satisfy demand, rather than treating storage capacity as the only purchasing criterion.

MLPerf Storage v3.0, released September 1, 2026, formalizes part of this measurement problem by adding KV cache workload tests. The benchmark also adds an S3 data layer along with tests for POSIX and vector database workloads. Its KV workload gives engineers a standard way to compare storage under LLM-serving I/O patterns, so they can measure the bandwidth budget they use in their designs.

The strongest objection: when PCIe interconnect bandwidth, not NVMe media, is the actual binding constraint

A single PCIe 5.0 link at 64 GB/s can be narrower than a modest NVMe array’s aggregate read bandwidth. In that configuration, faster drives do not remove the bottleneck. The CPU–GPU interconnect limits delivery, so attributing the constraint solely to NVMe media would direct investment at the wrong component. This is a substantive limit on any drive-only cost model.

It does not invalidate bytes-moved accounting. The same bytes must traverse a path whose narrowest segment determines usable bandwidth, whether that segment is the drive, interconnect or network. ICMSP’s DPU architecture and GDR pathways address movement beyond the media itself. cuFile bypasses CPU staging, while direct NVMe-to-GPU paths avoid the conventional host-staged route, though direct transfer should not be confused with unlimited interconnect bandwidth.

DualPath exposes the related network case. Idle decode-side storage bandwidth cannot help saturated prefill-side links without scheduling that redistributes the load. The accounting unit remains unchanged, but the remedy shifts from device bandwidth to path utilization. Instrumenting bytes moved per token makes these constraints visible as faster media moves the bottleneck elsewhere.

Treating NVMe tier bandwidth as a first-class metric in production observability

Engineers should record NVMe reads and writes per request separately, distinguishing cold-prefix fetches from decode write-back. Total bytes moved divided by tokens generated in the same window gives the normalized metric. Keeping the traffic breakdown prevents a large initial read from being mistaken for recurring decode demand.

Peak-concurrency measurements should include NVMe queue depth and latency percentiles, together with hit rates in HBM, DRAM and NVMe. Queue and latency behavior helps distinguish media saturation from interconnect limits. Tier hit rates show how reuse changes the amount of data in a cold fetch.

Compression changes are testable against that baseline. Engineers can compare NVMe traffic before and after enabling FP8 quantization, NVFP4 on Blackwell or lossless encoding, testing the predicted 2× or 4× reductions where applicable. MLPerf’s KV workload provides a reference for testing whether the configuration sustains the throughput predicted under realistic I/O patterns. Measured reductions, rather than nominal compression settings, belong in cost attribution.

Gen4 versus Gen5 NVMe, CPU-staged versus direct-to-GPU I/O, and software-managed versus DPU-offloaded tiering affect the cost of serving the measured byte demand. These choices need not change the logical KV payload to change attainable throughput. Production observability must therefore expose both bytes moved per token and the bandwidth available to move them, treating storage as a first-class engineering problem.

More in Distributed Serving