BenchmarkingLong read

Benchmarking NVMe Sequential and Random Read Performance for KV Cache Patterns

How to test NVMe drives against real KV cache offload I/O.

· 11 min read · Updated
Cover illustration for “Benchmarking NVMe Sequential and Random Read Performance for KV Cache Patterns”
Benchmarking · September 30, 2026 · 11 min read · 2,488 words

KV cache offload workloads produce read and write patterns that no standard storage benchmark was built to test, and the gap between what a spec sheet promises and what an inference server actually gets matters a great deal. Every attention layer in a transformer keeps a running record of keys and values for each token processed so far, and generating the next token means reading that record back rather than recomputing it from scratch. The size of that record is not a vague estimate; it follows a fixed formula: 2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × bytes_per_element. Running the numbers for Llama 3.1 70B at BF16 precision with a 128,000-token context and a single user puts the cache at roughly 42.9 GB Spheron Blog Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serv… AnchorKV: Anchor-Residual KV Cache Compression. Eight concurrent users on that same model push the requirement to around 320 GB, several times more than an H100's entire pool of high-bandwidth memory Spheron Blog Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serv… AnchorKV: Anchor-Residual KV Cache Compression.

That is not a research curiosity. Context windows of 128K and even 1M tokens are already in production, and in-GPU storage for multiple concurrent users is physically impossible on a single GPU at these lengths Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serv…. A tiered approach works: spill cold KV blocks to DRAM, then to NVMe, and reload on demand. Once that happens, NVMe is no longer a passive storage tier sitting quietly at the bottom of the stack. NVMe now is on the critical path of token generation, and it performs well only if it was tested with the right workload, not with a standard storage benchmark.

Eviction policy, prefix reuse, and batch concurrency: each shapes a different read signature

Three variables decide what the storage layer actually sees, and none of them behave independently of the others. Eviction policy comes first: a reactive, LRU-style scheme throws out blocks based on recency alone. State that could have stayed resident often gets discarded and then has to be re-read from NVMe minutes or seconds later. The eviction trigger, in effect, decides which blocks land on the drive in the first place and how often the system goes back for them.

Prefix reuse changes the equation in a different direction. When identical prompt prefixes show up across requests, prefix caching produces cache hits, reducing prefill cost (time to first token); a 75% hit rate at 64K sequence length shifts the I/O mix heavily toward reads of shared blocks Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serv…. But that benefit comes at a storage cost: instead of a simple write-then-discard pattern, the system now holds blocks and reads them repeatedly. The Tutti paper's benchmark measured a 75% prefix cache hit rate at 64K sequence length, and at that hit rate the I/O mix tilts heavily toward reads of shared blocks rather than fresh writes Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serv….

Batch concurrency is the third variable, breaking the tidy mental picture of "one request, one sequential stream." Many requests run at once, each maintaining its own KV blocks, and at high concurrency the drive doesn't see one clean stream of sequential reads. It sees many streams landing on it at the same time, interleaved.

Agentic workloads add a wrinkle that's easy to miss. Research on Leyline (arXiv:2606.01065) documents that agentic LLMs break the monotonic-append assumption (failed tool calls, trajectory pivots, and policy-driven edits produce mid-stream cache mutations, not just appends, generating reads and re-reads of non-contiguous spans). Failed tool calls, changes in reasoning trajectory, and policy-driven edits to earlier steps all produce mutations in the middle of the cache, not just appends, and that generates reads and re-reads of spans that aren't contiguous. When these variables are put together, the conclusion is straightforward: a benchmark that only measures sequential throughput, or only measures 4K random IOPS, will match none of these real patterns precisely, and the meaningful test must reflect the specific mix produced by the serving policy in use. The test has to reflect the specific mix the serving policy in front of it actually produces.

The logical-vs-physical sequentiality gap that makes KV cache I/O look sequential from the application but random at the device

That description is accurate, and it's also incomplete, because the device underneath doesn't see tensors. It sees logical block addresses, and the LBA stream diverges from that clean, tensor-level sequentiality the moment more than one request is in flight.

Prefill writes do stay strictly increasing, which is about as ordered as an access pattern gets. Decode reads don't. Each concurrent request produces its own internally sequential sub-stream, but those sub-streams arrive at the drive interleaved with each other, not one after another. The block-size mechanics documented for the Samsung PM9A3 make this concrete: a 33 MB KV cache block file gets segmented by the Linux block layer into sub-requests, mostly landing around 2.8 MB and 6.1 MB, and those sub-requests get chopped further by the device's maximum data transfer size into MDTS-aligned commands Scaling AI Inference with KV Cache Offloading White Paper AnchorKV: Anchor-Residual KV Cache Compression. So a single logical KV cache block turns into a cascade of smaller commands before it ever reaches flash.

The drive, in practice, receives several overlapping sequential-read sequences at once, and whether it holds onto its rated throughput or slides toward random-read performance comes down to whether its controller can detect and track sequential access across those interleaved streams. A single-stream sequential benchmark will always report sequentiality, because there's only one stream for the controller to track. A multi-stream concurrent test tells you whether the controller can still find the pattern once several requests are competing for its attention at once.

The sub-4K access size problem that standard NVMe benchmarks never stress

Enterprise SSDs have spent roughly a decade being tuned around 4KB random reads, because that matched what databases and virtualized workloads needed. KV cache doesn't ask for that size. Embedding lookups run in the hundreds of bytes, and KV cache blocks themselves often come in well under a kilobyte depending on configuration.

That mismatch has a real cost.

NVIDIA has taken this seriously enough to build a program around it. The Storage-Next initiative, shown publicly at GTC 2025, states its goal: maximize 512-byte IOPS per GPU, within power and tail-latency limits. The choice of metric deserves scrutiny. NVIDIA is measuring 512-byte IOPS, not bandwidth, as the telling choice of metric for this initiative, distinct from the ICMSP/CMX platform, which tracks tokens-per-second and power efficiency instead. That's a tell. If the industry's own hardware vendor picks small-IOPS as the metric to build a program around, testing only 4K random IOPS and calling it done will produce a number that flatters the drive far beyond what it delivers in production. A benchmark that reflects the specific mix produced by the serving policy in use must be trusted over one that tests only sequential throughput or only 4K random IOPS. Most controllers do roughly the same work to serve a 512-byte read as a 4KB one, so sub-4K requests run at 4K cost, imposing up to 8× read amplification and wasting bandwidth and controller cache.

What measured NVMe performance numbers look like under KV cache workloads

Vendor numbers in this space are not small. Memblaze's PBlaze7 claims similar territory: up to 14 GB/s sequential read and 3.3 million 4K random read IOPS, which Memblaze describes as more than double what a typical PCIe 4.0 drive delivers, enough to load KV cache for many inference sessions at once Scaling AI Inference with KV Cache Offloading White Paper KV Cache Meets NVMe: The Key to Accelerating LLM Inference.

Rated numbers and measured numbers diverge, though, and the Samsung PM9A3 shows the gap clearly Scaling AI Inference with KV Cache Offloading White Paper AnchorKV: Anchor-Residual KV Cache Compression. Rated at 6.5 GB/s sequential read and 580,000 random reads per second, the drive measured in benchmark testing came in at 5.5 GB/s sequential and up to 328,000 random reads per second at queue depth 32 Spheron Blog SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cach…. That's not a failure of the drive; it's a reminder that spec-sheet figures assume ideal conditions that KV cache traffic rarely provides.

The Tutti paper's own test system used two SSDs together, reaching a peak of 29 GB/s read and 12 GB/s write, evaluated against Llama3-8B at 64K sequence length with that same 75% prefix cache hit rate mentioned earlier. That hit rate detail isn't incidental. It changes the read-versus-write balance the drives actually experience during the test, which is exactly the kind of configuration detail a benchmark needs to report alongside the throughput number, not bury underneath it.

Across all of these figures, one pattern holds: raw sequential throughput on modern PCIe 5.0 drives isn't the binding constraint anymore. That throughput must be reachable under the interleaved, mixed-size, concurrent pattern a serving stack actually generates, and none of the vendor numbers cited here confirm it, because none of them report sub-4K IOPS or multi-stream interleaved-read throughput. NVIDIA's Storage-Next initiative, surfaced publicly at GTC 2025, quotes 512-byte IOPS, not bandwidth, as its primary metric, targeting maximizing 512-byte IOPS per GPU under power and tail-latency constraints. That's the gap between what a spec sheet advertises and what an inference engineer actually needs to know before betting a serving budget on a drive. The Samsung PM1753 delivers up to 14.5 GB/s sequential throughput and 3.3M random read IOPS, positioned explicitly for KV cache workloads Scaling AI Inference with KV Cache Offloading White Paper KV Cache Meets NVMe: The Key to Accelerating LLM Inference.

Serving framework choices that change what the storage layer is asked to do: disaggregation, offload libraries, and eviction strategies

Prefill-decode disaggregation has become common practice, and it reshapes what the storage layer is asked to do on its own. In a disaggregated setup, prefill workers compute the KV cache and then have to transfer it to separate decode workers before generation can start, and that transfer step is not free SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cach… LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferen…. SmartGen's research puts a number on it: KV cache transfer can eat up to 42.2% of total job completion time SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cach… LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferen….

Where NVMe physically sits in a disaggregated system changes what kind of traffic it sees. Placed at the prefill node, holding cache while generation happens elsewhere, the drive sees large sequential writes followed by reads that feed the transfer step. Placed as a shared cache store serving multiple decode workers, it sees concurrent, multi-tenant random reads instead. Same hardware, two very different jobs.

Software libraries built on top of this add another layer of variation. LMCache supports both DRAM and SSD tiers, and it can either pipeline I/O with compute layer by layer (non-GDS mode), or use GPU Direct Storage, without layer-wise pipelining LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferen…. SGLang's HiCache and vLLM's own automatic prefix caching shift the hit-rate distribution that eventually reaches the drive in similar ways. Higher hit rates mean fewer cold reads pulled from NVMe and more warm re-reads of blocks the system already has cached somewhere upstream, so a benchmark that ignores expected hit rate is testing the wrong thing.

Cost-control decisions ripple through this too. Context truncation, a standard technique for keeping inference costs down, can cut the prefix cache hit ratio roughly in half according to LMCache's own enterprise usage data. That means identical hardware, running the same model, produces a noticeably different I/O pattern purely based on whether truncation is switched on. Model architecture adds yet another variable. DeepSeek-V3 uses a multi-head latent attention design, and falling back to MHA-equivalent sizing assumptions with 8-way tensor parallelism drops the maximum batch size an 80 GB H100 can support from 104 down to 15. The batch size the storage layer has to support is baked into the model's architecture, not just a function of expected traffic. It's baked into the model's architecture. SmartGen quantifies the transfer cost: transferring 48K-token long-context prompts takes 6.5× more time than the prefill computation itself on 25 Gbps RDMA links, and selective transfer reduces time-to-second-token by up to 4.3× versus full KV cache transfer SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cach… LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferen….

Structuring a benchmark that reflects actual KV cache read patterns rather than storage convention

Start with the workload, not the drive. Before picking a benchmarking tool or reading a single result, the eviction policy in use, the expected prefix hit rate, the sequence length distribution, and the concurrency level all need to be known quantities, because each one shifts what "good performance" even means for this system.

From there, sweep access size deliberately. Test at 4K, sure, but keep going down through the sub-4K range, because that's where rated IOPS and delivered IOPS pull apart the most. Run multiple independent sequential streams concurrently rather than relying on a single-threaded sequential test, since only that setup shows the controller keeps its sequential-detection intact once several requests interleave. Calibrate the read/write mix to the actual serving scenario in front of you. Prefill-heavy traffic writes heavily at offload time and reads heavily at reload, a high-hit-rate prefix cache is almost all reads of shared blocks, and a disaggregated system adds a transfer read that may bypass the local drive entirely.

Queue depth deserves different treatment here than it gets in a typical database benchmark. KV cache reads during decode are latency-sensitive, since each one blocks the next token from being generated, so tail latency at moderate queue depths tells you more than peak IOPS measured at maximum queue depth ever will. And be explicit about GPU Direct Storage: cuFile removes the CPU copy path entirely and changes the latency profile of every read that goes through it, so results collected with GDS enabled and results collected without it are not interchangeable, full stop. The test configuration has to match what the serving stack in production actually uses.

Infrastructure platforms purpose-built for this workload (including storage systems designed around GPU-direct NVMe paths, BlueField DPU offload, and AI-native access-size targets) produce results that synthetic benchmarks rarely replicate. That's reason enough to treat benchmark methodology as seriously as the headline numbers themselves when evaluating any storage option for this workload. A number without the method behind it is not a number worth trusting. The 33 MB logical block and its 2.8 MB and 6.1 MB sub-request breakdown gives a practical reference for the actual I/O unit the block layer delivers to the drive, and benchmark sub-request sizes should bracket this range.

Sources

  1. Leyline: KV Cache Directives for Agentic Inference
  2. SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
  3. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  4. AnchorKV: Anchor-Residual KV Cache Compression
  5. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving
  6. Scaling AI Inference with KV Cache Offloading White Paper
  7. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving
Filed underBenchmarking

More in Benchmarking