Selecting NVMe Drive Count and Topology for KV Cache Offloading Workloads
Bandwidth, not storage capacity, determines how many NVMe drives you need for KV cache offloading.

The instinct when sizing storage for KV cache offloading is to ask how many terabytes of NVMe a node needs. That question has the wrong unit attached to it. When context windows stretch into the tens of thousands of tokens, capacity stops being the thing that fails first. A single long-context user's KV cache on a 70B model can reach tens of gigabytes on its own, and if you serve several such users at once, you need to pull that data back off NVMe fast enough that the GPU never sits idle waiting on blocks that haven't arrived yet. A drive array can hold a staggering amount of cached context and still be the wrong choice for a serving workload: what matters is how many bytes per second it can push back to the GPU, not how many bytes it can hold. Bandwidth, not capacity, is the variable that decides whether offloading helps or hurts.
The bandwidth arithmetic that sets your drive-count floor
The number of drives a deployment needs comes from a fairly simple division: take the sustained KV restore bandwidth the workload demands, and divide it by the sequential read throughput a single drive actually delivers under the access pattern KV offloading produces. That second number is whatever the drive can sustain once the real I/O pattern of restoration is applied to it, which can differ substantially from the figure printed on the drive's spec sheet.
Several inputs feed the first half of that equation. KV block size per token depends on the model's architecture, specifically its layer count, head dimension, and number of KV heads, and also on the precision used to store it, since BF16 and FP8 produce very different byte counts for the same logical state. And a target time-to-first-token or restoration latency budget is what converts all of that from a bandwidth number into a time constraint the hardware either meets or doesn't.
I/O characterization work on these workloads shows the access pattern is dominated by large block reads, a size that is considerably more favorable to sequential drive performance than the tiny random reads that fragmented memory layouts tend to produce. That favorable pattern only shows up, however, when the I/O path is built to preserve it. A system that lets logically contiguous KV cache scatter across random logical block addresses throws away that advantage before the drive ever sees a request. That's the thread the I/O path discussion later in this piece picks up.
How KV cache precision changes the drive count
The most consequential decision affecting drive count has nothing to do with storage at all. It's the precision chosen for the KV cache itself, and it needs to be settled before any bandwidth math gets run, not after. Halving the precision of a KV cache halves the number of bytes that have to be read from NVMe for every restoration event, which can eliminate the need for drives that would otherwise have been bought.
BF16 is the baseline most serving stacks start from. Concretely, setting the KV cache precision flag to FP8 in a serving engine like vLLM before running the drive-count calculation can cut the resulting drive floor in half. On Blackwell hardware, NVFP4 goes a step further and halves the per-token KV footprint again relative to FP8, which changes the drive-count requirement substantially enough that practitioners on that hardware should factor it in before specifying anything.
The sequencing matters more than any individual precision choice. Precision selection is step one. Drive count calculation is step two. Running the steps in reverse order, sizing hardware first and adjusting precision later, produces a storage tier that's either over-provisioned for the workload it ends up serving or specified against numbers that no longer apply once precision changes.
PCIe topology and the ceiling it places on aggregate NVMe bandwidth
Correctly counting drives doesn't guarantee they can deliver their rated bandwidth, because the PCIe fabric connecting them to the GPU imposes its own ceiling. When multiple drives share a PCIe switch, a root complex, or a CPU interconnect, they contend for the same upstream bandwidth, and the aggregate throughput the array can sustain saturates well below the simple sum of each drive's individual rating.
The GPU-to-NVMe data path is the critical path here. Server platform topology ends up mattering more than any individual drive's spec sheet: how many PCIe root complexes the platform has, how NVMe slots are distributed across them, and whether a DPU, such as NVIDIA's BlueField-4 in ICMSP deployments, offloads I/O scheduling away from the host CPU, all shape the aggregate bandwidth the array can actually deliver. Multi-socket servers add a further wrinkle through NUMA effects: NVMe drives attached to one CPU's PCIe domain show higher latency and lower effective bandwidth when accessed by a GPU sitting on the other socket's PCIe domain, so KV offloading systems need to be deliberate about which socket's drives serve which socket's GPUs.
You can map this out before specifying drives using a consistent method, regardless of the specific server. Enumerate the PCIe root complexes on the platform and the lane budget each one has available. Identify which root complex each GPU is physically attached to. If the physical layout allows it, place NVMe drives on that same root complex. And account for every other PCIe consumer sharing that complex, network interface cards and host bus adapters included, since those devices draw down the same lane budget the NVMe drives are counting on.
I/O path, drive count, and topology: the determinants of realized throughput
A drive's rated sequential bandwidth and the bandwidth KV offloading actually achieves in production are often two very different numbers, and the gap between them comes down to how well the I/O path preserves sequential access and keeps the CPU out of the critical path. Paged KV layouts, left unaddressed, work against both of those goals by default.
PagedAttention, the memory management scheme widely used in serving engines, fragments GPU memory into many small, non-contiguous blocks. Together, these produce GPU stall rates severe enough that in naive implementations, restoring from SSD ends up slower than simply recomputing the KV cache from scratch.
The fix researchers have converged on is architectural. A GPU-centric KV cache object store, demonstrated in the Tutti system, limits the CPU's role to asynchronously loading I/O kernels once per layer, introduces a GPU-native object abstraction for bulk KV transfers, uses GPU io_uring to support asynchronous direct object I/O, and applies slack-aware scheduling to avoid contending with the GPU's other work. So you get inference performance that nearly matches a DRAM-backed cache while offering effectively unlimited capacity. A further principle, sequentiality preservation, maps logically related KV blocks to adjacent physical addresses on the drive so the sequential throughput and the reduced flash translation layer overhead that come with contiguous placement survive all the way to the hardware.
None of this appears on a drive's spec sheet, because that figure is measured without the real I/O path; a poor I/O path leaves the array well short of the bandwidth the sheet promises, while a well-designed one lets it reach that bandwidth. A system with a well-designed one, using GPU-direct access, scheduler-aware preloading, and contiguous block placement, will pull the full rated bandwidth out of a smaller array than the naive math would suggest is possible.
How striping across multiple drives multiplies effective bandwidth
Striping KV cache across several NVMe drives is the main way to get past the bandwidth ceiling any single drive imposes, but doing it naively leaves most of the benefit on the table. A single drive is bandwidth-bound for KV restoration well before you reach production-level concurrency, so the obvious answer is to spread KV blocks across multiple drives, so a restoration event reads from several of them in parallel and adds their bandwidth together.
Round-robin striping, where blocks are distributed without regard to how they're used, does scale aggregate bandwidth, but it ignores which KV pairs tend to be restored together. Achieving close to linear scaling requires the placement to be aware of co-activation: KV pairs consistently restored together, whether because they belong to the same layer group or the same prefix segment, should be spread across different drives so their simultaneous restoration saturates every drive in the array at once. That requires the serving system to track access patterns and feed them into placement decisions, not a static configuration set once and left alone.
This scaling has a ceiling of its own, and it's the same PCIe topology constraint discussed earlier. Adding drives beyond what the upstream PCIe port can carry yields no further throughput, because the effective scaling limit is the port's aggregate bandwidth divided by what each drive delivers. The practical sequence is to start from the bandwidth target the sizing calculation produced, divide it by the per-drive sustained sequential throughput, derated for the actual access pattern rather than the drive's peak spec, confirm the resulting drive count fits inside the topology's aggregate ceiling, spread the drives evenly across available PCIe root complexes, and configure the KV management layer to place co-activated blocks on separate drives.
NVMe-oF and rack-scale pooling as an alternative to per-node drive counts
Once prefill and decode run on separate nodes, the relevant topology is no longer the PCIe fabric inside one server, but the network connecting the nodes that produce KV blocks to the nodes that consume them. Disaggregated serving splits prefill workers, which generate the KV cache, from decode workers, which consume it, and the cache has to cross the network to get from one to the other. That makes network bandwidth the new constraint in place of PCIe bandwidth.
NVMe-oF lets several compute nodes share a common pool of NVMe storage over RDMA, letting GPUs read from remote drives with much of the CPU involvement stripped out, though eliminating CPU participation entirely on both ends still requires additional technologies such as GPUDirect Storage or DPU-based target offload. The bandwidth ceiling in this architecture comes from the network fabric itself: a 100 GbE RDMA fabric has a defined aggregate budget, and that budget has to be sized against the combined restoration demand of every node sharing the pool at once, exactly the way per-node drive bandwidth was sized against per-node demand earlier.
The trade-off between pooled and local storage follows familiar lines. Pooling NVMe over the network raises utilization of the cluster's total capacity, makes KV blocks reachable from any compute node, and allows context to persist across separate inference runs, at the cost of added network latency on the restoration path and a dependency on RDMA-capable networking. NVMe-oF becomes the right architecture once you need KV cache sharing across nodes, in multi-agent systems or multi-turn sessions that get routed to different nodes, not just when you find it convenient.
The serving stack's cache hit rate in the drive sizing decision
Aggregate NVMe bandwidth demand doesn't just depend on how many concurrent sessions a deployment serves. It's modulated by how often the serving stack's cache actually gets hit, so if a stack has a higher hit rate, it needs meaningfully less NVMe bandwidth to serve the identical request load. A hit means the requested prefix is already sitting in HBM or DRAM, and an NVMe fetch only gets triggered on a cold miss, so a serving stack generating fewer misses per second is, by construction, consuming less NVMe read bandwidth.
Hit rate isn't purely a function of how repetitive the workload is. It also depends on how the serving engine manages its cache index. Prefix-tree-based management, such as RadixAttention in SGLang, achieves substantially higher hit rates than the flat block-level LRU eviction used in engines built around PagedAttention, even though RadixAttention still applies an LRU policy at the leaf-node level of its tree. Research on py-kvcache makes an admission-control point worth building directly into sizing: for workloads where the average request falls below the break-even point between recomputing a KV cache and restoring it from an external tier, external caching, NVMe included, is a net loss. The drive tier should be sized only for the traffic that genuinely benefits from caching.
High-reuse workloads, retrieval-augmented generation, system-prompt-heavy chat, agentic tool loops that repeat the same scaffolding, drive heavy NVMe demand and justify a larger array. The sizing calculation therefore carries a probabilistic input at its core: the expected cold miss rate under target concurrency, which depends on the workload and needs to be measured from real traces before any hardware gets specified.
A systematic framework for making the drive-count and topology decision
Everything above resolves into a sequence of steps, each one constraining the next, where skipping or reordering a step produces a storage tier mis-specified for the workload it has to serve.
Step one is characterizing the workload itself: measure or estimate average context length, prefix reuse rate, and target concurrent sessions, then classify the workload as high-reuse, like RAG or multi-turn agents, or low-reuse, like single-turn generation, since low-reuse workloads may not justify NVMe tiering.
Step two sets KV precision before any bandwidth number gets calculated. Default to FP8 unless accuracy constraints rule it out, since that choice halves every subsequent bandwidth and capacity figure, and on Blackwell hardware, evaluate NVFP4 before settling on FP8.
Step four derates single-drive bandwidth for the real I/O path, not the spec sheet. Check whether the system uses contiguous LBA placement and asynchronous direct I/O; when either is missing, apply a conservative derating factor to the single-drive bandwidth figure.
Step five divides the aggregate bandwidth target from step three by the derated per-drive bandwidth from step four to get the minimum drive count, rounding up and adding margin for endurance and operational headroom.
Step six verifies that count against the platform's PCIe topology: map the available root complexes and their lane budgets, confirm the drive count fits the aggregate bandwidth ceiling without contention from other PCIe consumers, and if the topology caps aggregate bandwidth below the target, the right response is to evaluate NVMe-oF pooling rather than keep adding local drives that the fabric can't feed anyway.
Step seven specifies placement and striping: distribute drives evenly across PCIe root complexes, configure the KV management layer for co-activation-aware block placement, and for disaggregated deployments, size the network fabric itself as the binding topology constraint.
Where the research systems point for the field
The clearest direction in current KV offloading research is the steady removal of the CPU from the I/O path, and each system that manages it changes the drive-count math by recovering bandwidth that software-managed paths otherwise forfeit. Tutti's GPU-centric object store, by eliminating CPU intervention from the critical path, gets NVMe performance close to what a DRAM-backed cache delivers. The effective bandwidth available per drive improves substantially under a redesigned I/O path, potentially shrinking the drive count needed to hit a given latency target. py-kvcache's scheduler-aware preloading shows a related effect from a different angle: starting NVMe reads while requests are still queued hides restoration latency behind ongoing compute. This expands the time budget available for restoration without touching the hardware. CXL memory expanders, mostly CXL 2.x today with CXL 3.0 beginning to appear in early deployments, are being explored as a possible architectural alternative at the warm-cold boundary of the memory hierarchy, with latency between DRAM and NVMe, though they remain early enough that they shouldn't be treated as a substitute for NVMe in any sizing decision made today.
The practical conclusion is to specify drives against today's I/O path characteristics while keeping the software stack built so it can be replaced. As GPU-centric and DPU-managed I/O paths move from research systems into standard practice, the drive count a given workload requires will keep falling, and the framework in this piece should be rerun, not assumed, as that happens.
Sources
- Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving
- Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
- An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD
- Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving


