NVMe Offloading for KV Cache in Long-Context vLLM Deployments
NVMe offloading overcomes GPU memory limits for serving multiple long-context requests concurrently.

Long-context serving in vLLM runs into a hard physical wall: the key-value cache that every request needs grows linearly with context length, batch size, model width, and the number of layers in the model, and at the context lengths production workloads now demand, GPU memory alone cannot hold what multiple concurrent users require. The cause sits in how transformer attention works. Every token in a sequence has to attend to every token that came before it, and the only way to avoid recomputing the key and value tensors for each prior token, at each layer, on every forward pass, is to store them. That stored set of tensors is the KV cache, a structural cost of autoregressive generation itself. If one user holds a long conversation, or sends a single document-analysis request spanning tens of thousands of tokens, that alone can use up a GPU's spare HBM capacity. So when you multiply that by dozens or hundreds of concurrent sessions on a shared server, the arithmetic fails well before any GPU runs out of raw compute. The limit occurs in capacity and data movement rather than in speed, and it appears first as a wall on how many users a GPU can serve rather than as a wall on how fast any one of them is served.
Yes, NVMe SSD offloading is a viable and well-tested way to extend KV cache capacity past what GPU HBM and CPU DRAM can hold. The three-tier hierarchy works as follows: hot KV blocks stay on GPU HBM, warm blocks move to CPU DRAM, and cold blocks offload to NVMe, where commercial servers can offer more than 100 TB of capacity, enough to retain over an hour of KV cache for long-tail sessions and historical context that DRAM cannot hold. Two independent research systems, Tutti (now integrated into vLLM) and DUAL-BLADE, demonstrate that the approach works in production when the CPU is removed from the I/O control path entirely, with Tutti cutting time to first token by 78.3% under strict SLO constraints and reaching retrieval bandwidth up to 25.9 GB/s on a dual-SSD setup. The main caveat is that NVMe offloading is a workload-specific admission decision: on a fast GPU serving mostly short-context requests, GPU memory alone may already retain enough prefixes that reaching for disk adds latency rather than relieving it.
What PagedAttention Solves
vLLM's PagedAttention answers a narrower question than "how do we fit more KV cache in memory." It answers "how do we stop wasting the memory we already have." Before PagedAttention, KV cache allocation in GPU serving systems suffered from the same problem that plagued early virtual memory systems: over-allocation and fragmentation, where blocks reserved for a sequence sat unused because the allocator had no way to pack them tightly. PagedAttention borrows the operating system's answer to that problem. It virtualizes KV cache storage into fixed-size blocks that can be allocated and addressed independently, the same way an OS pages physical memory, and that virtualization pushes fragmentation waste below 4% while unlocking real throughput gains in production serving. If you take it away, a meaningful share of allocated KV memory sits idle before a single long-context request ever gets served, so PagedAttention is a necessary piece of any serving stack that wants to support long contexts at scale. But necessary is not the same as sufficient. PagedAttention makes the GPU's existing memory usable, not larger. It does nothing to expand the total pool of memory available for KV cache, and the very mechanism that eliminates fragmentation, chopping logically contiguous KV data into small, independently addressable blocks, scatters that data physically across GPU memory. That scattering doesn't matter while everything stays on the GPU. It becomes the central obstacle the moment any of that data needs to move to or from a storage device, because the fragmented layout that solved GPU capacity waste turns into fragmented I/O when KV blocks have to travel somewhere else. PagedAttention leaves open the capacity beyond what HBM physically holds, and that gap forces the rest of the serving stack to look past the GPU.
Can NVMe SSDs extend KV cache capacity beyond GPU memory?
Once capacity, not fragmentation, is the binding constraint, the only way to keep serving long-context, multi-user workloads is to spread KV cache across a hierarchy of memory tiers, each with different capacity, speed, and cost characteristics. This structure is the only one that makes multi-user long-context serving work at all, economically or physically. The hierarchy assigns KV blocks to a tier based on how recently they were used and how likely they are to be needed again. Hot blocks, the ones tied to a generation in progress, stay on GPU HBM, where access latency is lowest. Warm blocks belong to sessions that just finished but are likely to resume in a multi-turn conversation, so you move them down to CPU DRAM. Cold blocks, covering historical context and the long tail of prefixes unlikely to be reused soon, move further down to NVMe SSD. CPU DRAM extends the usable pool past HBM, but it runs out faster than it sounds: even roughly 2 TB of DRAM holds only about five minutes of KV cache under realistic serving conditions. NVMe is the tier that actually changes the equation. Commercial servers can offer more than 100 TB of NVMe SSD capacity, enough to retain more than an hour of KV cache, so NVMe is the only tier that can hold cold sessions and long-tail prefixes at the scale real deployments need. That capacity comes with a cost that the earlier tiers don't impose in the same way. A cache hit against disk trades GPU prefill computation for a disk read plus a CPU-to-GPU transfer, and for short prompts, workloads with low cache-hit rates, or very fast GPUs, that trade loses. NVMe offloading is a workload-specific admission decision: on an H100 serving mostly short-context requests, GPU memory alone may already retain enough prefixes that reaching for disk only adds latency.

Why Naive NVMe Offloading Stalls the Serving Stack
You can have NVMe capacity available without benefiting from it, and SSD bandwidth is not why. Modern NVMe SSDs are fast. The bottleneck is a mismatch between how KV blocks sit in GPU memory and the access pattern SSDs need to hit their peak performance, made worse by control paths that still depend on the CPU at every step. Four distinct failure modes produce that mismatch.
The first is fragmentation-driven random I/O. PagedAttention's paged layout, the same mechanism that eliminated GPU-side fragmentation, scatters a logically contiguous KV cache into a large number of small blocks. If you restore the KV cache for a long context, you pull back a very large number of these scattered objects in a single restore operation, and that scattered, small-object access pattern is the one NVMe handles worst.
The second is a CPU-centric control path. GPU Direct Storage (GDS) removes the DRAM staging copy that older designs required, letting data move more directly between SSD and GPU. But GDS still needs the CPU to prepare descriptor addresses, submit requests to NVMe queues, and receive completions for every single I/O operation. So at the scale of a long-context restore, with thousands of small reads in flight, that per-I/O CPU involvement stalls the GPU badly. Tutti's measurements report that this bottleneck leaves the GPU stalled for most of its time.
The third is page-cache thrashing in file-based designs that rely on mmap and the kernel page cache. During prefill, large writes trigger synchronous write-back to disk. During decode, the autoregressive access pattern repeatedly reads the growing KV cache, which evicts pages before they can be reused and collapses the effective page-cache hit ratio.
The fourth is contention between concurrent reads and writes for the SSD's internal write buffer, which degrades the combined effective bandwidth well below what either sequential reads or sequential writes achieve running alone. A common fix for DRAM-backed caches, pipelining KV transfers with ongoing computation, fragments transfers, cuts effective bandwidth, and adds CPU scheduling overhead on SSDs, making the problem worse. Compounding all four, the Linux multi-queue block layer spreads requests across multiple NVMe queues, which takes what started as a contiguous KV cache stream and turns it into an interleaved, non-sequential arrival order by the time it reaches the SSD controller. Each of these failure modes points toward a specific design fix. The engineering decisions addressing them come in the next two sections.
Transfer Granularity and Scheduler-Aware Preloading
Whether a disk-backed KV cache helps or hurts a serving stack comes down to three variables: how data is chunked for transfer, how much intermediate memory sits between SSD and GPU, and when in the request lifecycle the transfer actually starts. Raw device bandwidth is a much smaller factor than any of the three. The py-kvcache characterization traces KV blocks through every stage of vLLM's internals, the scheduling logic, the KV connector and transfer interfaces, CPU memory, GPU memory, and NVMe storage, to find where time actually goes, and the overhead it finds is dominated by how I/O interacts with the scheduler, not by how fast the SSD can read.
The clearest lever is preloading. py-kvcache's scheduler-aware preloading starts disk reads while a request still sits in the queue, before the GPU is ready to consume the data, so the I/O overlaps with compute ahead of when the GPU needs it. At long context lengths, this overlap alone accounts for a meaningful share of py-kvcache's total speedup over LMCache: at 80,000 tokens, loading from disk runs roughly twice as fast as LMCache, with preloading contributing a substantial share of that improvement on its own. A second lever is bypassing the kernel page cache altogether: py-kvcache uses asynchronous direct I/O with bounded shared staging, so it never routes through the page cache in the first place and avoids the thrashing described earlier. With GPU, CPU, and disk caching all enabled together, the system runs noticeably faster than LMCache and lands within about 4% of vLLM's native KV Offload implementation.
The GPU matters as much as the I/O design. On a weaker GPU, preloading from disk measurably improves time to first token. On an H100, the average request falls below the break-even point where disk-backed caching pays off, because GPU memory alone already retains enough prefixes to serve most requests without touching disk. The same preloading design that helps one deployment adds pure overhead to another. That is why py-kvcache treats NVMe offloading as a policy decision, engaged selectively by weighing context length, GPU capability, and cache-hit rate. Results on LongBench and SCBench show these preloading gains aren't confined to synthetic, uniform-length test cases. They extend to irregular prefix chains and multi-turn conversational workloads, the kind of traffic real deployments actually see.
Two architectural approaches to eliminating the CPU from the I/O control path
Two research systems, built independently for different deployment contexts, converge on the same architectural move: taking the CPU off the I/O control path, not just the data path, to close the gap between what SSDs can deliver and what the serving stack can actually use.
Tutti (arXiv 2605.03375, May 2026, now integrated into vLLM) starts from the observation that GDS only solved half the problem. It removes the DRAM staging copy, but the CPU still has to prepare descriptors, submit requests to NVMe queues, and collect completions for every I/O, and that per-operation involvement is what produces the GPU stalls described earlier. Tutti's fix is to give the GPU its own asynchronous I/O control mechanism, independent of the CPU, built from three pieces: a GPU-native object store that uses Scatter Gather List addressing to move KV data in bulk, a GPU-resident io_uring that mirrors the structure of Linux's asynchronous I/O subsystem but runs entirely on the GPU, and a slack-aware scheduler that uses offline profiling to keep reads and writes from contending with each other or with ongoing GPU compute. So under this design, the CPU only has to load I/O kernels asynchronously once per layer, and that stays fully off the critical path of serving a request. Measured against GDS-enabled LMCache, Tutti cuts time to first token by 78.3% under strict service-level objective constraints, roughly doubles the request rate a server can sustain, cuts serving cost substantially, and reaches retrieval bandwidth up to 25.9 GB/s on a dual-SSD setup. Its performance comes close to DRAM-backed LMCache, but its capacity is, for practical purposes, unlimited.
DUAL-BLADE (arXiv 2604.26557, April 2026, from Sogang University, Florida State University, and Samsung Electronics, set to appear at ICDCS 2026) targets a different deployment profile: edge AI systems running on fixed, tight memory budgets between 8 and 32 GB, often on unified-memory platforms where CPU and GPU (or NPU) share the same physical memory pool. The constraints differ from cloud serving, but the underlying I/O problem is the same one Tutti addresses. DUAL-BLADE's answer is dual-path KV residency: KV tensors that fit within the page-cache budget go through the standard mmap path to preserve a high cache-hit ratio, while tensors that exceed that budget go through an NVMe-direct path that maps them to contiguous logical block address (LBA) regions and bypasses the filesystem layer. That sequential LBA placement restores the sequential locality that the Linux multi-queue block layer otherwise destroys when it spreads requests across NVMe queues. DUAL-BLADE pairs this with io_uring_cmd for kernel bypass, a lighter-weight path than a full SPDK deployment, and adds adaptive pipeline parallelism to overlap storage I/O with GPU DMA transfers. Across a range of memory budgets, the result is a substantial reduction in both prefill and decode latency, alongside improved SSD utilization.
Tutti and DUAL-BLADE arrive at the same structural answer from opposite directions, one built for cloud-scale multi-user serving, the other for memory-constrained edge inference. Both treat the CPU's presence in the I/O control path as the mechanism to remove, not the data path alone, and both demonstrate that the physical wall described at the start of this piece has an engineering answer once the KV cache is allowed to live where it actually fits: across GPU HBM, CPU DRAM, and NVMe, moved by a control path built for the job.
Sources
- Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
- DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
- Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving
- KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference


