KV Cache Offloading Tools Ranked by Production Inference Overhead
Choose your KV offloading tool by workload: prefix reuse, agent sessions.

Production KV cache management requires matching the offloading architecture to the workload. LMCache favors repeated prefixes, Mooncake favors long agent sessions across nodes, and NIXL/Dynamo favors strict prefill-decode handoffs. Their production ranking depends on time to first token, GPU utilization drag, and transfer latency, with compression and storage infrastructure determining whether retrieval beats recomputation.
Why the KV cache grows into a first-order infrastructure problem at production scale
Model weights stay fixed once loaded, but KV cache grows linearly with context length, batch size, model width, and layer count. At long context and meaningful concurrency, that growth exceeds available GPU HBM. Prefill and decode then compete with inactive sessions whose KV blocks remain resident until eviction forces recomputation. At very long contexts, KV memory can dwarf parameter memory, so faster GPUs alone do not resolve the storage bottleneck.
HeadInfer illustrates the scale of this imbalance. Its head-wise offloading reduces the GPU KV footprint from over a hundred gigabytes to roughly a gigabyte at very long context lengths. The reduction comes from changing what stays resident while GPU speed remains unchanged. That distinction underlies the offloading decision.
What "overhead" actually means across the memory hierarchy, and why the answer differs by tier
The costs of offloading can change independently of one another. Transfer latency can increase time to first token, or TTFT, when cached blocks must load before decoding begins. GPU utilization drag arises when streaming multiprocessors, or SMs, wait for those blocks during computation. PCIe bandwidth is a small fraction of HBM bandwidth, so using host memory to hold more cache also gives attention a slower path to that data.
Meng et al.’s bottleneck analysis found that transfers consumed the vast majority of request latency in offloaded serving, while GPUs ran well below their rated power draw. Their κ_crit denotes the cached-to-prefill token ratio at which execution becomes memory-bound. It marks a boundary in the model, with limits on its use as an admission rule. The py-kvcache study brings scheduling into this model because starting disk reads while requests queue can lower observed TTFT even when device bandwidth stays the same.
The transfer-path architecture that separates one tool from another
Offload granularity determines how much memory a transfer releases and how often transfers occur. Layer-wise offloading moves an entire layer’s KV tensors, head-wise offloading moves individual attention heads, and block-wise paging uses fixed-size blocks. Finer granularity reduces peak GPU memory but multiplies transfer round-trips.
CPU-staged transfers pass through host RAM, which can become a bottleneck under pressure, while cuFile and GPU Direct Storage support direct storage-to-GPU movement on compatible hardware. Depending on scheduling integration, reads begin during queueing or only at dispatch. For multi-node serving, local versus disaggregated storage adds another distinction because shared pools permit reuse across replicas. The same raw I/O bandwidth can therefore yield very different observed TTFT depending on granularity, staging, scheduling, and cache placement.
How vLLM's native CPU offloading and py-kvcache define the baseline overhead floor
vLLM’s native CPU swap provides a predictable, low-setup reference. At low KV footprint, short prompts and low concurrency can retain baseline latency with minimal overhead. The cost emerges when fetching blocks makes PCIe the critical path, with scheduler-oblivious I/O offering little opportunity to hide it.
py-kvcache implements asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading through vLLM’s KV Offload API. Its characterization study reports that disk loading is roughly twice as fast as LMCache at long contexts, and much of that gain comes from preloading. With all tiers enabled, it runs within a few percent of native KV Offload. Hardware changes the result, however. On an H100, the average request in the Bailian production trace falls below the external-cache break-even point because GPU memory already holds enough prefixes, while the gains are clear on a weaker GPU.
That result illustrates the workload dependence behind κ_crit. A fast configuration can remain below the critical cached-to-prefill ratio and gain nothing from additional caching. For teams characterizing their own traffic, native offloading and py-kvcache provide the appropriate starting point before adopting a more complex system.
Where LMCache's tiered architecture wins and what it trades away
LMCache shifts the overhead profile through asynchronous prefetch, compression before transfer, and distributed backends such as Redis. CacheGen cuts the amount of KV data sent over the network, so compression is built into the transfer architecture even when storing less data is optional. This architecture works best when prefixes repeat. Identical or near-identical system prompts, RAG documents, and shared document Q&A create reuse across users.
The py-kvcache study reports 34–58% smaller inter-token latency for LMCache than the native path at high QPS with repeated prefixes, together with dramatically smaller TTFT. Those gains do not make retrieval free. Compression and retrieval add warm-path latency that can make the system unsuitable for short-context or single-user traffic. Its NVMe backend improves throughput on cache hits by avoiding prefill recomputation.
Google Cloud uses LMCache in GKE Inference. CoreWeave and Cohere have also benchmarked it on CoreWeave infrastructure, providing evidence of evaluation at scale. The relevant production question remains whether shared-prefix reuse repays the extra pipeline work.
Mooncake's disaggregated pool model and the overhead trade-off of RDMA-backed KV sharing
Mooncake changes cache placement from per-node tiers to a shared, cross-node pool integrated with vLLM. GPU HBM is registered as RDMA buffers, enabling GPUDirect RDMA reads and writes without CPU staging or SM use for the transfer. This broadens where a session’s KV can be reused. Mooncake’s vLLM integration reports substantially higher throughput, lower TTFT, and lower end-to-end latency on realistic agentic traces, scaling across a large number of GB200 GPUs with a high sustained cache hit rate.
The strongest fit is long, unique agent traces whose per-user KV diverges. LMCache instead favors high-overlap prompts and documents, where reuse is already achievable within a node and pooling adds less. NIXL/Dynamo addresses strict 1:1 prefill-decode handoffs.
Mooncake also requires an RDMA fabric to deliver this path. Block-location coordination adds latency on cold misses, while prefill-decode separation introduces network round-trips.
NVIDIA ICMSP and NIXL/Dynamo change the overhead model through hardware acceleration
NVIDIA ICMSP uses BlueField-4 DPUs to manage KV block movement outside the GPU compute path. The DPU acts as an asynchronous I/O coprocessor, while cuFile handles GPU-direct NVMe transfers. For long-context, prefix-heavy workloads, retrieving cached blocks can then be far faster than recomputing the prefix.
NIXL is the KV block transport protocol within ICMSP. Dynamo uses it between prefill and decode workers, orienting the transport toward strict 1:1 pairing. This differs from Mooncake’s pooled sharing and LMCache’s tiered cache reuse. ICMSP addresses the same movement problem at the hardware layer, but its acceleration requires BlueField-4 DPUs in the server. Software-only deployments, including most cloud bare-metal instances, cannot use that path and must rely on software equivalents.
Why the SSD tier is the most contested tier, and when NVMe creates GPU bubbles
Tutti’s 2026 finding presents the sharpest warning about SSD offloading. With naive paged KV layouts, mismatched SSD access patterns caused GPU bubbles to exceed 70% of total inference latency in all cases studied. Layer-wise SSD transfers made the result worse. Yet ICMSP and LMCache report large TTFT improvements for prefix-cache workloads, creating an apparent contradiction.
High prefix hit rates and long contexts can make recomputation slower than an NVMe load, whereas short prompts or fast GPUs can reverse that comparison. The py-kvcache study makes this break-even setup-specific. SSD capacity is therefore useful only when the saved prefill work outweighs the retrieval path, rather than whenever GPU memory is scarce.
DUAL-BLADE addresses the access-path mismatch by assigning KV tensors dynamically to either a page-cache path or an NVMe-direct O_DIRECT path according to runtime memory availability. It reduces prefill and decode latency and improves SSD utilization relative to file-based designs relying on the kernel page cache. Predictability introduces a further trade-off here. O_DIRECT exchanges average-case performance for tail-latency stability, so the preferred path can differ between throughput-oriented and latency-sensitive deployments.
Compression as an offloading multiplier changes the transfer cost equation
Quantization reduces both cache footprint and the bytes crossing each tier. FP8 halves KV data volume relative to BF16 with negligible accuracy loss. On Blackwell GPUs, NVFP4 reduces KV memory by three-quarters relative to BF16 and half relative to FP8. The vLLM team reports substantially higher burst throughput from hardware-native FP8 Tensor Core operations because FP8 reduces attention computation as well as storage.
Lossless encoding can reduce that volume further. A 2026 result demonstrates Huffman coding that exploits skewed FP8 exponent distributions, especially in early tokens, and the larger exponent width in BF16. A separate 2026 result uses cross-token clustering and ZSTD for additional reduction on LLaMA 3.1 8B in BF16. As CacheGen demonstrates within LMCache, reducing transferred bytes shifts offloading’s break-even toward shorter contexts and lower hit rates, although encoding remains part of the overhead profile.
Token pruning as a structural alternative reduces what needs to be offloaded
SlimInfer cuts the amount of cache data before transfer begins. Its layer-wise dynamic pruning removes less critical prompt tokens during the forward pass, drawing on the diffusion of token information through later layers. This enables an asynchronous KV manager to prefetch required blocks without complex predictors. SlimInfer reports improved TTFT and end-to-end latency on LLaMA3.1-8B-Instruct on a single RTX 4090 without accuracy loss on LongBench.
HeadInfer’s head-wise offloading keeps one attention head’s KV on the GPU at a time, enabling million-token contexts on a single RTX 4090. More transfer round-trips are the corresponding cost, making adaptive head grouping and chunked prefill important to controlling latency. Its fit is memory-constrained single-GPU inference. The most capable production systems will likely layer pruning, quantization, and tiered offloading.
Which KV cache offloading tools are best for a production workload?
The workload-to-tool matrix begins with prefix overlap and session length. High-overlap system prompts and RAG favor LMCache with compression and a Redis or disk backend because reuse amortizes retrieval overhead across users. Long, unique agent traces favor Mooncake’s shared RDMA-backed pool, which makes session KV available across the fleet. Strict prefill-decode pairing instead favors NIXL/Dynamo’s transport-oriented design.
Long-context NVMe prefix retrieval with high hit rates favors ICMSP’s low SM overhead where BlueField-4 DPUs are available, while LMCache’s disk tier and DUAL-BLADE provide software alternatives. Memory-constrained single-GPU long-context inference favors HeadInfer. Moderate-context serving on a fast GPU can need no more than native vLLM offloading and py-kvcache for characterization.
The practical sequence is to apply suitable compression and quantization before choosing the offloading architecture. Those reductions improve the transfer equation across tiers, while the remaining workload determines which retrieval path earns its cost. Storage then sets whether the bandwidth available in theory leads to lower production TTFT. Purpose-built NVMe systems, RDMA fabrics, and compression pipelines aware of BF16 and FP8 structure are part of the inference design.
Sources
- SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
- Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
- Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
- Compute Or Load KV Cache? Why Not Both?
- HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading


