Cross-Request KV Cache Persistence Patterns for Session-Aware LLM Serving

Persisting KV state pays off only when fetching beats recomputing.

· 8 min read
Cover illustration for “Cross-Request KV Cache Persistence Patterns for Session-Aware LLM Serving”
Long-Context Serving · November 17, 2026 · 8 min read · 1,897 words

Cross-request KV cache persistence reduces production LLM serving costs only when fetching reusable state costs less than recomputing it. Efficient session-aware serving therefore combines referenceable blocks, session-aware lifetimes, tier-aware eviction, and a transfer path that meets the GPU’s latency budget. GPU HBM, CPU DRAM, and persistent storage must operate as one managed hierarchy with coordinated policies.

Why the KV cache can no longer be treated as a per-request temporary buffer

At 40 units of memory, a KV cache is a placement problem. Context windows growing from thousands to hundreds of thousands of tokens increase memory use proportionally. Multi-turn chat and RAG reuse prompts and retrieved contexts, while agentic workflows branch and resume earlier states. Freeing everything at request completion discards that reusable work.

Texas Tech University’s From Tensor Buffer to Distributed Memory Hierarchy survey frames this as a shift toward explicit memory management. The design space spans local virtualization and phase-aware scheduling, eviction or compression, prefill/decode disaggregation, cross-request reuse, offload tiering, and pooled-memory substrates. These are connected architectural choices. Architectural choices must enable persistence to eliminate recomputation without merely relocating its cost.

The four decisions every persistent KV cache design must answer

Locality determines which memory tier or node holds a block. It sets both available capacity and retrieval cost. Addressing comes first, because placement requires a referenceable unit. PagedAttention-style fixed blocks support fine-grained eviction and copy-on-write, while content-addressed prefix blocks support deduplication across sessions.

Lifetime determines when an entry remains valid and when it must be invalidated. Survival beyond request completion is only the beginning. A session may continue after an idle interval, so recency alone cannot express the value of retaining its state.

Ownership identifies the process or service authoritative for that state. It also constrains the available lifetime policies. A shared cache accessed by different workers requires an ownership model that extends beyond a single request handler.

Substrate identifies the hardware or protocol that exposes and transfers blocks. It limits which placements can meet serving latency. A remote tier is useful only when its transfer path can deliver reusable state within the workload’s budget.

The survey maps these axes to five archetypes, namely local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. Session-aware serving primarily occupies shared-store and hybrid-tier designs. Its cache must outlive requests and may serve different workers. The axes help compare designs, but a choice on one axis affects the others.

Why the bandwidth gap between memory tiers shapes every offload decision

Persistence exchanges computation for data movement. That exchange can lose. Fetching KV from a slower tier may take longer than rebuilding it from the prompt, particularly when the reusable context is short. The working set must therefore be sufficiently reusable to justify its transfer cost.

Each boundary changes that calculation, from HBM to CPU DRAM, from DRAM to NVMe, and from NVMe to object storage. Greater capacity comes with higher transfer latency. Shrinking blocks before transfer lowers that cost, so compression belongs in the architecture from the start. A system that ignores tier bandwidth will either over-evict into storage that cannot return blocks in time or under-evict by retaining stale blocks in HBM and starving new requests.

The three-tier storage hierarchy and how its tiers divide the work

GPU HBM holds blocks needed for active decode. Capacity is the immediate constraint at this tier. FP8 KV quantization halves per-block GPU memory, but it does not remove transfer costs when eviction becomes necessary. Reference counting and copy-on-write let sessions share physical prefix blocks until their sequences diverge.

CPU DRAM provides a burst buffer for recently evicted or prefetched blocks. It absorbs movement without immediately requiring a storage fetch. Beihang University’s CrossPool illustrates a related capacity decision by separating the KV-cache pool from the weights pool. Attention and non-FFN modules retain local KV access, while the separate FFN pool exchanges hidden states.

Persistent storage keeps large, cold prefixes after sessions end. Its access semantics need adaptation. S3 serves one object or byte range per request, whereas inference needs prefix-selected blocks delivered layer by layer. ETH/HPE’s ObjectCache aligns delivery with GPU consumption and overlaps transfer with compute across concurrent requests.

A large capacity tier is not automatically a useful retrieval tier. Blocks belong there when expected reuse and fetch latency justify admission, rather than simply because they are oldest in a pool-wide LRU queue. DRAM remains the staging layer between active GPU state and slower persistence.

Block addressing schemes and the progression beyond prefix matching

Prefix-tree addressing matches token sequences exactly and at low cost. Its reuse boundary is strict prefix alignment. Changing the system prompt or the RAG chunk order can stop otherwise shared content from being reused.

WombatKV (2026) extends sharing across engines through BLAKE3 chain-hashed blocks in an S3 bucket. Engines pointing at that bucket can reuse matching cached content. Cross-engine sharing does not itself remove positional constraints. Another system (2025–2026) addresses the persistence side by adapting log-structured storage techniques, including compaction and tiered storage, to KV management.

Position-independent caching widens reuse beyond aligned prefixes. That flexibility introduces a repair cost. Independently prefilled chunks lack the cross-chunk attention of joint prefill, so PIC methods selectively recompute tokens after assembly. LinearKV (2026) extends this approach to hybrid models such as Mamba-2 and Gated DeltaNet, finding that a single cached state can initialize a recurrent layer without exact composition of all cached states.

ReKV (ICWS 2026) widens compatibility across model precision boundaries. Its extended FlashInfer CUDA kernels support multi-segment KV representations, allowing high-precision KV state to survive a transition to a lower-precision model without prefix recomputation. ReKV reports up to 1.36× system throughput improvement and an average 86.59% reduction in SLO violation rates against static high-precision configurations.

How should production teams manage KV cache eviction and admission?

Not every displaced block deserves slower storage. Standard LRU treats evictions as equivalent, although a reversible move into DRAM differs sharply from a move into object storage. Recomputation-cost-weighted policies separate blocks that are cheap to rebuild from long, shared prefixes that cost more to rebuild.

Admission adds a separate test at each boundary. Expected reuse probability multiplied by retrieval’s latency savings over recomputation must exceed the fetch cost. The policy must also account for session continuity, because a conversation that is idle but still ongoing may have state that can be reused. The survey identifies tiered eviction among seven missing KV-specific measurements, leaving production teams dependent on workload profiling in the absence of an agreed evaluation standard.

Cache abstractions can limit those policies. The vLLM RFC for non-prefix-aligned token reuse, issue #25950, was filed in 2025 and closed “not-planned” in May 2026 with a request-batching workaround. It illustrates a lifetime-extension need that a strictly prefix-aligned cache cannot express. The practical recommendation is at least two eviction queues, separating blocks that fit within the recomputation budget and can be dropped freely from blocks whose recomputation exceeds next-tier fetch latency and should be retained or admitted downstream.

The I/O substrate behind fast KV transfer

Correct policy cannot compensate for late delivery. The transfer path must keep blocks ahead of GPU consumption. py-kvcache (2026), a Python external KV cache engine for vLLM on shared filesystems, uses io_uring for asynchronous I/O and lower CPU overhead. This improves the mechanics of offload without changing which blocks deserve persistence.

NVIDIA Magnum IO GPUDirect Storage removes intermediate staging through CPU DRAM. It connects storage directly to GPU memory. The py-kvcache researchers prototyped this path using KvikIO bindings to cuFile, while noting that removing a CPU copy does not eliminate the total bandwidth constraint. NVIDIA’s cuObject API, generally available in January 2026, extends GDS to S3-compatible object stores.

RDMA addresses the distributed transfer path. It reduces CPU involvement when blocks cross nodes. Novita AI’s Pegaflow (2026) offloads KV to host memory or SSD and shares it through RDMA, with connectors for vLLM and SGLang. ObjectCache was prototyped on a 100 Gbps RoCE cluster using NIXL, Ceph RGW, and DAOS, demonstrating the object-storage path on that fabric. These mechanisms cut different transfer costs and work alongside placement policy, which is still needed.

Lossless compression before tier crossing and the constraint of output fidelity

Compression reduces the bytes that must cross a constrained boundary. It must cost less to process the blocks than it saves on transfer. SplitZip (2026) targets ultra-fast lossless KV compression for disaggregated serving, where a slow compressor would simply become the next bottleneck. BF16-specific lossless compression remains unresolved enough that codec choice requires evaluation against actual tensor distributions.

Lossless compression is the defensible production default when cache transport must not alter output. It preserves the stored state without adding an output-verification mechanism. General-purpose options such as zstd and LZ4 therefore need tests of how much they shrink blocks and how fast they process them. A smaller block is useful only if the full path becomes faster.

VeriCache (2026) demonstrates a legitimate alternative with additional machinery. It drafts generation from a lossy compressed cache and verifies against the full cache, reporting up to 4× higher throughput while preserving full-KV output equivalence. Verification is the essential distinction. A deployment must establish that its draft acceptance rate justifies the added compute and complexity, rather than treating lossy cache compression as a transparent storage substitution.

What MLPerf Storage v3.0’s KV cache benchmark means for inference storage evaluation

MLPerf Storage v2.0, announced August 4, 2025, added checkpointing tests. MLPerf Storage v3.0, released September 1, 2026, extends coverage to inference with dedicated KV cache and vector database tests. KV retrieval now has a publicly reproducible comparison workload.

The KV test creates an opportunity for cluster-specific evaluation. Local NVMe, remote DRAM pools, and object storage can be compared with the same tests. Results can also be submitted for open-source storage. Teams evaluating offload can use the benchmark to test their assumptions about transfers, though they still need workload profiling to set admission policy.

NVIDIA’s AIStore reported approximately 3.97× UNet3D training I/O and approximately 3.99× Llama 3 1T checkpoint recovery throughput when its cluster size quadrupled. The dedicated KV test is the relevant opportunity for comparable inference-storage evidence.

Assembling the full pattern into a coherent session-aware serving architecture

The practical decision path starts with the reuse pattern. Content-addressed blocks with S3 backing, as in WombatKV, are a good fit for long, stable system prompts or RAG documents. Frequently diverging multi-turn sessions favor radix-tree prefix caching with copy-on-write, as in one serving system. The selected representation establishes what can be shared and which repair or compatibility costs follow.

Lifetime policy comes next, with a clear model of who owns the blocks. Active sessions call for session-pinning, while inactive state calls for recomputation-cost-weighted eviction. Admission then applies the fetch-versus-recompute threshold at each boundary. A content-addressed shared store needs eviction policy suited to shared blocks on persistent storage.

Substrate selection follows the deployment boundary. Local NVMe with Tutti/gio_uring fits the single-node path, while a Pegaflow-style RDMA store supports multi-node sharing. ObjectCache with cuObject addresses capacity-bound deployments with substantial long-context reuse. Lossless compression belongs before tier crossing, with VeriCache-style draft-verify reserved for cases where throughput requirements exceed lossless bandwidth and verification overhead is acceptable.

The break-even point will move as prefill improves. Short contexts may increasingly favor aggressive persistence bypass, while long contexts with high prefix reuse can retain a retrieval advantage. Architectures should expose a configurable threshold and revisit it as models and hardware change. Huawei’s Unified AI Gateway model points toward joint model routing and KV management, where routing accounts for reusable state.

Sources

  1. REKV: Cross-Precision Model Switching with Persistent KV Cache Reuse for LLM Serving
  2. LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs
  3. CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
  4. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  5. Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management
  6. ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
  7. Recency/Frequency Adaptive KV Caching for Large Language Model Serving
  8. Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading

More in Long-Context Serving