Eviction Policy Design When KV Cache Growth Saturates GPU Memory in Serving Clusters
How to choose among retaining, offloading, and discarding KV blocks.

Efficient KV cache management in production LLM serving needs a policy for where to place blocks as well as a rule for dropping tokens. When GPU HBM fills, blocks should be retained, offloaded, or discarded according to token salience, recomputation cost, and retrieval latency. Those three inputs link the eviction policy to how much memory it uses, output quality, and serving latency.
Why KV cache growth creates a placement problem, not just an eviction problem
KV cache memory grows linearly with context length across layers, heads, and sequence positions. As context windows expand from thousands to millions of tokens, the cache can become the dominant HBM consumer and constrain batch size before compute saturates. Cached state therefore needs a placement beyond the model when it no longer fits beside it.
A drop policy removes cached state, while offloading preserves it below HBM for later retrieval. Treating those operations as interchangeable can force recomputation even when a DRAM transfer would have been cheaper. But moving every displaced block leaves out scoring overhead and the delays that slower tiers add. Eviction policy design must therefore treat token salience, recomputation cost, and tier-retrieval latency as first-class inputs.
The memory hierarchy an eviction policy must reason about
The SIGCOMM 2026 CXL paper describes a six-tier hierarchy comprising GPU HBM, CPU DRAM, CXL memory expansion, local NVMe, RDMA network or object storage, and a parallel filesystem. The paper places HBM at the highest-bandwidth end with sub-microsecond access and DRAM in the microsecond range. Each destination also changes how quickly displaced blocks can return to active use.
The same paper describes a CXL memory expander with an on-device DMA controller at roughly 64 GB/s and around 500 ns. It presents CXL as a staging tier between DRAM and NVMe once DRAM fills. The paper places RDMA network or object storage between NVMe and a parallel filesystem, reporting 50 GB/s at 1–100 µs. Local NVMe also has limits on endurance and write-bandwidth because KV traffic is continuous and write-heavy.
NVIDIA cuObject, shipped in January 2026, formalizes an RDMA path between S3-compatible object storage and GPU memory. NVIDIA NIXL, announced in March 2025, provides an open-source library with RDMA and GPUDirect backends integrating into vLLM, SGLang, and Ray. These paths make it practical to access lower tiers, but leave open the choice of which blocks to retrieve. That decision starts with a salience signal.
What makes a token "worth keeping" — the salience signal design space
CacheCraft, published on arXiv in May 2026, identifies the scorer family as the load-bearing choice in eviction design. Its decomposition finds that changing the scorer improves accuracy by orders of magnitude more than changing the chunk-selection structure does. The main design choice is therefore what evidence a policy accepts as a sign of future usefulness.
Attention-pattern heuristics take the attention a token has already received as that evidence. H2O balances heavy-hitter tokens with recency, while SnapKV pools the attention received over a recent observation window. These methods are model-agnostic and single-pass, but a single statistic can become brittle when very few tokens survive. Structure-aware methods such as KeyDiff and Knorm instead use geometric properties of KV entries to preserve representational diversity. They capture coverage that attention-only scoring can miss, without providing a unified objective.
CAPKV, presented at ICML 2026, derives a closed-form mutual information objective under a linear-Gaussian surrogate of attention. It interprets attention-based and structure-based methods as approximations of a shared capacity-maximization principle and uses statistical leverage scores to preserve information in the retained subset. This supplies a common lens. In the prefill-stage regime, these scoring approaches remain single-pass and training-free, so their scoring cost is paid once.
Static eviction heuristics and where they break
Sliding windows retain only the most recent tokens with zero scoring overhead. They work acceptably for short, single-turn requests when recency tracks relevance. Relevant tokens may lie anywhere in the context, so a fixed window can discard the evidence a later answer needs.
Prefix-cache LRU operates at a different granularity. vLLM uses LRU eviction of KV page blocks, while SGLang organizes shared prefixes in a radix tree and evicts unused branches LRU-style under memory pressure. In the common case, these policies save recomputation when a shared prefix matches. However, an agent waiting for a tool result can appear cold even though it will resume the same session. Evicting its prefix turns that temporary pause into a recomputation cost.
UNISON, published on arXiv in September 2026, analyzes this lifecycle failure on agent traces. It finds that recency, timeout, and estimated-return-time policies remain above a Bélády oracle in AMAT, extra prefill, and TTFT. A fixed timeout cannot distinguish abandonment from a live tool wait. Static policies therefore miss two separate dimensions, namely within-context token salience, which requires a scorer, and cross-turn session lifecycle, which requires a model of return.
Dynamic token-level scoring policies that adapt the eviction budget at runtime
Dynamic policies address the first gap by computing importance from runtime signals. Some change which tokens are kept, while budget-adaptive methods also change how many are kept. CONF-KV, from CMU in 2026, converts the next-token distribution into a scalar confidence score at each decode step. Uncertainty preserves more context. Confidence permits more pruning. Unlike a prefill-only scorer, this policy revisits the retention budget during generation.
CONF-KV reports substantially better accuracy at retrieving targeted information from long contexts than sliding window and H2O baselines. It also retains the vast majority of full-KV success on VisualWebArena agent tasks at substantially lower peak memory. Those results support confidence-gated budgeting in the tested settings. They do not establish that one scorer dominates across workloads. H2O remains an established attention-mass baseline for testing how well this added runtime signal works.
K-VEC, introduced in 2026, addresses overlapping eviction scores across heads and layers through a coverage-aware diversity criterion. Its concern is lost coverage from repeatedly dropping the same tokens. CacheCraft takes another route by using automated scorer search. Its discovered policy, FRC, combines local attention received normalized by its maximum absolute value, neighborhood attention density, and KV-head maximum salience with fixed weights of 0.55, 0.30, and 0.15. The reusable contribution is a search method that can adapt scorer programs to a model and workload without redesigning selection structure.
Session-lifecycle-aware policies for agentic and multi-turn workloads
Token-level scoring does not predict whether an idle session will return. For agent loops, the session becomes a placement unit because its growing prefix persists across tool waits. UNISON shows monotonically growing prefixes in coding and web-agent traces across sessions of varying length. The pause is part of the execution structure. Its expected duration can provide information that neither token attention nor simple recency encodes.
UNISON combines Survival-Penalty Eviction for Agent Return-gap, or SPEAR, with Tiering in Idle-window DMA Events, or TIDE, on a near-memory scheduler. SPEAR uses a gap-average and turn-indexed hazard function to rank sessions by how likely they are to return. TIDE treats the observed tool wait as a DMA budget, migrating sessions unlikely to return soon into slower storage while they are idle. The two policies share one live ranking. That coupling keeps the decision about who leaves the fast tier consistent with the decision about who migrates.
The near-memory design makes scheduling part of the memory architecture. A session can be inactive yet worth preserving, and its idle interval can absorb migration work. Semantic-aware prefix-cache eviction represents another lifecycle-related direction when shared-prefix reuse provides the main savings opportunity.
Tail-latency-aware eviction in production serving frameworks
Eviction decisions also affect which requests absorb recomputation delays. A policy can reduce average memory pressure while worsening P95/P99 time to first token if it removes blocks needed by the longest queued requests. Throughput and tail latency are separate objectives. The serving scheduler determines how a local cache decision propagates into the request latency distribution.
T-LRU, presented at NeurIPS 2025, extends prefix-cache LRU with two queues and a user-specified TTFT latency SLA. It preferentially evicts blocks that are provably safe to remove without violating that SLA, targeting P95/P99 TTFT. Its queues distinguish frequently referenced blocks from once-referenced candidates for fast eviction. The vLLM proposal in 2026 closed without merging. A submission to SGLang at the same time suggests the design can work across frameworks, but does not prove it has been adopted.
An offload connector addresses a different part of this path. It lets KV state move beyond the serving framework’s GPU cache, while the eviction policy and scheduler decide when to move or remove it. A transfer path alone is not a latency guarantee. Tail-aware scheduling remains necessary because preserving capacity does not establish which queued request will bear the retrieval cost.
Offload-aware policies that use tier latency as an eviction input
Offload-aware policies connect salience to the cost of recovering displaced state. A low-scoring token in a short context may be cheapest to drop, while the same score in a long context may favor DRAM offload because recomputation is more expensive. The comparison weighs salience and recomputation FLOPs against transfer cost from each available tier, including the constraints of the path back to HBM.
KVDrive, introduced in 2026, addresses the problems caused by treating NVMe as simple DRAM overflow. That arrangement can trigger frequent high-latency transfers and decoding stalls. KVDrive instead tracks per-block access frequency and explicitly schedules promotion and demotion across DRAM and NVMe. Tier movement becomes part of the policy. Coordinate placement and retrieval when choosing the eviction rule.
Storage I/O engineering changes which placements are viable. GPUDirect Storage and Tutti’s GPU io_uring reduce NVMe retrieval overhead toward near-zero CPU involvement, making cold but potentially reusable blocks more practical offload candidates. Remote storage introduces another retrieval path through S3-compatible interfaces and RDMA. Lossless compression can keep output quality the same while cutting the amount of data stored. None of these mechanisms replaces the cost comparison, but each changes the transfer side of it.
How can KV cache memory be managed efficiently in a production cluster?
Short-context, homogeneous, high-throughput requests can use sliding windows or LRU with prefix caching, including SGLang’s radix-tree approach. In that regime, offload overhead may not be justified. Long-context reasoning and document QA instead favor salience-aware retention, using H2O, CONF-KV, CAPKV, or FRC, with DRAM offload for lower-scoring blocks. Accuracy is the governing objective in that case. Which scorer to use depends on whether attention mass, confidence, information preservation, or a searched signal works best for the workload.
Multi-turn conversations add session-specific idle thresholds and tail-latency constraints. T-LRU offers a design for the latter, but its closed vLLM proposal should not be treated as a merged feature. Agent workloads with structured tool waits call for the SPEAR-and-TIDE design class, where proactive migration uses the wait interval to preserve a returning session. Multi-agent workloads with long shared prefixes also raise coverage and paging questions. K-VEC addresses cross-head and cross-layer overlap, while asymmetric granularity paging becomes relevant when shared and private prefix lengths differ substantially.
A token scorer can govern HBM retention, a lifecycle signal can govern session migration to DRAM, and a KVDrive-class layer can coordinate NVMe promotion. A tail-aware scheduler then constrains which evictions can proceed without P99 spikes. Each policy operates at a different granularity. Their interfaces must preserve those distinctions instead of treating every memory-pressure event as the same eviction problem.
Benchmarking should precede tuning of the combined policy. MLPerf Storage v3.0’s KV Cache test gives teams a workload for public comparisons of storage-tier offload, while RULER lets them compare token-level accuracy. The 2026 work described here is changing policies fast, but salience signal × recomputation cost × tier latency remains the framework for testing them over time.
Sources
- UNISON: A Co-Designed Near-Memory Scheduler of Session KV Residency for LLM Agents
- Discovering KV Cache Eviction Policies via LLM-Guided Program Evolution
- Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
- Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
- CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
- Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading


