Per-Request KV Cache Budget Enforcement as the Primary Cost Lever in Long-Context Serving
Capping KV memory per request is the biggest cost lever in long-context serving.

Long-context LLM serving infrastructure needs page-based KV allocation, per-request memory budgets, and storage fast enough to support cache recovery when offloading is used. Enforcing a KV cache budget for each request is the main way to control cost because cache memory grows linearly with context length, and one uncapped request can crowd out a batch. Adaptive capacity, learned head-wise allocation, and risk-controlled eviction make that lever usable without treating output quality as an average-only constraint.
Why KV memory is the binding constraint in long-context serving
KV cache memory determines concurrency at long context because it can exceed every other memory cost in the system. For a large long-context model in BF16, one request at extended context consumes tens of gigabytes of KV. The cache can outgrow the model weights themselves. At the longest supported context lengths, weights and a single request’s cache can exhaust a high-memory GPU’s VRAM, leaving no room for another concurrent user.
Batching makes this a fleet-level allocation problem. Several long-context requests can require tens of gigabytes for KV alone, with weights pushing the combined footprint onto multiple high-memory GPUs. Architectural techniques including GQA, MQA, MLA, and CLA reduce per-token KV size but do not eliminate linear growth. An uncapped request can therefore crowd out the entire batch, making concurrency depend on the longest request in flight.
What per-request budget enforcement actually does mechanically
Budget enforcement sets how much KV memory each request can keep before or during generation. PagedAttention supplies the underlying allocation unit. It assigns fixed pages containing small token blocks on demand and frees them when requests end, avoiding contiguous pre-allocation. Reference counting lets requests share pages, so common system prompts or conversation history can be stored once and reused through prefix caching. Enforcement then controls two separate decisions, which are the states retained within an allocation and the total capacity assigned to a request.
Reuse and memory controls already have concrete implementations. VeriCache, built on vLLM and LMCache, achieves up to 4× higher throughput than full-KV inference on long-context decoding and up to 2× on remote prefix caching, with identical outputs. The kvcached project integrates memory limits with SGLang and vLLM through a production-control API. As of April 2026, it added automatic prefix caching for vLLM and RadixCache for SGLang. vLLM GitHub RFC #37003, opened March 2026, proposes a Context-Aware KV-Cache Retention API for Prioritized Evictions. These developments formalize enforcement, but most existing eviction methods share a restrictive assumption. They set total capacity ahead of time and change only which states they keep.
Why static per-request budgets break down under reasoning workloads
GrowPage identifies two sources of demand mismatch. The smallest KV budget that is enough for one request may be too small for another, and attention concentration changes within a request during decoding. Reasoning makes both kinds of change more pronounced. Across chains of thought spanning thousands to tens of thousands of tokens, a model explores, verifies, and revises, so its later memory needs are not predictable at admission. A fixed budget cannot keep up with that changing working set.
A larger fixed allocation protects requests that need more memory but sets aside more than other requests need. Reducing it improves concurrency at the risk of evicting reasoning-critical history. This is a structural mismatch. Methods that change retained states without changing capacity also leave PagedAttention’s incremental page-allocation capability unused. The resulting costs are asymmetric, because excess capacity displaces other users, while insufficient capacity damages output quality in ways aggregate scores can conceal.
How dynamic capacity allocation addresses the demand-mismatch problem
GrowPage makes total per-request capacity a runtime variable. It keeps a long-timescale query summary of attention behavior that persists and a short-timescale summary of recent behavior. How their attention working sets compare shows how demand is changing. At each capacity boundary, the policy either compresses historical KV within the existing allocation or acquires another physical page. The decision fits directly into PagedAttention’s memory manager, with no need for another allocation abstraction.
GrowPage’s 2026 paper reports substantially higher average throughput for a distilled reasoning model than full-KV inference, with pass@1 accuracy within a fraction of a percentage point of the full-cache baseline. Requests that need less memory free pages for those that need more, with no need to negotiate capacity for each request at admission. Continuous batching, prefix caching, and CUDA Graph execution remain intact. The policy is drop-in compatible with vLLM-like stacks, requiring no serving-engine code changes.
Why average quality metrics hide the real deployment risk of aggressive eviction
Risk-Controlled KV-Cache Eviction distinguishes average degradation from the probability that a request degrades substantially. A small mean loss does not show that the chance of an unacceptable loss is small. Choosing a budget from average benchmark scores can let through policies that keep harming particular types of request while the overall result still looks good.
The paper evaluates SnapKV, AdaKV, DefensiveKV, and Layer-DefensiveKV across Llama and Mistral on LongBench and Llama on RULER. The same reliability contract certifies SnapKV at a high retention rate on LongBench but no tested compressed SnapKV policy on RULER, where the other three methods are certified. The acceptable policy depends on the workload. Even an empirical degradation rate below a target can fail finite-sample certification because the rate seen in the tests may not give the required confidence.
How risk-controlled eviction converts a reliability requirement into a memory operating point
The risk-control framework defines a material degradation event as a request whose task utility falls by more than a deployment-specified tolerance relative to full-KV inference on that same request. Deployment risk is the population frequency of those events. This gives reliability a measurable reference point. An operator specifies the tolerated degradation, target risk level, and confidence requirement instead of choosing a retention fraction and hoping its average score is acceptable.
The framework uses Learn-then-Test with an independent calibration set to evaluate a pre-specified sequence of policies. It returns the most aggressive policy certified before the first failure, with finite-sample guarantees. The underlying compressor does not need to change. The procedure can wrap SnapKV, AdaKV, DefensiveKV, or another eviction algorithm, making risk control an additive capability. When no compressed policy is certified, full-KV fallback preserves the contract. That fallback is a valid operating point that sets a firm limit on how much memory can be saved while meeting the reliability requirement.
How learned head-wise budget allocation improves on heuristic eviction rules
Even when total capacity can change, memory still has to be split across attention heads. Heads need different amounts of memory, so heuristic budgets can give some too much and leave too little for those the task depends on. SnapKV uses uniform budgets, PyramidKV applies layer-wise pyramid allocation, and DuoAttention imposes an optimization-based distinction between retrieval and streaming heads. Each approach prescribes structure before the task determines it. Such priors can prevent allocation from capturing more complex sparsity patterns.
LKV, or Learned KV Eviction, instead formulates compression as end-to-end differentiable optimization. LKV-H learns task-optimized global budgets across heads, while LKV-T derives token importance without materializing attention matrices. This removes the need for the scores that guide heuristic selection. AdaKV, for example, allocates head budgets using attention scores already required by a base method such as SnapKV, rather than introducing an additional query-key computation itself. LKV reports near-lossless LongBench performance at a very low retention fraction, with learned budgeting contributing more to fidelity than learned token selection. With the same total memory, task-driven allocation can support stricter enforcement than heuristic allocation.
What quantization and lossless compression contribute to the memory budget
Eviction controls retained history, while quantization and compression change the bytes needed to represent it. These are distinct levers with different fidelity consequences. Lossless compression introduces no KV approximation, making it appropriate when output fidelity is a hard constraint and further lossy reduction would compound eviction errors. BF16 KV tensors contain more invariant bits than FP16 tensors, giving lossless codecs more structure to exploit. That format-level advantage applies across models.
GAAC, the Group-Adaptive Additive Codec, exploits exponent redundancy in BF16 KV tensors to reduce storage and accelerate attention without approximation. IBP also exploits invariant-bit structure in BF16 weights and KV caches, although efficient GPU-initiated transfer is its dominant KV-cache benefit. These tools can compose with enforcement, with FP8 quantization reducing baseline size, lossless compression reducing the quantized representation, and risk-controlled eviction limiting retained states. Lossless compression preserves the representation it receives, rather than undoing approximation introduced by another stage.
What storage infrastructure does long-context LLM serving need?
Moving KV out of GPU memory loses its advantage if retrieval from the storage tier takes longer than recomputation. Unoptimized SSD-backed retrieval can produce excessive time-to-first-token because I/O is serialized and filesystem overhead dominates the path. Unified KV Pooling addresses these bottlenecks and demonstrates substantially lower time-to-first-token and blocked I/O time than existing offloading techniques.
Disaggregated prefill changes where KV is produced and consumed. vLLM 0.8+ and SGLang support dedicated prefill GPUs that send cache to decode GPUs over RDMA or NVLink. Mooncake established a cache-centric disaggregated architecture and received the Best Paper Award at USENIX FAST '25. RDMA with GPUDirect Storage enables transfers between GPU memory and storage without CPU involvement, reducing latency and copy overhead. Lossless compression complements that path by reducing transferred bytes.
ObjectCache demonstrates layerwise KV retrieval from an S3-compatible object store, extending the storage tier beyond local devices. Its compatibility layer permits backend changes without inference-code changes. Latency tolerance still determines the economic choice. AWS instance-selection experiments show that a lower-cost instance without offloading can outperform offloading on a higher-cost instance for latency-sensitive online workloads, while offloading saves cost for offline workloads. Storage architecture must be tested for how well it serves the workload as well as how much it can hold.
How the emerging benchmark landscape is making budget enforcement decisions measurable and comparable
MLCommons announced MLPerf Storage v3.0 on September 1, 2026, adding a KV Cache test for LLM inference cache read/write operations. The release also has tests for training, checkpointing, and vector-database workloads. Its S3 object storage access layer builds on the existing POSIX layer, so both interfaces can be tested on the same workloads. This makes cross-architecture evaluation more practical. Budget policies that depend on cache recovery now have a public benchmark against which to assess their storage assumptions.
Everpure's FlashBlade//EXA achieved 1,623.4 GiB/s, the round’s largest KV cache result. HPE Cray K3000 achieved over 8,700 tokens per second and over 151 GiB/s across 16 clients in KV caching. Checkpoint results measure a separate part of the storage workload. Everpure achieved 877.5 GiB/s write and 833.0 GiB/s read, the largest checkpoint numbers of the round. Azure Managed Lustre achieved up to 642 GiB/s write and 490 GiB/s read. HPE Cray K3000 achieved over 83 GiB/s read across 8 clients.
Samsung's PM1763 NVMe SSD demonstrated single-device KV cache serving at competitive token throughput with concurrent read and write bandwidth. The benchmark thus spans individual drives and large parallel filesystems. For teams evaluating KV offloading, the practical next step is to require MLPerf Storage v3.0 KV cache results. Per-request budgets govern GPU memory, but enforcement that relies on offloading is only as effective as the storage tier is fast, and MLPerf Storage v3.0 provides the standard way to test that dependency.


