KV Cache Compression Techniques for Long-Context LLM Serving
Quantization and pruning trade memory for speed in different ways depending on your workload.

Model weights used to set the ceiling on how much memory an LLM deployment needed<sup>1</sup><sup>2</sup>. In long-context serving, that ceiling moved. The key-value cache is the record of every token's key and value vectors at every layer, and when a deployment runs at production context lengths, this cache can claim more memory than the weights themselves. The cache grows linearly with input length: each new token adds a key and a value at every layer, so memory pressure builds with both how long the conversation gets and how many requests run at once. Decoding itself is bound by how fast data moves, not by how fast the GPU can compute. When the model generates a token, it has to attend over the entire cached sequence so far, and each step pulls a growing block of KV state out of HBM, so inter-token latency climbs as context grows instead of staying flat. KARA's own test case for Qwen3 at FP16 precision shows that holding a long context in memory takes tens of gigabytes, before a single byte of model weight enters the count. Under memory-constrained concurrent serving, this growth has a direct operational consequence: incoming requests sit in a queue waiting for memory to clear, and throughput collapses under the wait.
Inter-token latency grows with context length because decoding is memory-bandwidth-bound, not compute-bound. Each generated token requires the model to attend over the entire cached sequence, pulling a larger block of KV state out of HBM with every step, so the data-movement cost rises linearly as the conversation grows. In production, the practical fix is KV cache compression, but the choice of method and the metric used to evaluate it both matter: quantization (FP8 or INT4) cuts bytes transferred per attention step and directly attacks the bandwidth bottleneck, while eviction methods like SnapKV deliver the strongest throughput gains under concurrent load by permanently removing low-salience cache entries. Compression ratio alone does not predict whether latency or throughput actually improves; realized compression, mean output throughput, mean time-to-first-token, and task quality all have to be measured together against the specific workload the deployment will face.
How the compression problem divides into three structurally distinct families
Three techniques get grouped under the label "KV cache compression," and they do three mechanically different things. Quantization lowers the numerical precision of the values already stored, changing how a value is represented but not which values survive. Token eviction, often called pruning, removes KV entries for chosen positions outright, so which states remain in the cache changes, but the survivors stay at full precision. Merging sits between the two: instead of deleting low-value entries, it folds several of them into one representative entry, trading a clean deletion for an approximation. The benchmark built by Nikita Agrawal and Ruben Mayer at the University of Bayreuth organizes its evaluated methods along exactly this split: KIVI and TurboQuant represent quantization, SnapKV represents pruning, and CaM represents merging, all run on a shared evaluation harness so the comparison is apples-to-apples for the first time. KVDiagnosis pushes the taxonomy further, sorting 25 representative methods into five mechanism families: token eviction and position selection, budget allocation and hierarchy, query-aware access, chunk, semantic or learned state, and tensor or representation compression. Even methods inside the same family can be transforming different parts of the cache, so the taxonomy is not just a naming convention. It has a direct practical cost: a compression ratio quoted for one method is not comparable to a ratio quoted for another, because "50% compression" might mean half the token positions are gone, half the key channels are gone, or every retained value now sits at half the bit-width. None of this means the three families are deployed in isolation. Production systems frequently compose them, running quantization on top of an already-evicted cache, or applying merging after a token-scoring pass has identified which entries are expendable.
Why does inter-token latency increase as context grows, and what does quantization cost in long-context serving?
Quantization is the most direct way to cut the memory-bandwidth cost that dominates long-context decoding, and it does the least structural damage to what the cache represents. Dropping stored key and value tensors from BF16 down to FP8 or INT4 cuts the bytes that have to move per attention step by half or by three-quarters, which goes straight at the bandwidth bottleneck described above. In the Bayreuth benchmark, a 2-bit quantization method paired with a small residual window of full-precision recent tokens produced the most stable quality of any method tested, across both models evaluated. It's consistency across models that matters, since that matters more for an operator choosing one method to run everywhere than a technique that peaks on one benchmark and dips on another. TurboQuant sits in the same benchmark as a second quantization representative, and because both run under one harness, the comparison can hold across models, datasets, and memory budgets at once.
Quantization's failure mode is specific to the KV cache and doesn't show up when quantizing model weights. Weight quantization is static: the weight values are fixed once, and the error introduced at that one step stays bounded. KV cache quantization is dynamic. Every token's key and value gets quantized as it's written, and every later token attends over that quantized value during decoding, so the error introduced at step one keeps participating in every attention computation that follows. That error compounds rather than staying flat, and the compounding gets worse as the sequence gets longer: a longer sequence means more decoding steps, so each quantized value gets used more times, giving its error more opportunity to propagate forward. That runs against what you'd expect, because quantization schemes usually get validated on short inputs, where the accumulation never has room to build.
PackKV, presented at IPDPS 2026, builds a lossy compression scheme around the actual statistical structure of KV cache tensors rather than applying a generic quantization scheme built for weights. It reaches a higher memory reduction than state-of-the-art quantization methods while holding the same accuracy floor, and it delivers throughput gains for both the key and value matrices on A100 and RTX Pro 6000 GPUs, cutting the memory-bandwidth demand while speeding up the matrix-vector multiplications that dominate decoding cost. It also has to handle a cache that keeps growing during generation, something many simpler quantization schemes just assume away. The Bayreuth benchmark's broader finding applies here too: a higher compression ratio doesn't reliably buy better throughput or lower latency on its own. The bandwidth savings only pay off if they actually reduce stalls, and whether they do depends on the hardware and the access pattern, not on the ratio reported in a paper.
Token eviction's throughput gains and its effect on reasoning coherence
SnapKV delivered the strongest long-context throughput of every method tested in the Bayreuth benchmark, which makes it the clear leader on the single metric that matters most once a system is serving at scale. The mechanism is straightforward: SnapKV and similar eviction methods score every token by its accumulated or expected attention weight, then permanently delete the KV pairs for whichever positions score lowest. Those pairs are gone. They cannot contribute to any future attention computation, which is a stronger and riskier move than quantization's precision cut.
Hard deletion carries a cost that doesn't appear in throughput numbers. The researchers describe it as a region wipe-out effect: because global top-k selection naturally keeps the highest-salience individual tokens, the "heavy hitters," it has no mechanism for protecting a reasoning span that's spread across many tokens, none of which individually scores high enough to survive, but which collectively carries the logic a later step depends on. That span can get removed in its entirety. Yang and Shen trace this to three distinct downstream failures. Problem Drifting is when the model quietly changes a constraint from context that's no longer there to check against. Premature Overturning is when a valid intermediate conclusion collapses because the evidence that supported it got evicted. Repetition Collapse is when the model, starved of context, falls into a degenerate loop. The failure is sharpest on reasoning-heavy workloads, mathematical benchmarks like MATH500, AIME, and AMC 2023, along with multi-step code tasks, where the chain-of-thought steps depend causally on each other in ways that an attention score, computed token by token, cannot fully capture.
Eviction has a second, system-level problem that this sliding-window method identifies separately from the region wipe-out effect. Many eviction policies trigger compression only when the cache crosses a size threshold, and under memory-constrained concurrent serving, those compression events arrive in bursts that hit different sequences at different times. That unevenness can reduce output throughput rather than protect it, cutting against the reason eviction gets chosen. KARA's answer targets both problems at once. It compresses only a recently generated sliding window rather than the whole cache, scores KV pairs within that window using bidirectional attention, and uses a Token2Chunk module to expand a subset of the selected pairs into chunks of flexible size rather than forcing them into fixed boundaries, which is what preserves semantic continuity instead of cutting through it. The result holds accuracy close to the uncompressed baseline while keeping only a fraction of the full cache in memory, and KARA's KvLLM serving framework outperforms vanilla vLLM on throughput under memory-constrained concurrent load.
AMS goes after the same underlying problem, but from the allocation side rather than the policy side. Instead of letting every token compete individually for survival under one global top-k cutoff, this allocation-side method partitions the cache into regions based on how attention mass is distributed across it, and it guarantees each region a minimum retention quota before any local token selection runs inside that region. An EMA smoothing mechanism keeps segment boundaries from jittering between decoding steps, and the same history-aware smoothing also damps quota jitter across repeated compression events over a long generation. AMS is built to sit on top of existing scoring methods, including TOVA, Expected Attention, KeyDiff, R-KV, and TriAttention, and it's compatible with vLLM's PagedAttention, so it functions as a layer added to an existing eviction pipeline rather than a replacement for it. KARA and AMS solve different halves of the problem: KARA targets the bursty threshold policy and the rigidity of fixed chunk boundaries, while AMS targets the region-level allocation failure that causes whole regions of the cache to be wiped out.
Why merging's quality gains vary by workload
Merging takes a position between quantization and eviction: rather than deleting a low-importance KV entry, it folds several such entries into one aggregate representative, keeping a degraded signal in place of no signal. In theory, that should soften the worst damage hard eviction causes, because nothing is ever fully erased from the cache. In the Bayreuth benchmark, CaM, the merging representative, backs that theory up on certain workloads: on selected QA tasks, it produces large gains and, under the right conditions, beats both the quantization and the eviction methods it was tested against.
The same benchmark shows CaM is also the least predictable of the three families. Its quality and its realized compression ratio both swing significantly depending on the task, so the same memory budget that performs well on one workload can perform quite differently on another, whether that's multi-document QA, summarization, or few-shot learning. The swing comes down to how redundant the merged entries actually are in a given task. QA formats with heavy lexical and semantic repetition give merging an easy job, because the entries being folded together really are similar, and collapsing them loses little. Tasks that depend on retrieving one precise fact buried somewhere in a long document give merging a much harder job, because that same collapsing operation destroys the distinctiveness that made the fact findable. Realized compression also shifts for a related reason: the number of tokens eligible to be merged depends on how many pass the importance threshold, so a document packed with high-salience tokens leaves fewer merge candidates and yields less actual compression even when the budget setting hasn't changed.
KVDiagnosis supplies the evidence for why this variability is easy to miss in standard benchmarking. Its correct-to-wrong methodology isolates the specific cases where a compression method breaks an answer that the uncompressed model got right, rather than reporting one aggregate accuracy number that averages wins and losses together. Using that lens, KVDiagnosis found that coverage failures, where the compressed cache gave the model only partial or low evidence coverage of what it needed, account for 63.2% of all compression-induced errors across the methods it studied. Lost evidence explains CaM's variance: aggregate accuracy can look stable while individual examples fail for the same underlying reason at very different rates depending on the task. The operational consequence follows directly: before putting a merging-based method into production, the workload distribution it will actually face needs to be profiled in advance, because a benchmark score from one task type says little about how the method will behave on another.
Compression ratio's failure to predict throughput or quality
The Bayreuth benchmark's central finding, not a side note, is that compression ratio by itself fails to predict end-to-end performance. A method can report an impressive reduction in cache size and still fail to speed anything up, because a high compression ratio only pays off if the access pattern stays cache-friendly and if whatever decompression happens at inference time doesn't eat back the bandwidth it saved. Reporting one number, the ratio, hides whether any of that held.
Four metrics have to be read together to know what a compression method actually does in production. Task quality is accuracy on the downstream task itself. Mean output throughput is tokens generated per second under concurrent load, and it tells you how many users a deployment can serve at once. Mean time-to-first-token captures prefill cost, and that matters separately from steady-state decoding speed. Realized compression ratio is what the method achieves while actually running, which can diverge sharply from the nominal budget it was configured with. That divergence is not a rounding error. For an eviction method like SnapKV, realized compression depends on how many tokens actually fall below the eviction threshold, so a low-entropy document packed with high-salience tokens gets evicted less than its budget allows. For a merging method like CaM, realized compression tracks the redundancy structure of the specific document being processed. Both cases mean the number printed on a configuration file is not the number the system delivers at runtime.
KVDiagnosis adds one further layer to this discipline: aggregate task quality is itself a number that can mislead. Its C→W methodology, run across a benchmark that found 12,520 rows where the uncompressed cache got an answer right and the compressed setting got it wrong, shows that two methods can post nearly identical aggregate scores even as they fail on completely different examples for completely different reasons. Under the same fixed diagnostic rules, most of those C→W failures trace back to low or partial evidence coverage, the same coverage failure pattern that explains CaM's workload sensitivity. Taken together, the four-metric discipline and the C→W diagnostic point to the same conclusion: choosing a KV cache compression method by its headline compression ratio is choosing blind, and the only way to know what a method will do to a specific workload is to measure quality, throughput, time-to-first-token, and realized compression against that workload directly.
Sources
- PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
- Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving [Experiment, Analysis & Benchmark]
- Adaptive Mass-Segmented KV Compression for Long-Context Reasoning
- KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
- KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression


