Llama.cpp KV Cache Quantization and Memory Tradeoffs

KV cache size is not a footnote to model size. Past a certain context length it becomes the dominant memory cost on the GPU, and llama.cpp gives operators three ways to shrink it: leave it at f16, drop it to q8_0, or push it down to q4_0. Each choice trades memory for decode speed and, in some cases, output quality, and the tradeoff is measurable. Model weights are fixed once loaded. The KV cache grows with every token generated, every concurrent request, and every batch slot in flight. The memory budget that looked fine at load time can quietly blow up mid-session.

The formula behind that growth is simple enough to write on a napkin: 2 × number of layers × number of KV heads × head dimension × sequence length × bytes per element. Total parameter count barely matters here. What matters is how many KV heads the architecture keeps and how deep the model runs. A model with fewer parameters but more layers and more attention heads can carry a heavier cache than a bigger model built leaner. Llama-3-8B running at 4.9 GB in Q4_K_M weights adds another 4.0 GB of FP16 KV cache once context hits 32K tokens. That's 9.4 GB combined on a card with 8 GB of VRAM. The math doesn't fit, and no amount of clever batching changes that.

Beyond a certain context length, in fact, the KV cache routinely outweighs the model itself in memory terms. It's the half of the GPU budget most people forget to plan for, because during short-context testing it barely registers. Then context length climbs and the bottleneck shifts from raw compute to memory bandwidth: every generated token has to read the accumulated KV tensors off the GPU, and that read, not the matrix multiplication, is what slows decoding down. Shrinking the cache, then, isn't purely an exercise in fitting within memory limits. It changes how much data has to move per token, which is a direct lever on throughput.

How attention architecture determines KV cache size before quantization

Before any quantization flag gets set, the attention mechanism itself has already decided how expensive the cache is going to be. Standard multi-head attention (MHA) stores a separate key and value tensor for every head in every layer, and it's the most expensive baseline against which everything else gets measured.

Llama 3 doesn't use plain MHA. It uses grouped-query attention (GQA), running many query heads against just 8 KV heads, which cuts the cache substantially compared to a full MHA setup. That's why the 4.0 GB figure at 32K context already looks reasonable relative to older architectures: it's compressed before quantization even enters the picture.

DeepSeek-V2 pushed the idea further with multi-head latent attention (MLA). Instead of caching full key and value tensors per head, MLA compresses each token's hidden state down to a low-rank latent vector, caches only that, and reconstructs the keys and values via projection at attention time. The result is a dramatic cache reduction against a comparable dense architecture. At the same context length, the KV footprint under MLA is a fraction of what plain MHA would require.

Choosing between architectures changes the KV memory bill by a multiple, and that decision gets made before a single quantization setting is touched. Anyone picking between models for a memory-constrained deployment should treat attention variant as an infrastructure question first.

What llama.cpp's KV cache quantization flags do and what they require

Llama.cpp exposes cache type through two flags: --cache-type-k (-ctk) and --cache-type-v (-ctv), both defaulting to f16. Setting either one lower is where the memory savings start, but there's a hard requirement attached: quantized V caches are intended to be used with Flash Attention enabled ( -fa 1 / --flash-attn ). Operating without it may negate the memory savings through dequantization overhead on every attention pass.

The format options split into two families. The legacy block-wise formats, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0, split weight matrices into fixed-size blocks that share a single scale factor. The K-quant family (Q2_K, Q3_K variants, Q4_K, Q5_K, Q6_K) adds superblocks and extra per-block metadata to squeeze out better quality at a given size. For KV cache specifically, Q8_0 and Q4_0 are the two workhorses. Q8_0 is symmetric 8-bit, with quality close enough to f16 that it functions as a high-fidelity default. Q4_0 is 4-bit block-wise and the most aggressive setting commonly tested in practice.

Quantizing model weights is a one-time conversion done offline, before inference starts. KV cache quantization happens live, on every forward pass, on data generated during inference itself. The compression and decompression overhead gets paid per token, every single time, not once at load.

The -ctk/-ctv interface applies the same format to both keys and values. Keys and values don't have the same distributional shape, and the published research on this (covered later) treats that difference as significant. Mainline llama.cpp doesn't expose separate formats for each, so operators pick one setting and accept it as a blanket approximation across both tensors.

Memory savings measured: what q8_0 and q4_0 recover on real hardware

Numbers make this concrete fast. Llama-3-8B on an RTX 4060 with 8 GB of VRAM, running at 32K context: FP16 KV cache pushes total memory to 9.4 GB, which doesn't fit. Switching to Q8_0 drops the total to 7.4 GB, fitting with barely anything to spare. Dropping to Q4_0 falls the total to 6.4 GB, leaving 1.6 GB of headroom, enough room to run a small embedding model alongside it for a RAG pipeline.

Qwen2.5-32B tells a similar story at a larger scale. At 32K context, the FP16 KV cache alone reaches 8.0 GB, which is the entire budget of an 8 GB card before the model weights even load. Q4_0 KV cache is what makes short-context inference with a 32B model plausible on a single consumer GPU with partial CPU offload; without it, the option doesn't exist.

On a DGX Spark GB10 running Nemotron-3-Nano-30B-A3B at 128K context (llama.cpp build 8399, memory read from llama.cpp logs and free -h, since nvidia-smi reports memory as N/A on GB10's unified memory architecture), the KV buffer measured 768 MiB at f16, 408 MiB at q8_0 (a 47% cut), and 216 MiB at q4_0 (a 72% cut, 552 MiB recovered against the f16 baseline). At short and medium context lengths, that recovery looks close to free. The throughput cost hasn't shown up yet. The throughput cost appears once context stretches long, which is the next problem.

The throughput cost: decode speed degradation as context grows

On that same DGX Spark GB10 hardware, generation throughput barely moves at short context. Around 6K tokens: 44.7 tok/s at f16, 44.9 at q8_0, 45.0 at q4_0. Q4_0 is actually marginally faster there, a rounding-level difference that shouldn't be read as a real signal either way.

Stretch to 24K context and the gap opens up: 44.6 tok/s at f16 versus 39.7 at q8_0 and 39.3 at q4_0, putting q4_0 noticeably behind. At 110K context the gap turns into a real penalty: 38.0 tok/s at f16 against 25.0 at q8_0 and 24.0 at q4_0, a roughly 37% slowdown for q4_0 relative to full precision.

Prefill (prompt processing) shows none of this. Across all three context lengths tested, prompt throughput stayed essentially flat regardless of cache format: around 1,211, 1,207, and 1,206 tok/s at 6K context, and around 815, 810, and 813 tok/s at 110K. Quantization format simply doesn't touch prefill speed.

The mechanism explains why. Dequantization happens per generated token, and every token generated has to read and decompress the entire accumulated KV cache to attend over it. At 110K tokens, that's a large read made larger by the overhead of reconstructing quantized blocks on the fly. It's that the arithmetic doesn't get harder, but that more data has to move, and moving compressed data costs a decompression step every time. It's that more data has to move, and moving compressed data costs a decompression step every time.

Server-mode workloads add another wrinkle. On server-mode workloads with short generation lengths, the throughput cost of KV cache quantization is modest, because prefill dominates and prefill doesn't care about cache format. As generation length grows, decode makes up a larger share of total work, and the penalty from dequantization becomes more significant.

The throughput penalty, in short, isn't a flat tax. It scales with both context depth and generation length. Someone running short completions at moderate context pays almost nothing for quantizing the KV cache. Someone running long completions at 100K-plus context pays for it severely, and should plan accordingly rather than discovering it in production.

Accuracy loss and when it matters

Quality loss from KV cache quantization is real but frequently smaller than people expect. On Qwen 2.5 Coder 7B running a Q6_K base model, switching KV cache from f16 to q8_0 raised perplexity by just 0.0043 points, a difference that sits well within normal measurement noise for most applications.

The KIVI paper (ICML 2024) pushed the test much further, down to 2-bit KV quantization, and still found the damage limited: 44.27 on LongBench versus 44.52 for FP16, a gap of 0.56%, while cutting KV cache size by 87.5% against FP16 and total peak memory by more than half. That result matters because it shows aggressive bit widths can preserve most task performance, provided the quantization is applied along the right axes of the data, a point covered in more depth in the next section.

Model scale changes the risk profile. Smaller models lean more heavily on each individual parameter, so quantization error hits them proportionally harder. Above roughly 13B parameters, aggressive KV quantization tends to degrade quality less than it does at 7B or 8B, simply because there's more redundancy in the representation to absorb the error.

Context length compounds the risk independently. The longer the sequence, the more quantization error accumulates across the cache, and tasks that depend on precise retrieval from deep in a 128K context window are considerably more exposed than short-context generation tasks. Reasoning-heavy workloads carry their own specific risk: dedicated studies on reasoning-focused models find aggressive low-bit KV settings particularly damaging to multi-step reasoning chains, and the interaction between weight quantization and KV cache quantization matters too. Conclusions drawn purely from weight quantization studies don't transfer cleanly here.

The practical signal is straightforward. Q8_0 is a safe default across most workloads. Q4_0 deserves task-specific validation before deployment, especially for math, multi-step reasoning, and long-context retrieval, where the margin for error is thinner.

Decision framework: matching format to hardware, context length, and workload

The right choice depends on which constraint actually binds, not on picking a format because it's popular.

VRAM-constrained with short context, under roughly 16K tokens: Q8_0 recovers meaningful memory at close to zero cost in accuracy or throughput. It's the obvious default in this band.

VRAM-constrained with long context, 32K to 128K tokens: Q4_0 may be the only setting that lets the context fit. The throughput penalty here is the price of feasibility, so validate accuracy against the actual target task before committing to it in production.

VRAM-unconstrained, throughput-sensitive: f16 KV cache with Flash Attention enabled gives maximum decode speed with no dequantization overhead. Quantization buys nothing when VRAM was never the limiting resource.

Large model on a single consumer GPU, something like a 32B model with partial offload: Q4_0 KV cache at short context can make the model run at all instead of not running.

Flash Attention isn't optional once quantized KV enters the picture. Always confirm -fa 1 is actually active, because leaving it off turns a memory win into a speed loss. Generation length matters just as much as context length for server throughput: short completions at long context pay the dequantization cost lightly, while long completions pay it heavily, so the shape of the workload, not just the context window, should drive the format choice. And the current interface forces one format across both K and V, even though research covered next argues keys and values would ideally get different bit widths. Mainline doesn't expose that split yet, so a single conservative choice is what's available.

What research-grade KV quantization methods do beyond llama.cpp's block formats

Q4_0 and Q8_0 share a structural weakness: naive block-wise quantization doesn't account for how attention keys and values are actually distributed. High-variance outlier channels get clipped, and that clipping error hurts quality more than a uniform quantization scheme would suggest.

One method proposed at a research conference addresses this directly, treating keys and values differently based on their distributional properties, rather than applying a single uniform block format to both. The insight behind it is that key distributions stay fairly stable across tokens but vary meaningfully across channels, which is exactly the opposite of what a uniform per-token block format assumes. That approach targets extreme long-context inference that would be infeasible under standard block-wise schemes.

KIVI (ICML 2024) takes the asymmetry further still, applying low-bit quantization with different schemes for keys and values, matching each tensor type's actual distributional shape rather than forcing one scheme onto both. That asymmetric treatment is the core engineering lesson llama.cpp's symmetric -ctk/-ctv interface hasn't caught up to yet.

Rotation-based preprocessing is a different angle. Applying a Hadamard transform to smooth the KV distribution before quantization helps explain why plain Q4_0 tends to underperform rotation-smoothed methods at the same bit width: the rotation makes the data easier to quantize cleanly.

TurboQuant (ICLR 2026, from Google Research, Google DeepMind, and NYU) builds on that rotation idea directly. It applies a fixed random rotation to make KV vector coordinates more uniform, then quantizes using a precomputed scalar codebook, calibration-free and online, with no per-model fitting step needed before generation starts. TQ3, its 3-bit variant, hits an MSE of 0.034 with several times the compression against FP16. TQ4, at 4 bits, hits an MSE of 0.009 with several times the compression. It also supports a fractional 2.5-bit scheme that isn't a literal scalar format: for a 128-dimension head, 32 outlier channels get 3 bits and 96 regular channels get 2 bits, averaging out to 2.5 bits per channel.

As of August 2026, TurboQuant remains outside mainline llama.cpp. Upstream rejected the pull request in June 2026, and the only production inference engine currently shipping it is vLLM, from version 0.20 onward. A community fork, TheTom/llama-cpp-turboquant, exists for anyone who wants to run it against llama.cpp today, but it sits outside the mainline codebase, and there's no confirmed timeline for that changing.