vLLM PagedAttention Memory Allocation Under Concurrent Load

PagedAttention fixed a real problem: pre-allocating KV cache to a request's maximum context length wastes most of that memory, since most requests never fill the window. Kwon et al. showed that fix delivers 2 to 4 times the throughput of FasterTransformer and Orca at equal latency (SOSP 2023). But that gain rests on an assumption that concurrent load attacks directly: that the block allocator has enough free blocks to keep the batch full. Once you understand the block table, the allocator, and the eviction logic that manages them, the failure modes that appear under real traffic, from OOMs to stalled queues to silent accuracy loss, stop looking like mysteries and start looking like arithmetic.

How gpu_memory_utilization partitions memory, and the weight-vs-cache confusion that causes OOMs

The gpu_memory_utilization parameter in vLLM sets the fraction of total GPU memory the engine is allowed to touch for everything: model weights, activations, CUDA graph capture, and KV cache. It is not a KV cache knob. That distinction trips up more deployments than almost anything else in the stack.

Model weights load in full no matter what this parameter says. A 70B model in BF16 needs roughly 140 GB across the fleet at a utilization setting of 0.7 and at 0.95 alike. Lowering the parameter doesn't free up VRAM for the weights, it shrinks whatever's left over for the cache. On a model that already eats most of the GPU, that leftover can be small to begin with, so dialing utilization down just squeezes an already-thin cache budget even thinner. The default in vLLM 0.6.x remains 0.9, and no official change to 0.85 has been documented, but the fact that teams keep tuning around that default at all is itself a signal that 0.9 produces failures often enough to warrant attention in practice.

The multi-GPU case makes this concrete. On a multi-GPU 40 GB A100 setup running a 70B model with, setting utilization to 0.95 fails, because each GPU's KV cache budget is nearly consumed just holding its shard of the model. There's no cache left to allocate. Engineers hit this without warning because the math isn't obvious until you've written it out: four GPUs sharing the model still leaves each GPU carrying a sizeable share of the weights, and at 0.95 utilization on a 40 GB card, there isn't much room left for anything else.

Another wrinkle: vLLM claims its full allocated memory at startup, even while idle. That's by design, but it means the process blocks other GPU workloads from running alongside it and creates contention in any shared-cluster environment where multiple teams expect to timeshare a device. Startup, not steady-state load, is often where the first surprise happens.

Block budget cost of KV cache size per request at scale by precision format

The numbers get real once you run them for a single request. A Llama 3.1 70B request at 128K context uses roughly 42.9 GB of KV cache at BF16 precision. That figure comes straight from the shape of the model: 2 (key and value) times 80 layers times 8 KV heads times 128 head_dim times 131,072 tokens times 2 bytes per BF16 value. One long-context request, on its own, can eat nearly all the cache budget on a single GPU. Concurrency doesn't degrade gracefully from there, it hits a wall.

FP8 cuts that 42.9 GB roughly in half. NVFP4 on Blackwell hardware cuts it again by roughly another half. The format landscape by 2026 breaks into five rough tiers: BF16 as the baseline, FP8 or INT8 at about half the BF16 footprint, 4-bit approaches at about a quarter, 3 to 4-bit approaches like TurboQuant, and a 4-bit format on one newer hardware generation only.

TurboQuant, described on the vLLM blog on May 11, 2026, compresses down to 3 to 4 bits but dequantizes back to BF16 before running the actual attention computation. That's a different design than FP8, which quantizes both storage and the attention matmul itself. Most of TurboQuant's published results ran on small models over short contexts, so how it behaves under real concurrent pressure at scale isn't established yet. On the compression side, a vectorized CUDA kernel for INT8 quantization reportedly hits a speedup over CPU baselines that runs into the thousandfold range, with reconstruction error under 0.004 and attention score error under 0.1 even at large head sizes, a useful data point for teams weighing software-side compression on hardware they already own.

At long context lengths, per-token KV cost grows substantially. Whether a deployment can serve 10 concurrent long-context requests or 100 comes down to which precision tier the cache runs in, not to how many GPUs are in the rack.

The FP8 KV cache regression that invalidated a common headroom strategy

FP8's pitch is straightforward: half the memory of BF16 means roughly double the concurrent headroom, which is a good trade for anyone running long-context workloads at scale. Stress testing in 2026 found the catch. On Hopper GPUs, the FP8 Flash Attention 3 kernel suffered accumulation precision loss at long context lengths. On a 128K needle-in-a-haystack test, accuracy fell from 91% at BF16 down to 13% at FP8, a collapse traced to imprecise FP32 accumulation inside the Tensor Cores (vLLM blog, April 22, 2026).

That's not a rounding error; that's a broken benchmark. The fix, once identified, restored long-context accuracy close to the BF16 baseline, but only in versions that shipped the accumulation correction. Any team that adopted FP8 for the memory savings before that fix landed was running on a headroom strategy built on a number that didn't hold.

A100s behave differently again: A100 lacks hardware FP8 Tensor Cores, so FP8 there is a memory optimization only. A100 lacks hardware FP8 Tensor Cores, so FP8 there is a memory optimization only: vLLM stores the cache in FP8 and dequantizes to BF16 before running attention. Memory shrinks, but throughput doesn't move, because the compute path is still BF16 underneath. For models using sliding-window attention layers, the FP8 inter-token latency slope came out to 96% of the BF16 slope, practically identical, which pushed the throughput break-even point out past 700K tokens, well beyond any context length most deployments actually run.

None of this makes FP8 a bad choice. It makes format choice a variable that has to be checked against the specific vLLM version, the specific hardware, and the specific context lengths in play, not treated as a fixed multiplier on capacity. Teams planning block budgets around FP8 headroom should confirm the accumulation fix is in their build before counting on the memory savings as reliable.

Block allocator behavior and stalling under concurrent pressure

Under concurrency, the allocator's job is to match incoming token generation, request by request, to free physical blocks in real time. When the free pool runs dry, it has three options: stall the scheduler, evict an existing sequence, or preempt a request. None of those are free, and which one happens decides whether a deployment degrades gracefully or grinds to a halt.

Most published eviction strategies, SnapKV, PyramidInfer, PyramidKV, Ada-KV, target compressing the input KV cache, the tokens from the prompt itself, by identifying which tokens can be dropped. That's useful, but it misses the workload that actually breaks production systems: reasoning models. Reasoning models generate long internal chains of tokens before ever emitting a final answer, and none of that internal generation is input KV, it's output KV growth, which none of the input-focused eviction methods touch. A single one of these requests can consume block capacity that would otherwise serve many shorter, ordinary requests. That's the shape of what gets called a memory wall: a GPU that could serve dozens of short requests ends up supporting a handful of reasoning requests at a time.

Block exhaustion, when it hits, isn't gradual. As each concurrent request's sequence grows, so does its block table, and when the shared pool empties, the scheduler has to act immediately, not eventually. Continuous batching is built on the assumption that free blocks reappear as earlier requests finish, and under heavy reasoning-model load, that assumption breaks: completions come slowly, blocks return slowly, and the whole batching cadence stalls out.

Automatic Prefix Caching (documented in the vLLM docs) helps at the margins. When requests share an identical long system prompt, APC lets the KV blocks for that shared prefix get reused instead of recomputed and reallocated, which cuts prefill cost and block consumption for the shared portion. But it only covers the shared prefix. It does nothing for the unique reasoning trajectory each request generates after that point, which is exactly the part dominating memory under reasoning workloads. For context on what prefix sharing can do at the kernel level, ChunkAttention, a prefix-tree-based approach from Microsoft, reports a speedup of several times over on the attention kernel itself for system prompts between 1024 and 4096 tokens (arXiv:2402.15220), which shows the gains compound when prefix sharing happens at both the block level and the kernel level.

What operators see on a dashboard, when none of this is visible, is a stalled queue, a latency spike, and GPU utilization that drops instead of climbs. That pattern looks like a compute bottleneck. It's usually a block starvation event in the allocator.

Zipage and GrowPage: two architectural responses to block exhaustion under reasoning load

Two research systems, published within months of each other in 2026, tackle block exhaustion head-on rather than working around it.

Zipage, from Microsoft Research (Findings of ACL 2026, arXiv:2603.08743), introduces what its authors call Compressed PagedAttention. The core idea is token-wise eviction inside the existing page structure, not page-wise eviction. That distinction matters: evicting whole pages risks throwing away individual tokens that carry disproportionate attention weight, while token-wise eviction inside a page keeps the granularity needed to preserve quality. Zipage's scheduling supports prefix caching and runs compression asynchronously, and it stays compatible with continuous batching, so it doesn't force a rewrite of the serving infrastructure around it. On large-scale mathematical reasoning benchmarks, Zipage holds onto about 95% of a Full KV inference engine's performance while delivering more than double the speedup. That 5% quality gap isn't a flaw to be fixed later, it's the stated price of the throughput gain.

GrowPage (arXiv:2609.03494, September 2026) attacks a narrower moment: what happens when per-request memory growth threatens the concurrency of the broader batch. Its answer addresses that specific pressure point, combining compression and preemption strategies to free up room without collapsing the batch. Like Zipage, GrowPage is designed to work within existing serving infrastructure rather than requiring a wholesale redesign.

Put side by side, the two systems make an unavoidable tradeoff explicit rather than hiding it. Zipage buys throughput with a small, quantified quality cost. GrowPage buys it with the latency cost of preemption when compression alone cannot resolve the pressure. Neither approach makes the tradeoff disappear, and which one a team should reach for depends on how sensitive the workload is to output quality versus tail latency.

Agentic workloads break the forward-only eviction assumption that PagedAttention's design takes for granted

PagedAttention was built on a quiet assumption: a prompt arrives once, gets prefilled, and the KV cache only ever grows forward from there. Prefix caching and forward-only eviction both depend on that assumption, because they only work correctly if content and its position in the sequence never change once prefill is done.

Agentic workloads don't behave that way. A tool call fails and gets retried. A stale output gets dropped from the conversation. A long investigation block gets summarized down to a few lines. In each case, the conversation history isn't just extended, it's actively edited, which is a fundamentally different operation than anything PagedAttention's eviction logic was designed to handle.

Research out of the Erlangen National HPC Center, published as Leyline (arXiv:2606.01065v1), splits this into two distinct problems. The first is position-invariant reuse: identical content shifts to a new position in the sequence between turns, which invalidates an exact-prefix cache match even though the underlying KV values are, in principle, still reusable. The second is policy-driven mutation: an agent needs to actively remove or replace part of the cached content without re-prefilling every downstream token that follows it.

Not solving this raises costs directly in production. Anthropic's 2025 reporting found that agentic harnesses currently fall back to a full re-prefill on every edit, because no existing serving primitive accepts a directed edit to the cache. Paying the full cost of prefix recomputation on every retry or summarization step is exactly the cost prefix caching exists to eliminate.

Leyline's proposed fix is a declarative four-part directive that lets a policy state precisely what to edit, paired with a per-architecture kernel that applies the edit in place using a closed-form RoPE-rotation correction (referred to as δ-rotation) that restores correct attention math without re-prefilling the untouched downstream content. In measured results, the splice kernel lifts replay cache-hit rate by a substantial number of percentage points and cuts latency by up to 241 milliseconds, by reusing prefix work that a naive prompt edit would otherwise have thrown away. A simple ten-line truncation rule, routed through the same interface, lifted agentic solve rate by a substantial number of percentage points on the debug-gym benchmark.

The implication for block management runs deeper than one benchmark. Agentic workloads need eviction policy and block table logic that can handle non-monotonic edits, appends, and one-directional evictions. The allocator needs a primitive for an in-place splice. Append-and-evict isn't enough anymore.

NVMe as the next tier when the GPU block pool is exhausted: a storage engineering problem

Running the arithmetic far enough leads to an unavoidable conclusion: at long context lengths, KV cache cost per token climbs into the hundreds of kilobytes, and no single GPU's HBM can hold enough of it to sustain high concurrency on its own. The only architecture that scales past that ceiling is a tiered one: GPU HBM at the top, CPU DRAM as a middle layer, NVMe SSDs underneath. Once the GPU's block pool is exhausted, the problem shifts from modeling to storage engineering.

NVIDIA's answer, announced at CES 2026, is called ICMSP. It standardizes offloading inference context across that same three-tier hierarchy (GPU HBM, CPU DRAM, NVMe) using BlueField-4 DPUs, moving the mechanics of KV cache management off the GPU's compute path entirely so the streaming multiprocessors stay focused on math instead of shuffling data. NVIDIA's claimed numbers are several times greater power efficiency and several times higher tokens-per-second, with BlueField-4 DPU availability targeted for 2026.

On the software side, LMCache serves a similar role as a tiered storage engine sitting behind vLLM, SGLang, and NVIDIA Dynamo. It persists KV blocks across GPU memory, DRAM, disk, and object storage, and open-source deployments using this kind of layered approach have reported considerable latency reductions on workloads that are heavy on cache hits.

The pattern across both approaches is the same: once the block pool on the GPU runs out, what matters most is no longer the attention kernels or eviction heuristics but how fast data moves between tiers of storage, and how intelligently a system decides what belongs on the GPU right now versus what can wait a hop or two away. PagedAttention solved fragmentation inside a single tier of memory. What comes next is about managing memory across tiers, which is a problem the original 2023 design was never built to answer.