vLLM Chunked Prefill Scheduling and TTFT Optimization
Chunked prefill moves prefill and decode from competing priorities to cooperative scheduling phases.

vLLM's scheduler faces one fight every single iteration: two very different kinds of GPU work competing for the same batch slot. Chunked prefill settles that fight by putting a hard cap on how much prompt-processing work can enter any one scheduling step. That cap moves through the whole system: into KV cache allocation, into the latency numbers on a dashboard, and understanding that chain is what separates someone tuning max_num_batched_tokens by guesswork from someone tuning it on purpose. Leaving chunked prefill's decode-first policy on its default settings is a mistake for bursty or long-context traffic. It's a starting point that needs tuning, and treating it as finished is how teams end up with tail latency they can't explain.
Prefill and decode are not the same job wearing different clothes. Prefill takes every token in a prompt and processes them at once, building the KV cache for that request in a burst of matrix multiplication that saturates compute. Decode does the opposite: it produces one new token, reads from a KV cache that already exists, and spends most of its time waiting on memory bandwidth instead of compute. One is compute-bound, the other bandwidth-bound. A scheduler that treats them as interchangeable units of work runs into trouble fast.
Picture ten requests mid-generation, each waiting on its next token. An eleventh request arrives with a prompt long enough to swamp the batch, maybe a long code file or a RAG context stuffed with retrieved documents. If the scheduler runs that prefill uninterrupted, which is what vLLM's default scheduler did before chunked prefill existed, none of the ten in-flight requests get a token until the whole prefill finishes. That's prefill-decode interference: head-of-line blocking specific to LLM serving, where one request's arrival stalls every other request's progress, and the stall scales with the size of the prompt.
The old default picked the wrong side of that trade almost every time. It prioritized prefill, refused to mix prefill and decode work in the same step, and bought a fast TTFT for the new request at the direct cost of inter-token latency for everyone already streaming. Measurements of real inference workloads show SM utilization hovering around 30 to 40% under static batching even at moderate request rates. The GPU sat idle, waiting for one oversized prefill to clear before decode could resume.
This gets worse as prompt lengths grow, and prompt lengths keep growing. Coding assistants, RAG pipelines, and agentic workflows routinely hand the model prompts in the tens of thousands of tokens. At that scale, one uninterrupted prefill can stall every concurrent decode stream long enough that a user actually feels it. That's why chunked prefill exists, and why it ships as a first-class vLLM feature, available as an opt-in flag, with V1 engine integration bringing it into the default scheduling path.
What chunked prefill does to the scheduler's iteration loop
The core move is simple to state: bound how much prefill work can enter any one iteration. Instead of processing an entire prompt in a single pass, the scheduler slices it into contiguous chunks and runs one chunk per iteration. Positional context from earlier chunks never gets lost, because the KV cache builds incrementally, chunk by chunk, so the model always has access to everything processed so far. The whole prompt still gets processed eventually, it just never gets to monopolize one giant iteration.
Turning this on flips the scheduler's priority order, and it should: decode requests go first. Every pending decode gets batched before a single prefill token is admitted. Whatever token budget is left in max_num_batched_tokens after decode is served goes to prefill, and if a pending prefill doesn't fit in what's left, it gets cut down to size. Only the portion that fits runs this iteration; the rest waits for the next one.
Walking through it iteration by iteration makes the shape obvious. In iteration N, decode requests A and B each emit their next token, and whatever budget is left over fills with the first chunk of request C's prompt. In iteration N+1, A and B decode again, and chunk two of C's prompt runs. Decode traffic never stalls waiting on C. C's own first token arrives later than it would have under the old scheme, but its prefill makes steady, predictable progress instead of either running instantly or blocking everything else.
What makes this fit naturally with vLLM's existing continuous batching is that prefill chunks and decode tokens end up treated as the same kind of thing: entries competing for space in one token budget pool. The scheduler no longer needs a hard architectural line between "prefill mode" and "decode mode." Chunked prefill is continuous batching extended to cover prefill.

The configuration knobs that control chunked prefill behavior in vLLM
Turning the feature on is the easy part: a command-line flag, or the matching setting in engine args. In vLLM V1, the current default engine, it's already on. Under the older V0 engine, someone had to request it explicitly. Flipping it on also triggers the decode-first scheduling policy described above, and the two are not independent settings. Treating them as separable is a mistake.
The knob that actually matters day to day is max_num_batched_tokens. It sets the total token budget available per iteration, and by extension, the largest size any prefill chunk can reach once decode has taken its share. Current vLLM releases default this to 8192 when chunked prefill is on. Pushing it down makes the scheduler favor decode even harder: ITL drops, but any given prefill needs more iterations to finish, so TTFT for that request stretches out. Push it up and prefill finishes in fewer iterations, improving TTFT and often overall throughput, but each iteration now runs longer, and decode requests sharing the batch wait longer for their next token. The knob picks where on the line between TTFT and ITL a workload sits, and most teams leave it at the default far longer than they should.
A handful of secondary settings interact with this budget. --gpu-memory-utilization defaults to 0.90 and sets how much of the remaining VRAM goes to the KV cache pool. Nudging it to 0.95 on a bare-metal box frees up more KV blocks, supporting larger effective batch sizes alongside chunked prefill. --max-model-len, set below the model's native context window, also frees KV blocks by shrinking the worst-case reservation per sequence, letting more concurrent requests coexist. --max-num-seqs caps concurrent sequences and defaults to 1024 in vLLM V1 and v0.18.0; raising it to 2048 or beyond makes sense on high-traffic API deployments where request concurrency, not per-request size, is the bottleneck.
One compatibility limit deserves attention before anyone builds a deployment plan around it. As of v0.18.0, --enable-chunked-prefill cannot run alongside draft-model-based speculative decoding (--speculative-model). The exception is a lookup-based GPU speculative decoding technique, added in vLLM V1 alongside chunked prefill support, which runs fine with it. Anyone leaning on draft-model speculative decoding has to pick one feature over the other, or take the NGram path instead.
How chunk-size decisions propagate into KV cache allocation
Every token a prefill processes generates key and value vectors that need somewhere to live for the life of that request. That's the KV cache, and it grows along five dimensions at once: transformer layers, KV heads, head dimension, sequence length, and how many requests run concurrently. Chunking a prefill spreads the writes out over time without shrinking that total memory bill. Each chunk that finishes commits its share of KV tensors to the cache right away, so a partially completed prefill is a real, already-incurred memory cost.
PagedAttention is what makes this workable underneath. Without paged allocation, every request would need a contiguous block of VRAM sized for the model's maximum context length the moment it arrived, and chunking would just fragment that reservation over time instead of avoiding the problem. PagedAttention instead divides the cache into fixed-size blocks, 16 tokens each, allocated on demand as tokens actually get processed. Each chunk's KV output lands on the next available page, no upfront reservation, none of the fragmentation penalty a naive allocator would carry. The original PagedAttention work documented cache waste of 60 to 80% under earlier allocation schemes; paged allocation brought that under 4%, unlocking throughput gains in the range of two to four times.
Smaller chunk sizes push against this system in a specific way. A smaller max_num_batched_tokens means more iterations to finish any long prompt, and for the whole stretch that prompt sits mid-prefill, it holds a partially filled KV footprint nobody else can free or reuse. At high concurrency, a scheduler can end up with a pile of requests all sitting in this half-finished state, each holding blocks it doesn't fully need yet but can't release either. That builds a soft ceiling on memory that has nothing to do with compute: the scheduler may have to pause admitting new prefills purely because KV blocks ran out, not because the GPU ran out of cycles.
When that ceiling hits, vLLM V1's default response is to recompute rather than swap. Earlier designs might swap displaced KV data out to CPU memory; RECOMPUTE mode just throws away the incomplete state and redoes the work later, on the reasoning that recomputation usually beats the round trip to CPU and back. Watching KV cache utilization alongside TTFT tells whether a slowdown is compute-bound or memory-bound, and the two call for genuinely different fixes.
What the latency metrics measure and how chunked prefill moves them
Three numbers matter here, and they measure different things. TTFT, time to first token, is the clock from request arrival to the first token out, and for long prompts it's dominated almost entirely by how long prefill takes. ITL, sometimes called TPOT, time per output token, measures the gap between successive tokens during decode, and it gets worse whenever an iteration is loaded down with heavy prefill work. Throughput, tokens or requests per second aggregated across everything running, tracks how efficiently the GPU actually gets used.
Chunked prefill sits directly on the trade-off between the first two. Shrink max_num_batched_tokens and decode requests almost never get blocked, so ITL drops, but any individual prefill now takes more iterations to finish, so TTFT for that request grows. Grow the budget and prefill finishes faster, improving TTFT, but each iteration gets heavier and decode requests waiting behind it feel the delay. The knob lets someone decide, on purpose, which side of this trade-off their traffic can afford.
A mixed-workload benchmark on an H100 SXM5 80GB running vLLM v0.18.0 makes the payoff concrete. With 50 concurrent requests, 10% carrying 32K-token prompts and the rest around 1K tokens, p50 TTFT barely moves when chunked prefill switches on, because short requests were never the problem. The p95 TTFT for the 32K-token requests, the figure that matters, drops 68%, from 2,800 milliseconds down to 890. That's the number that matters. Interference hits hardest in the tail, where a long prompt lands on top of an already-busy batch, and that's exactly where chunked prefill earns its keep. Short queries stop spiking just because a long-context request happened to land at the wrong moment.
Throughput gains track the shape of the workload. On OPT-30B running on a single H100, high-prompt-low-decode and low-prompt-low-decode workloads saw roughly 20% throughput improvement, while high-prompt-high-decode and low-prompt-high-decode workloads saw closer to 5%. Chunked prefill pays off most where prompt length dominates a request's total cost. In decode-heavy workloads, where output length swamps prompt length, there's less scheduling inefficiency sitting around to recover.

Where the stall-free decode-first policy creates its own fairness problem
Prioritizing decode without exception solves interference and creates a new fairness imbalance in its place. Decode tasks frequently finish with slack to spare, landing well inside whatever TPOT budget they're held to. Meanwhile, during traffic bursts, prefill tasks rack up real TTFT violations even while decode capacity sits there unused the whole time. A rigid decode-first rule has no way to notice that slack and hand it to a surge of newly arrived prefills. The priority order is fixed, so the capacity just goes unclaimed, and that's the flaw in treating decode-first as a finished design instead of a first pass.
Part of the problem is that TBT, time between tokens, is a shaky thing to schedule around during a burst. Guaranteeing it strictly means reserving compute for decode even in moments when no decode work is actually pending that needs it. And most client applications buffer tokens before rendering them anyway, so jitter in intermediate token delivery is mostly invisible to the person on the other end. Holding the serving layer to a strict TBT guarantee can end up spending compute to defend a target the user was never going to notice missing.
Research on this exact problem, published under the name FairBatching, found that letting prefill dynamically reclaim decode's unused slack during bursts, instead of locking the priority order in place, cut TTFT tail latency by up to 2.29× while still holding TPOT service targets. On realistic traffic traces, the same approach reported a 20.0% gain in single-node capacity and a 54.3% gain at the cluster level. The max_num_batched_tokens default is a reasonable place to start, and dynamic budget adjustment remains an open research question.
Workloads with bursty prefill arrivals, batch inference jobs, RAG systems where retrieval size swings from request to request, are the ones most likely to hit this fairness ceiling before they ever hit a throughput ceiling. For those workloads, the fixed decode-first rule that chunked prefill leans on is doing real work. But anyone running that traffic pattern and calling the default configuration good enough is leaving latency on the table they could recover with a different scheduling policy.


