Speculative Decoding Overhead in Production vLLM Deployments
Speculative decoding speeds up single requests but can cut throughput under heavy load.

Speculative decoding promises something close to a free lunch: faster token generation with no change to output quality. In production vLLM deployments, that promise holds only under specific load conditions, and the costs that eat into the gain (draft model memory and KV cache waste) tend to appear exactly when a team is scaling up, and are visible in production capacity planning, not when they first benchmark it on a single request.
Why speculative decoding exists
Generating one token at a time is slow for a reason that has nothing to do with math. Every forward pass through a large language model means pulling the full set of weights off the GPU's VRAM, and that transfer, not the arithmetic, is what eats the clock. For a 70-billion-parameter model, producing a single token takes something like 30 to 50 milliseconds, and almost all of that time is the GPU sitting there waiting on memory bandwidth while its compute cores go mostly unused.
That idle compute capacity is the opening speculative decoding walks through. Checking whether a handful of candidate tokens are correct costs roughly the same, in GPU time, as generating just one token. So instead of producing tokens one at a time, a small draft model proposes several tokens cheaply, and the big target model verifies all of them in a single pass. Whatever gets accepted is essentially free. Whatever gets rejected falls back to what the target model would have generated anyway. The output distribution ends up mathematically identical to standard decoding, so there's no quality tax to pay, only a bet on whether the draft model guesses well enough to make the extra step worth it.
The production methods available in vLLM today
vLLM ships several ways to run this draft-and-verify loop, and they trade off differently on memory, flexibility, and how much speedup they actually deliver.
Draft model speculation is the classic setup: a separate small model, drawn from a modest range of parameters well below the target model's, drawn from the same architecture family as the target model. Acceptance rates for instruction-following workloads typically run between 0.6 and 0.75, which translates to 2 to 3x throughput gains. It's the easiest method to stand up if a compatible draft model already exists, but it also carries the highest VRAM cost of the group, for reasons the next section gets into.
N-gram, or draft-free, methods skip the neural draft model entirely and instead pattern-match against the prompt's own history. There's no separate set of weights, so there's zero VRAM overhead from a draft model, but acceptance rates run lower. This method works best on prompts that repeat themselves a lot or follow a rigid structure, like structured data extraction or templated outputs.
Medusa attaches parallel decoding heads directly onto the target model rather than running a separate model at all, and it uses a tree-structured attention scheme to evaluate multiple candidate continuations at once. Measured speedups run 2.2 to 3.6x, though the heads are trained for specific tasks, which limits how well they generalize to other kinds of prompts.
EAGLE, and its successors EAGLE-2 and EAGLE-3, take a different approach: the draft model trains on the target model's internal feature representations, pulled from a layer near the top of the network, rather than on raw token embeddings. On LLaMA 2 Chat 70B, EAGLE delivers 2.7 to 3.5x latency reduction. EAGLE-2 adds a dynamic draft tree that adjusts its shape per request, and EAGLE-3 fuses hidden states from multiple layers, pushing speedups to 3 to 6.5x, with coding and reasoning workloads seeing 3 to 4x gains specifically. Pre-trained EAGLE checkpoints are already available for Llama, Qwen, and Mistral model families on Hugging Face, which lowers the barrier to trying the method against an existing deployment.
Draft model memory pressure and KV cache capacity
Every team that turns on speculative decoding accounts for the obvious cost: the draft model's weights take up VRAM. A draft model with a parameter count a small fraction of the target model's size sitting next to it is not free, and most teams size for that.
What gets missed is that the draft model also needs its own KV cache, and that cache eats into pool capacity disproportionately to the size of the model's weights. Benchmarks running Qwen3-14B on a single H200 GPU show the gap clearly. N-gram speculation, with no draft weights and no draft KV cache, costs just 2.8% of the KV pool. EAGLE3 costs 6.1%. A separate Qwen3-0.6B draft model, though, costs 43.8% of the KV pool, and that cost comes mostly from KV geometry, not from the weight size of a model that's a fraction the size of the target.
There's also a fixed tax that applies no matter which method gets chosen: enabling any speculative decoding mode in vLLM carries a baseline overhead of roughly 2.66 GiB, before a single draft model weight or KV cache page enters the picture. Anyone budgeting VRAM for a speculative decoding deployment needs to account for that fixed cost first, then layer the method-specific overhead on top.

The concurrency cliff: where speculative gains invert under load
Teams get caught off guard by this after they've already shipped. The exact mechanism that speeds up a single, isolated request can drag down aggregate throughput once the cluster is under real load.
Continuous batching is why. At low concurrency, each request is memory-bandwidth-bound, and the GPU's compute cores sit idle, waiting on weight transfers. Speculative decoding fills that idle time. But as concurrency climbs, continuous batching starts stacking memory lookups into dense matrix multiplications, and the workload shifts from memory-bound to compute-bound. Once the GPU's tensor cores are fully busy doing actual math, there's no idle cycle left for speculation to exploit. The draft model still costs what it always cost. The benefit it was buying just isn't there anymore.
Benchmark data on Qwen3-14B, tested on a single H200 with a 2K context window, puts numbers on this. Throughput crossover, the concurrency level at which speculative decoding starts performing worse than plain decoding, happens somewhere between 17 and 49 concurrent requests, depending on the method and the workload domain. By the time concurrency hits 100, every speculative method tested delivers only 0.706x to 0.796x of baseline throughput, a real regression. By the time concurrency hits 100, every speculative method tested delivers only 0.706x to 0.796x of baseline throughput, a real regression rather than a wash. That regression is visible in production capacity planning long after a proof-of-concept looked great.
Not every architecture hits this wall the same way. Testing on gpt-oss-120B, a mixture-of-experts model, running Eagle3 on vLLM in April 2026, showed throughput and latency improvements holding up through 200 concurrent requests. Peak output throughput reached 2,574 tokens per second with speculative decoding on, against a baseline of 2,024 tokens per second, a 27.2% peak improvement on a mixed conversational workload representative of real-world chat traffic. The likely explanation is architectural: MoE models activate only a sparse subset of their parameters per token, which leaves more compute headroom even at high concurrency. That's a property of sparse activation, not a general rule that applies to every model family, so it shouldn't be read as a reason to expect the same ceiling everywhere.

KV cache bloat from uncommitted draft tokens
Rejected draft tokens don't vanish for free. They still need KV cache entries during the verification step, since the target model has to check them against real hidden states, and vLLM's memory allocator works in fixed-size blocks that it can't reclaim until the rejection is confirmed. Until that confirmation lands, the memory sits there, occupied by tokens that turned out to be wrong.
Draft tokens also count against the global token limits that vLLM's scheduler uses to decide what to run next. Speculative decoding can quietly constrain scheduling decisions even before any memory pressure becomes visible. vLLM's own production documentation says as much: draft tokens consume extra KV-cache pages and count toward those global limits, and that's part of why speedups shrink as load increases.
The effect compounds. Every rejected draft step leaves behind a wasted page allocation before the system rolls it back, and the right draft length depends heavily on how the prompt is structured, and longer draft windows do not reliably help. There's no single universal setting to reach for.
One proposed fix worth watching is TransKV, a transactional KV-cache design that separates the stable, already-committed KV state from a packed buffer holding speculative writes. Under this design, speculative KV entries stay uncommitted until the system knows whether they were accepted, and only the accepted prefix gets written into the paged cache; rejected entries get discarded without a rollback step. It's a research direction, not something shipping in vLLM today, but it points at where the mitigation work is headed.
Draft-verification serialization and scheduling coordination cost
Beyond compute and memory, there's a coordination tax that's easy to overlook. vLLM has to coordinate KV cache state across the draft and target models, and at every single speculative step it has to run acceptance decisions and rejection sampling on top of that. This isn't a one-time setup cost. It happens on every draft-verify cycle, even when the draft model's own computation is already cached and cheap to run.
The draft and verify steps are also strictly sequential, not pipelined. Draft generation has to finish completely before verification can start, and verification has to finish before the next draft step begins. That dependency chain caps how much concurrency can actually help, even when each individual step runs fast on its own.
As of vLLM 0.15.0, pipeline parallelism doesn't compose with speculative decoding. Teams running pipeline-parallel configurations across multiple GPUs can't turn on speculative decoding at the same time, full stop.
Teams that need to reproduce exact results across configurations should validate their speculative decoding setup carefully before relying on it for auditable outputs.
Acceptance rate, draft model selection, and whether any gain survives
Everything above funnels into one number: the acceptance rate, often written as α, which is the fraction of draft tokens the target model actually confirms. At α of 0.6 or higher, paired with a reasonably long draft sequence, speedups of 2 to 3x are realistic and repeatable. Drop below α of 0.50, though, and the overhead from running the draft model, managing its KV cache, and coordinating the verify step typically cancels out whatever speedup speculation was supposed to deliver.
Acceptance rate isn't a fixed property of a method. It moves with the workload, the draft model's relationship to the target model, and the structure of the prompt itself. That's the real reason there's no single speculative decoding configuration that works everywhere: choosing a draft model, or choosing to skip one entirely in favor of n-gram matching, is a decision that has to be made against the actual traffic a deployment expects to see, not against a benchmark run on someone else's workload.


