SGLang vs vLLM Throughput Under Long-Context Request Mixes

RadixAttention outpaces PagedAttention when workloads mix prompt lengths and reuse shared text.

· 7 min read · Updated
Cover illustration for “SGLang vs vLLM Throughput Under Long-Context Request Mixes”
Inference Runtime · September 24, 2026 · 7 min read · 1,521 words

Picking between SGLang and vLLM by running a quick benchmark on whatever GPU happens to be free is the wrong way to start. Hardware barely explains the throughput gap you'll see under long-context traffic. What actually separates these two engines is how each one holds KV cache in memory, decides what gets reused, and schedules requests when prompt lengths vary wildly across a live batch.

That distinction matters because inference splits into two phases with almost opposite personalities. Prefill chews through every prompt token at once and is bound by raw compute. Decode, by contrast, generates one token at a time, and for each of those tokens it has to read the model weights and scan the entire KV cache built up so far, so it lives and dies by memory bandwidth. Once context windows stretch into the tens of thousands of tokens, decode becomes the dominant cost rather than a rounding error. At that point, whichever engine manages KV cache better wins, and FLOPS mostly stop being the deciding factor.

How PagedAttention structures KV cache memory in vLLM

vLLM borrows an idea straight out of operating systems design. PagedAttention treats the KV cache the way a classical OS treats virtual memory: instead of demanding one long contiguous slab of GPU memory per request, it slices the cache into fixed-size blocks, 16 tokens per block by default, and scatters those blocks wherever there's room.

Each request keeps a block table, a lookup that maps the logical blocks the model thinks it's working with to their actual physical addresses on the GPU. That indirection is what kills external fragmentation almost entirely, since the allocator no longer needs to find one big open stretch of memory before it can serve a request.

The payoff appears clearly when many requests share the same text. A system prompt, a set of few-shot examples, anything identical across sessions, can live in a single physical block while every request that needs it just points its virtual block at that same location. One copy in memory, many sessions reading from it. That sharing is baked into the allocator itself rather than bolted on as a separate cache layer.

How RadixAttention structures KV cache reuse in SGLang

SGLang uses a different structure. Instead of block tables, it builds a radix tree over cached prefixes, and that tree is what lets requests reuse shared prefixes automatically, with no explicit cache hint required from the application.

Eviction runs on a standard LRU policy across the tree. Nothing is pinned by default; entries get inserted, reused, or evicted the way any general-purpose cache behaves, rather than sitting inside a fixed block allocation scheme.

The real advantage sits in how the tree matches text. Because it works at the token level instead of matching whole fixed-size blocks, it can catch partial overlaps and branching prefixes, the exact pattern you get in multi-turn chat histories, agent traces that fork across reasoning steps, and RAG pipelines where the retrieved document changes but the surrounding template doesn't. A handful of scenarios show this off well, including a shared system prompt across every user of a chat product, a reused few-shot evaluation prompt applied across thousands of eval calls, a RAG template with a fixed instruction wrapper around swappable retrieved text, a conversation history that keeps growing turn over turn, and an agent workflow where several reasoning branches share a common root. In each case, the radix tree finds and reuses whatever overlap exists, however partial, without the application having to declare it.

The memory arithmetic that makes long context a different problem

The numbers behind long context explain why this isn't a marginal optimization problem. A single 128K-token request against Llama 3.1 70B needs roughly 42.9 GB of KV cache in BF16. FP8 reduces that to about 21.5 GB, roughly half. NVFP4 on Blackwell hardware reduces it to around 10.7 GB.

Even at 8K context, a far more typical production setting, a 70B model still needs about 20 GB of KV cache per request. Batching 32 of those requests together makes the cache alone demand roughly 640 GB, more memory than a single H100 has, more than most multi-GPU nodes have, before the model weights are even counted.

So at long context, one user's KV cache can eat most or all of an 80 GB H100 by itself. Every additional concurrent user past that point forces a choice: quantize the cache, spread it across more GPUs, or start evicting. And in production, prompt lengths never arrive neatly. Short chat turns land in the same batch as 128K-token document-analysis requests, and that mix creates memory pressure that swings wildly from one moment to the next, which is exactly the condition under which the two schedulers start to diverge.

H100 benchmarks on throughput divergence across workload types

On a single H100 running Llama 3.1 8B in BF16, PremAI measured SGLang at roughly 16,200 tokens per second against vLLM's 12,500, a 29% edge for SGLang. AI Multiple's own test landed almost on top of that, 16,215 versus 12,553.

That gap deserves a caveat before anyone treats it as gospel. The most widely cited version of this comparison pitted SGLang v0.2.3 against vLLM 0.11.0, two releases separated by roughly two years of engineering work on both projects. A benchmark with that much version drift can't cleanly separate architectural advantage from one engine simply having had more time in the oven.

And the gap doesn't hold at scale. At 70B, it compresses hard. At 50 concurrent requests, vLLM hits 1,850 tokens per second against SGLang's 1,920, a single-digit difference. At 100 concurrent requests, it's 2,400 versus 2,460. At a single concurrent request, 120 versus 125. Separate analysis from techsy.io lines up with the same story: a 3 to 5% gap at 70B and above, versus 29% at 7B to 8B.

Thinking through what changes at scale explains the reason. Bigger models push decode even further into memory-bandwidth-bound territory, and prefill shrinks as a share of total cost. RadixAttention's biggest win comes from cutting redundant prefill work, so when prefill matters less to begin with, that win contributes less to the final throughput number.

Scheduling under mixed prompt-length workloads amplifying or muting the cache advantage

Real traffic doesn't sort itself by length. Short prompts and long ones land in the scheduler at the same time, and a single long-context request can eat most of the GPU's free memory, forcing hard tradeoffs about what stays resident and what gets pushed out.

vLLm's answer leans on three mechanisms working together. Continuous batching lets new requests slide into a running batch the moment earlier ones finish and free their blocks, so the GPU doesn't idle waiting for a batch boundary. Chunked prefill breaks long prompts into pieces so one enormous prefill can't stall decode progress for every other request sharing the batch. And its prefix cache operates at the block level, giving predictable, hash-based cache hits when prompt structure repeats, though a long-context request under memory pressure displaces other requests' blocks in a way that's deterministic but not always favorable to whoever gets evicted.

SGLang's scheduler runs on the radix tree's LRU policy instead. When a long request needs memory, the least-recently-used cached prefix gets evicted, and that can mean discarding shared prefix state that a shorter, more frequent request was relying on. The zero-overhead batch scheduler helps offset the cost of that churn by overlapping scheduling decisions handled on the host processor with GPU compute, so scheduling overhead doesn't stack directly onto latency as concurrency rises. And because matching happens at the token level rather than the block level, a partially evicted prefix still yields a partial hit rather than forcing a full recompute, something block-level hashing can't do as gracefully.

Benchmarks on multi-turn workloads capture the practical result of that difference, with SGLang showing higher per-session decode throughput across turns compared to vLLM at elevated concurrency levels where the radix tree keeps finding reuse.

FP8 KV quantization's effect on memory residency and the throughput calculation

FP8 support on H100 now works across the major serving engines through --quantization fp8 in vLLM, the same flag in SGLang, and --qformat fp8 at the quantize.py step for TensorRT-LLM. None of this is exotic anymore; it's a default lever.

The payoff on decode is concrete. FP8 KV cache cuts the inter-token latency slope, the rate at which latency worsens as context grows, down to 54% of the BF16 baseline on H100. That's a direct answer to the memory-bandwidth bottleneck decode runs into at long context.

Lining up the precision ladder for Llama 3 70B at 128K context and a single user shows roughly 42 GB in BF16 or FP16, about 21.5 GB in FP8 E4M3 (close to half), and around 10.7 GB in NVFP4, though that last tier is limited for now to one hardware generation.

Before treating FP8 as a free win on Hopper hardware, one wrinkle matters: quantization gains at long context aren't automatic. They depend on the kernel underneath being correct at the context lengths actually being served, so check that detail against the specific engine version before trusting a production deployment to it.

Diagram: KV Cache Memory Demand Explodes at Long Context. Visualizes: Show the memory cost of KV cache for Llama 3.1 70B at two context lengths and across three precision formats, to make the scale of the problem visceral.

Sources

  1. vLLM vs SGLang 2026: H100 Benchmarks Inside
  2. vLLM vs SGLang: Performance, Features & Deployment Compared
  3. Efficient Memory Management for Large Language Model Serving with PagedAttention | Proceedings of the 29th Symposium on Operating Systems Principles
  4. lmsys.org
  5. vllm-project.github.io

More in Inference Runtime