TensorRT-LLM In-Flight Batching vs vLLM Continuous Batching

TensorRT-LLM calls it "in-flight batching." vLLM calls it "continuous batching." LMDeploy calls it "persistent batching." All three names point at the same scheduling trick, and the fact that three different names exist for the same scheduling trick tells you something about how obvious the fix was once someone finally saw the problem clearly. The real story isn't whether one engine schedules requests better than the other. It's that they schedule requests the same way, and everything worth arguing about happens somewhere else: kernel compilation, KV cache placement, and how far down the memory hierarchy each engine is willing to reach when GPU memory runs out.

The idea traces to Orca, a paper out of OSDI 2022, which reported a 36.9x throughput gain over NVIDIA's FasterTransformer at matched latency, using iteration-level scheduling alone. No new attention kernel, no fancy quantization trick. Just a smarter scheduler. That result means the single biggest lever in LLM serving performance wasn't a model architecture change or a hardware upgrade. It was admission control.

Two things should be separated early, because the industry conflates them constantly. Continuous batching (or in-flight batching, take your pick) is a scheduling policy. PagedAttention is a memory management technique. vLLM happened to popularize both at once, running them together so effectively that people started treating them as a package deal. They're not. You can run continuous batching without paged memory, and the two techniques remain separable in principle. Per inferenceengineering.tech, every serving engine of consequence now has some form of continuous batching built in. The feature gap closed somewhere in 2024 and 2025. So the question facing anyone picking an inference engine today isn't "which one batches requests dynamically." They all do, because the feature gap closed somewhere in 2024 and 2025. The question is which architecture, which memory strategy, and which operational tradeoffs fit the workload in front of you.

What the shared scheduler does, and why static batching made it necessary

Before continuous batching existed, engines ran static batches: a fixed group of requests processed together, locked to the pace of whichever request in the batch took longest to finish. If nine sequences finished generating in 50 tokens and the tenth needed 500, the other nine sat there, GPU slots reserved and idle, waiting for the slow one to wrap up. Under high variance in sequence length, this isn't a minor inefficiency. Per packet.ai, static batching throughput can collapse to as low as 81 tokens per second in workloads where prompt and output lengths vary widely. The KV cache side was just as bad: Kwon et al.'s 2023 PagedAttention paper found that static allocation strategies wasted 60 to 80 percent of reserved KV cache memory, mostly to fragmentation and to reserving space for a maximum sequence length that most requests never approached.

Continuous batching fixes this by making a scheduling decision at every single token step instead of once per batch. The moment a sequence finishes, its slot gets filled by a new request on the very next iteration. No waiting for the whole batch to clear.

A related but distinct technique involves eliminating the padding tokens that used to sit between prefill and decode requests sharing a batch. It's not a scheduling decision so much as a data layout decision, but it works hand in hand with iteration-level scheduling to keep GPU cycles from going to waste on padding.

There's a tension baked into all of this that both schedulers have to manage: prefill (processing an incoming prompt in one forward pass) is compute-bound, while decode (reading the entire KV cache to generate a single new token) is memory-bound. Mix the two in the same batch carelessly and one starves the other. Chunked prefill is the current fix: split a big prefill into smaller pieces so it doesn't block decode requests from making progress. Per vLLM's documentation, this is built into vLLM's V1 engine by default. How each engine handles the prefill-decode tension diverges in implementation, and that divergence is worth tracking closely as the two engines are examined in turn.

How TensorRT-LLM's scheduler and execution backend work

Requests coming into TensorRT-LLM get tokenized and admitted into an active sequence set that the runtime scheduler tracks directly, and the scheduler is deliberately decoupled from kernel execution so that admission decisions don't have to wait on GPU compute cycles. That separation is part of what makes in-flight batching work at all.

TensorRT-LLM release 1.2 dropped the TensorRT engine-compilation backend entirely, per Lyceum Technology's reporting, which changes the picture for anyone still working off older comparisons. PyTorch is now the sole execution backend. For years, "TensorRT-LLM" and "ahead-of-time compiled kernels" were treated as nearly synonymous, and that framing is now out of date.

What survives from the compilation-first era is the hardware-native optimization work. Kernel fusion is still aggressive: LayerNorm, matrix multiply, and activation functions get fused into a single CUDA kernel so intermediate results never get written back to global memory, which preserves memory bandwidth that would otherwise get burned on round trips. CUDA graphs cut down CPU dispatch overhead by capturing a sequence of GPU operations once and replaying it, instead of re-issuing each kernel launch from the CPU on every step. And the paged KV cache, often assumed to be a vLLM exclusive, is present on TensorRT-LLM's PyTorch backend too. That's not a feature gap. It's a shared foundation with a different execution engine sitting on top of it.

TensorRT-LLM exposes explicit scheduler knobs: maximum batch size, maximum sequence length, KV cache fraction. All of these get set before serving starts, which locks in the performance envelope at deployment time rather than letting the engine adapt on the fly. Per Lyceum Technology, extracting peak performance out of TensorRT-LLM takes careful upfront configuration, and that is why it appears most often at scale, where the engineering cost of tuning gets amortized across enormous request volume. That investment yields 15 to 25 percent higher raw throughput than vLLM on H100 hardware for dense models, per inferenceengineering.tech. NVIDIA has also reported up to a 4x throughput increase for GPT-J-6B running FP8 on H100 versus FP16 PyTorch eager mode on A100, climbing to 8x once in-flight batching gets added on top, per Lyceum Technology. That comparison changes the GPU, the precision, and the kernel stack all at once, so pinning the gain on any single one of those factors isn't really possible from the number alone.

How vLLM's scheduler and PagedAttention work

PagedAttention, introduced by Kwon et al. at SOSP 2023, takes an idea straight out of operating systems textbooks, virtual memory paging, and applies it to the KV cache. Instead of reserving one contiguous memory block per sequence, PagedAttention splits the KV cache into fixed-size blocks (16 tokens each, in the original design) and maps them through a per-sequence block table to physical GPU memory that doesn't need to be contiguous at all. Blocks get allocated as a sequence grows and returned to a shared pool the moment it finishes.

The payoff is stark: memory waste drops below 4 percent, limited to the last, partially-filled block of any given sequence, compared to the 60 to 80 percent overhead that static allocation left on the table. That freed-up VRAM goes straight back into batching, letting vLLM keep more concurrent requests in flight and the GPU's streaming multiprocessors busier for longer stretches. It pays off especially well for parallel sampling and beam search, where multiple candidate outputs share the same prompt's KV cache pages instead of duplicating them.

This flexibility carries a real tradeoff. TensorRT-LLM compiles kernels with a known memory layout ahead of time; vLLM maps virtual blocks to physical memory dynamically, during execution, which costs a bit of raw kernel efficiency in exchange for handling variable-length, unpredictable traffic without needing to be told in advance what to expect.

When the KV cache fills up anyway, vLLM has to preempt something, and it picks between two strategies: recompute, meaning discard the blocks and redo the prefill from scratch, or swap, meaning move the blocks out to CPU DRAM and bring them back later. Which one costs more depends heavily on sequence length and hardware, a point that returns once the memory hierarchy comes into view.

Chunked prefill ships by default in vLLM's V1 engine, breaking long prefills into smaller pieces so a giant prompt doesn't block decode progress for every other request sharing the batch. Per Lyceum Technology, vLLM tends to hold its throughput steadier as concurrent request counts climb, and for workloads with wildly variable prompt lengths and bursty traffic, it often delivers more consistent tail latency than TensorRT-LLM does. On the operational side, getting started is about as simple as serving frameworks get: pip install and support for more than 400 model architectures, per inferenceengineering.tech. That's not a minor convenience. It's an architectural choice with its own tradeoffs, and it shapes who reaches for vLLM first.

Where the KV cache becomes the real performance determinant

Model weights used to be the thing people worried about fitting on a GPU. That's no longer where the ceiling is. For Llama 3.1 70B running a 131K token context, the KV cache alone runs to roughly 43 GB per request, per localaimaster.com, which is larger than the model's own weights at FP8 precision. At long context lengths, KV cache capacity is the binding constraint, not model storage.

Both engines now run paged KV caches, so the real divergence isn't whether paging exists but how each engine manages eviction, sharing, and offload once the cache starts filling up.

vLLM automatically reuses KV cache pages across requests that share an identical prompt prefix, things like a system prompt, a tool definition, or a set of few-shot examples. Per backend.ai's reporting, warm production caches have shown prefix hit rates around 87 percent. That's a huge amount of redundant prefill computation simply avoided. The same block-table architecture that makes prefix caching work also makes KV cache sharing natural for parallel sampling and beam search, where several output sequences branch off the same prompt and don't need separate copies of its cache. TensorRT-LLM handles comparable use cases through its own KV cache management on the PyTorch backend, though the two engines expose and control this sharing differently, and that difference matters more as concurrency and context length both grow.

A bandwidth cliff governs every offload decision either engine makes. H100 SXM5 HBM runs at roughly 3.35 TB/s. CPU DRAM, reached over PCIe 5.0, tops out around 63 GB/s, a gap of roughly 50x. Moving a 50 GB KV cache off HBM takes about 15 milliseconds. Moving that same cache from CPU DRAM takes about 800 milliseconds, per arXiv 2601.19910. A 65,000-token document paired with a 32-token question, running on Llama 3.1 405B, needs to move roughly 33 GB across PCIe, which takes around 500 milliseconds. That's why the choice between recompute and swap during preemption isn't a rounding error. It's a decision with a half-second consequence attached to it.

The three-tier memory hierarchy both engines now navigate: HBM, DRAM, and NVMe

Lining up the full bandwidth stack makes the picture sharper. HBM runs at roughly 3.35 TB/s. PCIe 5.0-connected DRAM runs at roughly 63 GB/s. PCIe 4.0 NVMe storage delivers roughly 7 GB/s of sequential throughput, with sequential throughput around 7 GB/s, per spheron.network. Three tiers, three very different cost profiles, and both engines now have to reckon with all three.

NVIDIA Dynamo is the framework doing the connecting. It's a distributed inference framework built to offload KV cache from GPU HBM down to CPU DRAM, then to local SSD, then out to networked storage if needed, and it integrates with both vLLM and TensorRT-LLM rather than favoring one. Routing happens through NIXL, NVIDIA's Inference Xfer Library, which can move data over NVLink, InfiniBand, RoCE via UCX, NVMe-oF, or plain TCP depending on what's available.

NVIDIA's ICMSP and CMX framework, announced at CES 2026, goes a step further by formalizing NVMe-backed flash as its own tier, labeled G3.5, between local NVMe (G3) and networked storage (G4). BlueField-4 DPUs handle the KV cache movement off the GPU's compute path entirely, which matters because it keeps the streaming multiprocessors doing math instead of sitting around waiting on an I/O request to come back. That's the actual argument for treating NVMe as a first-class tier of context memory rather than an emergency swap device: if a dedicated data-processing accelerator handles the movement, the GPU doesn't stall waiting for it.

Dynamo also does KV-aware routing: it can direct a request to a node that already holds the relevant cache blocks rather than one that would need to fetch or recompute them. That couples the scheduling decision to the storage locality decision in a way that didn't used to exist. Neither vLLM nor TensorRT-LLM controls that routing layer directly, but both benefit from it when it's in place. Storage, in other words, is no longer a commodity beneath the compute stack. Which blocks get evicted, how fast they come back, and whether the retrieval path runs through DRAM or NVMe decides whether a preempted request costs 15 milliseconds or the better part of a second, and that's a decision that shapes latency at least as much as the scheduler itself does.

Prefill-disaggregation: the architectural extension that changes the comparison

Prefill and decode want different things from the hardware. Prefill wants raw compute, since it's running a full forward pass over the incoming prompt. Decode wants memory bandwidth, since generating each new token means reading the entire KV cache built up so far. When both phases run on the same GPU at the same time, they compete for the same resources, each one dragging on the other's performance.

The fix showing up across the industry is disaggregation: split the cluster so dedicated prefill workers handle prompt ingestion while separate decode workers handle token generation, transferring KV cache blocks between the two once prefill wraps up. vLLM, SGLang, and NVIDIA Dynamo have all adopted some version of this pattern. The payoff for long-prompt workloads is substantial: disaggregated prefill in vLLM 0.6 and later, and in SGLang, has delivered 2 to 3x throughput gains alongside roughly 2x lower decode latency. The cost is cluster complexity and a higher hardware floor to even get started.

TensorRT-LLM's route into disaggregation runs through NVIDIA Dynamo. A router decides which decode worker is the best fit for an incoming request, and depending on what KV cache blocks are already sitting on a given decode node, the request might skip the prefill stage on that node entirely. Once prefill finishes, KV cache blocks move over to the decode worker via UCX or NIXL, riding on RDMA or NVLink. MPI support is technically still present but has been deprecated, and it's not something to build a deployment plan around going forward.

This kind of architecture introduces failure modes that a single-node scheduler never has to think about. A GitHub issue filed in March 2026 against TensorRT-LLM v1.2.0.rc6.post2, running on an AWS P5en instance with 8x H200 GPUs, described a race condition in the async CacheSender thread: with block reuse enabled, the executor loop could evict a KV cache block from the reuse tree for a new incoming request in the narrow window between prefill finishing and the async UCX transfer to the decode worker, and the result was a silent deadlock. That's not a knock on the engine. It's a demonstration of what disaggregated serving actually costs in complexity once prefill and decode live on different machines: concurrency hazards that simply don't exist when everything runs on one node.

Once prefill and decode are physically separated, the scheduler stops being the thing that determines whether the throughput gain appears. The interconnect does. RDMA throughput, NVLink bandwidth, and how close the KV cache sits to where it's needed become the variables that decide whether disaggregation delivers the promised 2 to 3x, or just adds complexity without the payoff.

Scheduler knobs, operational complexity, and which workload profile favors which engine

Placed side by side, the two engines show a fairly clean pattern. TensorRT-LLM gets the highest raw throughput on NVIDIA hardware, 15 to 25 percent above vLLM on H100 for dense models, but every bit of that requires configuration decisions made before serving even starts. Lyceum Technology calls this "the configuration wall," and it's an apt name: batch size, sequence length limits, and KV cache fraction all need tuning up front, and getting that tuning wrong leaves throughput on the table. That investment makes the most sense for stable, high-volume, predictable workloads, the kind where the upfront engineering cost gets paid back many times over.

vLLM's profile runs the other direction. Support for more than 400 model architectures, a pip install, one CLI flag to get serving, and a memory manager that adapts to variable-length, unpredictable traffic without needing to be told in advance what shape that traffic will take. Per inferenceengineering.tech, it's the sensible starting point for most teams, with a move to something else reserved for cases where a specific, profiled bottleneck actually justifies the switch.

For workloads with sub-100 millisecond latency SLAs, TensorRT-LLM's kernel fusion and CUDA graph approach tends to keep time-per-output-token closer to the hardware's theoretical ceiling, since there's less runtime overhead sitting between the scheduler's decision and the GPU actually executing it. vLLM's continuous batching, by contrast, can let scheduling overhead eat into that budget when SLAs are tight enough that request blocking time starts to rival token generation time itself.

When the workload shifts to high concurrency with unpredictable, highly variable prompt lengths, vLLM's PagedAttention starts pulling ahead: freed VRAM means more requests batched simultaneously, and that runtime flexibility keeps tail latency in check when traffic spikes without warning. For long-context, prefix-heavy workloads, prefix caching sits in a category of its own, with warm production hit rates around 87 percent translating directly into prefill computation that simply never has to happen twice.

Neither engine is chasing the other on features anymore. The scheduling policy is the same idea wearing three different names. What's left to decide is which one's kernel strategy, memory architecture, and operational profile line up with the shape of the traffic actually hitting production, and that's a question with a different answer for nearly every team asking it.