Pipeline Parallelism Latency Bubbles and Mitigation in Multi-Node Inference

Recent systems show that pipeline bubbles stem from two separate problems requiring different fixes.

· 10 min read
Cover illustration for “Pipeline Parallelism Latency Bubbles and Mitigation in Multi-Node Inference”
Distributed Serving · October 6, 2026 · 10 min read · 2,190 words

Pipeline parallelism has become the default way to split large language models across multiple nodes, and it carries a structural cost in the form of idle GPU time, known as pipeline bubbles. Two distinct mechanisms produce these bubbles, and a wave of recent systems work, from Sarathi-Serve to VPP to TD-Pipe and gLLM, shows that treating them as one problem leads engineers to fix the wrong thing.

Why pipeline parallelism is the default choice for multi-node LLM inference

Splitting a model across GPUs demands a choice about where the communication happens, and that choice shapes everything downstream. Tensor parallelism splits individual layers across devices. Every layer needs an all-reduce collective to stitch partial results back together before the next layer can run. Pipeline parallelism takes a different approach: it assigns whole groups of layers to different devices, and the only thing that crosses the wire is the intermediate activations passed from one stage to the next. On commodity GPU servers without that kind of interconnect, it isn't. PipeMax describes pipeline parallelism as "the most promising option for commodity GPU servers": tensor parallelism becomes communication-bound in exactly that setting, while pipeline parallelism's activation-only traffic stays manageable. That difference in communication volume is what makes pipeline parallelism the practical starting point whenever a model has to span more than one node.

The cost of that choice is visible in waiting. A pipeline is a sequence of stages with dependencies between them: stage two cannot process a micro-batch until stage one has finished producing it. Any time a stage sits idle because the stage before it hasn't delivered work yet, that idle time is a pipeline bubble, and bubbles reduce the fraction of total GPU time spent on useful computation. Training systems have studied this problem for years, and they get some relief because the backward pass can be scheduled to fill gaps that the forward pass leaves open. Inference has no backward pass. There is no second computation phase to interleave with the first, so whatever idle time the pipeline structure creates has nothing to absorb it. The bill for choosing pipeline parallelism to avoid tensor parallelism's communication overhead comes due later, in the form of bubbles that cut directly into latency and throughput.

The two structurally distinct sources of pipeline bubbles

Pipeline bubbles in LLM inference come from two separate mechanisms, and the distinction between them is the single most useful fact for evaluating any mitigation technique. One is fill-and-drain idle time, which is inherent to running a pipeline. The other is execution imbalance, which comes from the workload itself and can be reduced, though only by the right kind of intervention.

Consider a four-stage pipeline processing a stream of requests. At startup, stage four cannot do anything until stage one has produced output, stage two has processed that output, and stage three has passed its own result along. That ramp-up period, where downstream stages sit empty waiting for the pipe to fill, is unavoidable given that the stages depend on each other. The same thing happens in reverse at the end of a batch: stage one runs out of new work and goes idle while stages two through four finish draining whatever is still in flight. VPP's own account of pipeline behavior calls this out directly, treating "cross-request bubbles (drain-phase idle time)" as a category that needs its own fix, separate from anything aimed at reducing imbalance. No scheduling trick removes fill-and-drain bubbles. The only levers that change their size are the depth of the pipeline and the number of stages involved.

Now suppose a long prefill request gets injected into that same four-stage pipeline midway through a run of decode-only micro-batches. Stage one suddenly takes far longer to process that micro-batch than it took for the decode batches around it, because prefill is compute-intensive while decode is bound by memory bandwidth. Stages two through four, having finished their share of the surrounding decode work quickly, now sit waiting for stage one's slow prefill chunk to arrive. That wait is a function of how much the compute time of one micro-batch differs from the compute time of its neighbors, and in LLM serving, the composition of prefill and decode tokens inside a micro-batch is the main source of that variance, not of pipeline depth. VPP describes this directly as chunking that results "in increasing chunk latency along the sequence." Equal-size chunks, in other words, do not produce equal execution times once the KV cache they attend to keeps getting longer.

These two mechanisms call for different fixes, a distinction that matters through everything that follows. A technique built to homogenize micro-batch compute time, such as chunked prefill, does nothing about the idle time a pipeline generates simply by filling and draining. A technique built to shrink the fill-drain window does nothing about imbalance caused by mixing prefill and decode tokens. If a system treats the two as one problem, it risks over-engineering whichever mechanism is easier to measure while leaving the other one untouched.

How prefill–decode mixing amplifies both bubble types simultaneously

LLM inference is a particularly unfriendly environment for pipelines because the prefill-decode split hits both bubble mechanisms at once. Prefill chews through many tokens in parallel and is limited by how fast the GPU can compute; decode produces a single token per step and is limited by how fast data can move through memory. When both of these sit inside the same pipeline iteration, on the same set of stages, the time each stage takes to finish its micro-batch becomes close to unpredictable. That unpredictability is exactly the imbalance mechanism described above, and prefill-decode mixing is its primary source in production serving systems.

The fill-drain mechanism gets worse too, for a related but separate reason. A pipeline only amortizes its fill-drain overhead well when it settles into a steady rhythm, where the cost of filling and draining is small compared to the useful work done between those events. Frequent switching between prefill-heavy and decode-heavy micro-batches prevents that steady state from forming. The pipeline is constantly adjusting to a new compute profile, so the window in which fill-drain overhead gets diluted by productive work never opens wide enough to matter.

Longer prompts and larger batch sizes push this further in the same direction. A long prefill occupies several consecutive iterations before decode can even begin, sustaining the imbalance it introduces across multiple pipeline cycles.

There is a third factor layered on top of both: KV cache capacity. PipeMax identifies a specific version of this problem on the decode side, where the KV cache belonging to batches that are not currently active still occupies GPU memory it does not need to occupy, tightening the pressure on everything else sharing that memory. Prefill-decode mixing is three compounding problems: compute-time variance from the phase mix itself, loss of pipeline steady-state from frequent phase switching, and memory pressure from KV cache growth that slows stages down independently of what they are computing.

Chunked prefill as the first mitigation

Sarathi-Serve, presented at OSDI 2024, built the first widely adopted answer to the pipeline imbalance caused by variance in iteration time, and it earned its influence by targeting the actual source of the variance. Its core observation is straightforward: if every micro-batch does roughly the same amount of computation, pipeline bubbles caused by imbalance shrink, because the large swings in iteration time across an LLM serving workload create the imbalance. Every micro-batch ends up carrying a bounded, roughly comparable amount of compute, and the problem of a long prefill derailing the rhythm of a stage gets substantially less severe. The result is a more even pipeline flow and a real reduction in imbalance-induced bubbles. That is why chunked prefill scheduling has become a standard building block in serving systems that came after it.

It does not, however, reach the second source of imbalance described earlier: the growth in attention cost across a long sequence's chunks. Fixed-size chunks avoid the problem of one chunk being enormous relative to its neighbors, but they do not avoid the problem of a chunk late in a long sequence attending to a much longer prefix KV cache than a chunk early in that same sequence. Equal token counts do not translate into equal execution times once the attention cost itself is climbing chunk over chunk. On long sequences, those costs can outweigh whatever balance the resizing buys back.

Chunked prefill, in its original form, also leaves two other things untouched. Fill-drain bubbles remain exactly as large as the pipeline's depth and stage count dictate, regardless of how chunks are sized, because chunk sizing is an imbalance lever and fill-drain is not an imbalance problem. Chunked prefill likewise says nothing about imbalance between different concurrent requests competing for the same stages, and nothing about the KV memory pressure that can force a stage to stall while evicted cache blocks are reloaded. None of this diminishes what Sarathi-Serve accomplished. It solved the problem it set out to solve, cleanly and durably. It simply was never aimed at the other two problems; later systems had to be built to address them.

Virtual pipeline stages and V-shaped traversal as a second-generation response to long-sequence imbalance

VPP takes the residual problem that chunked prefill leaves behind, the growth in chunk execution cost across a long sequence, and turns it from an obstacle into a planning tool. The growth is predictable: each chunk attends to a prefix that is exactly one chunk longer than the one before it, so the increase in attention cost follows a pattern that can be anticipated in advance. VPP's contribution is to exploit that predictability directly: it keeps every chunk the same fixed size and reshapes how chunks are assigned to pipeline stages over time, through a V-shaped traversal through virtual pipeline stages that overlaps each chunk's heaviest middle portion of work with the lighter head and tail portions of neighboring chunks, so that the heavy and light parts of different chunks land on the pipeline's stages at the same time.

VPP pairs that fix with two further mechanisms aimed squarely at the fill-drain overhead described earlier, which chunk sizing was never going to reduce. Asynchronous communication, combined with reordering stages across chunks, improves how much communication overlaps with computation, cutting down the communication latency exposed at stage transitions. Pipelined packing fills the drain period at the end of one request's processing with the leading chunks of the next request, amortizing some of the structural idle time that no scheduling policy can fully erase. Together, these two additions mean VPP is not only a fix for imbalance, it is a more complete treatment of both bubble mechanisms within a single system.

The results reported for VPP are substantial: on the 512K-token DeepSeek-V3.1 workload, the pipeline bubble ratio drops from 6.4% under dynamic chunk resizing to 0.1% under VPP, a 98.0% reduction. Within that scope, VPP's result matters because of what it declines to do. Rather than fighting chunk-to-chunk imbalance with ever-finer resizing and the overhead that comes with it, VPP accepts the imbalance as a known quantity and designs the pipeline layout around it.

Temporal decoupling and inter-batch work stealing as a third lever: TD-Pipe and gLLM

Chunking and layout are not the only places to intervene. A separate class of systems addresses pipeline bubbles by changing how work gets scheduled across requests and across time, which reaches imbalance sources that neither chunk sizing nor virtual stages touch.

TD-Pipe takes the opposite approach from Sarathi-Serve at the design level. Where Sarathi-Serve deliberately mixes prefill chunks into decode iterations to smooth out compute time, TD-Pipe separates prefill and decode in time, so the mismatch between their compute profiles never gets a chance to propagate through the pipeline's stages. Alongside that temporal separation, TD-Pipe applies work stealing, letting faster-finishing parts of the system take on work from slower ones. Work stealing targets a source of bubbles that neither chunked prefill nor virtual pipeline stages reach: imbalance between different batches of requests, as opposed to imbalance within a single stream of chunks. TD-Pipe has become something of a reference point for later systems in this space. PipeMax, for instance, builds on the baseline TD-Pipe established while departing from several of its specific choices, including its policy for switching between phases and its assumptions about when work stealing should trigger.

gLLM approaches the same imbalance between batches of requests from a different angle, through a mechanism it calls Token Throttling. Rather than fixing the prefill and decode token budgets in advance, gLLM adjusts the batch sizes allocated to prefill and decode independently, guided by the system's real-time inference status. By letting each phase's batch size move on its own, gLLM keeps the two from fighting each other for the same fixed allocation.

Taken together with chunked prefill and virtual pipeline stages, these scheduling-level approaches show that pipeline bubbles in LLM inference do not have a single point of attack. Fill-drain idle time, within-sequence imbalance, and inter-batch imbalance are three separate axes, each addressed by a different mechanism in the systems built to handle it, and the strongest serving systems treat them as three problems.

Sources

  1. 1 Introduction
  2. VPP: Virtual Pipeline Parallelism for Efficient Chunked Prefill in Long-Context LLM Inference
  3. gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
  4. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  5. gLLM: Global Balanced Pipeline Parallelism Systems for Distributed LLMs Serving with Token Throttling

More in Distributed Serving