Chunked Prefill Strategies for 128K Token Inputs in vLLM
Balancing GPU memory, cache capacity, and latency when processing prompts longer than 128K tokens.

Serving long-context inputs in vLLM requires tuning chunked prefill, the per-step token budget, and KV cache residency together. Chunked prefill prevents long prompts from blocking decode, but it does not remove the long-context memory wall. The infrastructure must balance GPU activation headroom, cached tokens, and offload bandwidth while preserving accuracy and latency under concurrent load.
Why long-context inputs break vLLM's default scheduler
A long prompt turns full prefill into a head-of-line blocking problem. Without chunked prefill, the compute-heavy operation holds up decode requests that need only their next token.
The scheduler works one iteration at a time, with a token budget that limits the batch for that step. Full-prefill behavior does not provide the smaller scheduling boundaries needed to interleave decode with a long input. As a result, a prefill that monopolizes execution prevents the scheduler from servicing those waiting requests during that window. With long inputs, this slow prompt processing increases serving latency.
How chunked prefill works mechanically inside vLLM's scheduler
Chunked prefill divides the input sequence X into non-overlapping chunks of size B_CP. Each chunk computes attention over its own keys and values plus the keys and values of all preceding chunks. The boundaries divide the scheduling work while preserving the context available to attention. Processing a later chunk therefore still depends on the KV state built earlier.
With chunked prefill, decode receives budget first, at one token per active sequence. The scheduler then assigns the remaining budget to the next prefill chunk. The next step shares the budget in the same way, so the prompt moves forward without starving decode.
Prefill is compute-bound, whereas decode is memory-bound, so placing them in the same GPU step can improve SM utilization. The benefit comes from both the scheduling boundary and the mix of work within it. Chunk size and the token ceiling must keep that balance, even when more prefill work could fit in a step on its own.
The memory pressure chunked prefill creates and cannot avoid
Chunking reduces the size of a prefill operation, but the resulting KV state still accumulates across the input. Larger chunks also have larger activation peaks, which leave less VRAM for the cache. Smaller chunks leave more room for KV blocks but require more steps before the long request can generate its first token. The scheduling solution thus introduces a direct trade-off between cache capacity and time to first token, or TTFT.
vLLM establishes that trade-off during startup. It loads weights, performs a profiling forward pass at the maximum batch, and reserves memory for the activation peak and CUDA graph pool. The remainder becomes the KV block pool, managed as a fixed integer free list that replaces runtime allocation. An oversized activation reservation therefore reduces cache capacity even when decode does not need the reserved space.
Florida State's KV cache comparison reports that OOM failures can occur earlier in multi-GPU configurations despite greater aggregate memory. Attention score computation remains localized to a single GPU and does not benefit proportionally from the added memory. Tensor parallelism therefore cannot be treated as an automatic escape from activation pressure. The three connected levers are chunk size, token budget, and KV cache residency, and each must be tuned against the constraints imposed by the others.
Chunk size and token budget: how to set them for long-context workloads
Chunk size should follow the workload’s latency priority. Smaller chunks protect short-request latency in mixed short/long workloads and leave more KV pool headroom for concurrency. The cost is more scheduling steps and higher TTFT for the long request. Larger chunks suit batch workloads dominated by very long inputs because they reduce total prefill time, but their activation reservation shrinks the cache pool.
The --max-num-batched-tokens setting caps how many tokens are processed in each iteration across all sequences. Raising it can increase throughput, but it also raises the activation peak for each step that is used to plan memory use. It must accommodate decode work and the intended prefill chunk without making the profiling pass over-reserve VRAM. A chunk that looks efficient in isolation may therefore be unsuitable once concurrent decode consumes part of the budget.
The --max-num-seqs setting governs scheduler concurrency, whose safe limit for long-context requests depends on available memory. Engineers should derive it from KV memory per user and the pool remaining after weights and activation reservations. For long-context requests, that calculation can yield a safe concurrency far below the default ceiling. Companies that build inference storage layers, including Open Lake, use benchmarks and standard tests of performance to study how to tune infrastructure when activation pressure affects the KV block pool.
KV cache residency decisions: what stays on GPU, what moves off, and at what cost
When the on-GPU pool is exhausted, residency becomes the binding constraint. A deployment facing single-digit concurrency on one GPU must either accept that limit or introduce a storage tier that manages KV blocks across GPU memory and NVMe. Keeping hot blocks in HBM avoids transfers, while moving blocks off GPU frees capacity at a reload cost. More storage alone does not establish that the trade is worthwhile.
HBM delivers TB/s, whereas CPU–GPU PCIe 5.0 delivers roughly 64 GB/s, about 2% of HBM bandwidth. With so much less bandwidth, reading a large cache across PCIe can take orders of magnitude longer than reading it from HBM. Offloading can turn prefill from compute-bound to memory-bound. Chunk size should consequently be set so that compute time for one chunk roughly matches the I/O time needed to prefetch the next chunk’s KV blocks.
Cake, by Jin et al., demonstrates the value of pipelining storage loads with prefill computation. It achieves an average roughly 2.6× TTFT reduction against compute-only or I/O-only methods by keeping both resources busy. NVIDIA Dynamo addresses distributed inference, and its KV transfer library NIXL supports RDMA, NVMe-oF, and S3 for inter-node movement. These tools deal with different parts of the residency problem, and bandwidth still limits what they can do.
Open Lake's storage layer is purpose-built to manage which blocks remain hot in HBM, which migrate to local NVMe, and when reloading them is worth the bandwidth cost. Its research into multi-tier KV management examines whether storage latency can be hidden behind concurrent decode work. The engineering distinction is an I/O pipeline that sustains bandwidth under concurrent KV-prefetch reads, rather than a generic NVMe configuration selected for capacity alone. Whether that design keeps the storage tier from becoming the next bottleneck depends on the workload’s shape and chunk size.
KV cache quantization as a multiplier on every residency decision
FP8 KV quantization roughly halves the memory cost of each cached token compared with BF16. That reduction changes every residency calculation because more tokens fit on GPU and fewer bytes must cross the offload path. It can support more concurrency than BF16 with the same cache budget. Quantization should therefore be settled before final pool sizing and the rest of the configuration are fixed.
The precision cost requires explicit testing at long context. vLLM's FP8 KV cache post describes accumulation precision loss in the FP8 Flash Attention 3 kernel on Hopper GPUs. On an extended-context needle-in-a-haystack task, FP8 accuracy dropped dramatically from the BF16 baseline because of imprecise FP32 accumulation in the Tensor Cores. A fix is available, but engineers must check that the vLLM version they run has it and run the long-context needle-in-a-haystack test themselves after they enable FP8.
Lossless compression in transit addresses a different part of the same bandwidth problem. The SplitZip preprint proposes fixed-length exponent coding for BF16/FP8 tensors in disaggregated serving pipelines. It uses NVIDIA's nvcomp library, including LZ4, Cascaded, and Bitcomp, as a benchmark for its own compression mechanism. This approach cuts the bytes transferred without another lossy change to how the data is stored, but the chosen KV precision still needs to be tested.
Prefix caching and KV eviction as complements to chunked prefill for long-context inputs
After a chunk produces KV blocks, later requests can reuse them to avoid repeating the computation. vLLM's Automatic Prefix Caching, enabled with --enable-prefix-caching, finds the longest cached prefix of an incoming request and skips its prefill. This is effective on a single worker. But in a multi-replica deployment, each replica has its own local cache, so cross-node reuse is zero without an external layer.
If overlap is high, prefix caching deserves tuning because it avoids repeated work. If overlap is low and generation spans long horizons, retaining KV blocks under a constrained KV budget becomes more important. Mooncake (FAST '25) targets agentic access patterns with lower prefix overlap by replacing single-producer broadcasts with KV block routing from prefill to decode through a distributed store.
Retention introduces an accuracy risk that prefix reuse does not resolve. Attention-guided eviction can discard a token that has not mattered recently but becomes critical much later. TRIM-KV (ICLR 2026) instead learns retention scores when each token is created to predict how useful it will be later, and reports gains over attention-guided baselines, especially when memory is low. Its distilled retention gate adds negligible inference overhead and produces a scalar score per token per layer/head. vLLM RFC #12254 (January 2025) proposed a pluggable framework for sparse KV management, but a proposal should not be mistaken for a deployed eviction capability.
Sparse and approximate attention during chunked prefill: what CompactAttention and QUOKA actually offer
If attention computation still dominates after memory and scheduling are tuned, sparse attention offers another avenue. These methods cut the work inside each chunk while keeping the way the scheduler shares its token budget. Their gains can compound with chunked prefill, but their selection rules can also omit information needed for accurate answers. Their readiness to replace dense attention in production requires evaluation on the workload.
CompactAttention uses Block-Union selection at the block level and is designed to integrate with the FlashInfer paged attention backend. QUOKA (2026) is a training-free, hardware-agnostic sparse attention algorithm for chunked prefill. Its more aggressive token-level query subsampling exploits the tendency of many queries to attend to a smaller group of keys. That subsampling can miss KV entries critical to queries that were not sampled, degrading tasks that require access to distributed information.
A good speed result leaves some retrieval patterns untested. Engineers should run long-context needle-in-a-haystack and broader long-context retrieval evaluations against any candidate sparse attention kernel before production deployment. Those tests should establish whether the reduced attention work preserves the workload’s accuracy requirements. Neither CompactAttention nor QUOKA removes the need to size the cache or match prefetch bandwidth to chunk computation.
A tuning decision tree for long-context chunked prefill deployments
The first gate is the workload’s access pattern. Test whether incoming requests have high prefix overlap or have both low overlap and long-horizon generation. High overlap leads to Automatic Prefix Caching and an assessment of whether replica-local reuse is sufficient. Low overlap leads to closer evaluation of KV retention and transfer patterns, rather than an assumption that prefix caching will absorb the cost.
The second gate is precision. Enable FP8 KV cache with --kv-cache-dtype fp8 and run a long-context needle-in-a-haystack test against the BF16 baseline. If accuracy holds and the corrected kernel is in place, use FP8’s smaller per-token footprint when sizing the pool. If it does not hold, do not base capacity commitments on that reduction. On Blackwell, evaluate NVFP4 as part of this precision decision.
The third gate is how much the GPU can hold at the target chunk size and token budget. Calculate the KV pool remaining after weights and activation reservations, then set --max-num-seqs from KV memory per user. If the pool supports the required concurrency, proceed to latency testing. If it does not, reassess chunk size and move to the residency gate, testing whether prefetch I/O can keep pace with chunk compute under concurrent load.
The final gate is the remaining bottleneck. If storage cannot keep pace, capacity offload has merely shifted the constraint downstream. If attention dominates instead, evaluate CompactAttention or QUOKA and retain the change only if the accuracy benchmarks hold. Across these branches, the decisions interact, so the fastest path to a tuned system is to instrument TTFT and inter-token latency under realistic load before changing more than one variable at a time.
Sources
- Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
- Compute Or Load KV Cache? Why Not Both?
- Comparative Characterization of KV Cache Management Strategies for LLM Inference
- QUOKA: Query-Oriented KV Selection For Efficient LLM Prefill
- CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection


