Tokens per Second vs Latency Tradeoffs in Throughput-Optimized Serving Configs
Optimizing for throughput often means sacrificing the latency users actually feel.

A single H100 can report a blistering tokens-per-second figure while the person typing into the chat window waits a full second for the first word to appear. Both numbers are true, both come off the same hardware, and both are the subject of this piece: aggregate throughput and per-request latency measure different things entirely, and treating them as one dial produces serving configurations that look excellent on a monitoring dashboard and feel sluggish to the end user. The Dell Technologies KV cache survey, published in March 2026, frames this as two competing objectives baked directly into the serving problem: pushing down latency for an individual user while pushing up the combined tokens-per-second figure across every concurrent request the fleet is handling. That the same model, run by different providers, produces wildly different speeds is not noise. Morph LLM's 2026 optimization guide notes that the gap is a direct consequence of serving configuration choices, not a hardware lottery. What follows traces that configuration gap down to its source: a resource allocation problem inside the key-value cache, where every batching decision and every scheduling policy either frees memory for more users or locks it up and makes someone wait.
How prefill and decode pull hardware in opposite directions, and where memory bandwidth becomes the real constraint
LLM inference serving exposes two metrics that appear to be on the same scale but optimize for different stakeholders: aggregate throughput (tokens per second summed across all concurrent requests) and per-request latency (time to first token and inter-token delay for a single user). Prefill reads the entire input prompt at once to produce the first output token, and it is compute-bound: the GPU runs near peak utilization, and the user feels that as the delay before anything appears on screen. Decode comes after, generating one token at a time, and it behaves almost the opposite way: GPU compute sits mostly idle, and the pace of generation depends instead on how fast stored keys and values can be pulled out of memory.
That memory-bandwidth limit is not theoretical. Hardware profiling by researchers at Alibaba Cloud examined vLLM's XFormers backend running Llama2 on a single H20 GPU and found the attention kernel suffering frequent cache misses during KV cache loading, repeated trips to high-latency HBM, and large numbers of stalled warps. Compute throughput was 23.35% of peak, while memory throughput stayed under half of peak, the signature of a system waiting on memory rather than one running out of arithmetic to do. That has a direct consequence for anyone buying hardware to fix a slow deployment: adding a faster GPU without changing how keys and values move through the memory hierarchy does nothing for the decode bottleneck. It simply means the GPU sits idle on more expensive silicon.
The Dell survey identifies memory bandwidth and GPU memory capacity as the two constraints that come under the most pressure as context windows stretch longer. Both constraints trace back to the same structure: the KV cache is the exact point where prefill's compute-heavy write and decode's bandwidth-heavy read meet, and it is the one resource both phases depend on and compete over.
Why KV cache allocation is the actual decision being made when an operator tunes batch size or scheduling policy
Once the KV cache is understood as the shared resource prefill and decode both pull against, the configuration knobs operators reach for first start to look different. Batch size, scheduling policy, memory reservation settings: each of these, at the hardware level, is a decision about how KV cache memory gets divided among requests competing for it, and that division sets both how many requests can run at once and how long each one sits waiting.
Batch size is the lever most people reach for, because it is the most visible one. Running more requests together raises the aggregate tokens-per-second number, but it does so by forcing every request already in the batch to give up some of its share of KV cache memory, which either caps how long a sequence can run or forces the system to evict and recompute data it already had. Static batching, the older approach, makes this worse by holding a batch together until every request in it finishes, even when short requests complete early and simply sit idle holding memory a new request could use. Continuous batching fixes that by inserting new requests the moment old ones complete rather than waiting for the whole batch to clear, and Morph LLM's 2026 guide describes the resulting throughput gain as large on identical hardware. The mechanism behind that number is simple: static batching leaves KV cache space idle under short completed requests while long ones keep running, and continuous batching reclaims that idle space immediately and fills it with new work, raising the share of GPU time spent on decoding that actually produces useful output.
The Dell survey reports that pre-allocating KV cache memory for the maximum possible sequence length, the simplest and most common early approach, wastes up to 90% of GPU memory, because most requests never come close to using the full context window they're allocated for. That number is the dominant cost of the allocation strategy, not a rounding error in an otherwise efficient system, and no scheduling policy layered on top of it can recover memory that was never efficiently set aside to begin with.
What PagedAttention Changed About the Allocation Problem
PagedAttention, the memory management approach built into vLLM, directly targets the waste that maximum-length pre-allocation creates. Instead of reserving a fixed block of memory sized for the worst case, it splits the KV cache into small pages allocated on demand, cutting memory waste by roughly 96% and letting far more requests fit into the same GPU memory at once. That gain comes at a small, well-understood cost: a modest per-token compute overhead from the page-table indirection needed to track which page holds which piece of a sequence, an exchange that holds up well in most production settings.
What PagedAttention does not touch is the interference between request types that still share the same GPU. PagedAttention solves fragmentation, not scheduling, and the memory-bandwidth bottleneck identified in decode persists regardless of how cleanly memory is packed. The clearest symptom occurs when a user submits a long prompt: prefill for that single request can occupy the GPU's compute for several seconds, and during that stretch every other active user's decode stream stalls, producing the kind of latency spike that shows up as a complaint rather than a metric. The Dell survey and Morph LLM both treat this as the next problem in sequence, not a separate one: once memory allocation stops being the bottleneck, the scheduler still has to decide how to interleave prefill and decode work across requests so that neither one starves the other.
Chunked-Prefill Scheduling and Head-of-Line Blocking
Sarathi-Serve's chunked-prefill scheduling addresses that interleaving problem directly, and it does so by giving up a small amount of speed for one user in exchange for stability for everyone else. Instead of running a long prompt's prefill in one uninterrupted block, the scheduler splits it into chunks of roughly 512 tokens, small enough to slot in alongside ongoing decode batches rather than displacing them entirely. New requests join the running batch without forcing existing decode streams to pause.
The cost of that approach falls on the request being chunked: its own time-to-first-token rises slightly, since its prefill is now spread across several scheduling rounds instead of processed in one pass. In exchange, every other concurrent user's inter-token latency stops spiking, because the GPU is no longer locked onto a single long prefill for seconds at a stretch. That exchange, a little latency given up by one request to stabilize latency for many, is the clearest illustration in the whole serving stack of what the throughput-latency tradeoff actually costs in practice.
Chunked prefill is a scheduling technique, but it only works because PagedAttention's on-demand paging provides the spare memory needed to hold the partially-processed state of an interleaved request; without that allocation technique underneath it, the whole scheme has nowhere to put its intermediate work. The Dell survey lists chunked prefill among the principal scheduling strategies used in high-throughput datacenter serving, and describes its pairing with paged allocation as the condition that makes stable latency possible under workloads where request lengths vary widely. The approach is not free of overhead. Splitting prefill into chunks means more scheduling decisions per request, and the 2025 paper "Beyond the Buzz: A Pragmatic Take on Inference Disaggregation" notes that related disaggregated prefill/decode designs run into similar overhead when the deployment's actual traffic pattern doesn't match the assumptions the scheduler was tuned for. Scheduling of this kind makes better use of the memory PagedAttention has freed up, but it does nothing to shrink how much memory each token's keys and values actually take up in the first place. That reduction is a separate lever.
KV Cache Quantization and Architectural Compression
Cutting the precision or the rank of the numbers stored in the KV cache attacks the decode bottleneck from a different angle: fewer bytes streamed per token means less pressure on the memory bandwidth that decode depends on, regardless of how well the scheduler interleaves requests. Several distinct techniques do this, and each trades accuracy against throughput in a different way, so the right choice depends on the workload rather than on a single best answer.
Storing keys and values in FP8 instead of BF16 halves the bytes that have to move during every decode step, striking directly at the memory-bandwidth bottleneck identified in the hardware profiling earlier in this piece. Measured results show a small drop on reasoning benchmarks, strong quality retention at context lengths up to a million tokens, and inter-token latency that scales far more gently than it does under BF16. FP8 e4m3 is described as the common production default, as near-lossless and hardware-native since Hopper.
Pushing further, KIVI's 2-bit quantization scheme delivers a substantial throughput gain, the Dell survey puts it at multiple times the baseline, with an accuracy drop ranging from near zero up to about 3%, depending on model and benchmark. The survey flags one caveat: models built with a single KV head may need to stay at 4-bit precision rather than drop to 2-bit, to stay within acceptable accuracy bounds. H2O takes an eviction-based approach instead of quantization, discarding less important cached tokens outright, and reaches up to roughly 29 times the throughput of FlexGen with 5 to 10 times less memory used. That comes with its own risk: H2O can lose "heavy-hitter" tokens to accumulated attention bias, a failure mode that matters most in reasoning workloads where a handful of earlier tokens carry disproportionate weight in later generation.
The more radical move changes the architecture rather than the arithmetic. DeepSeek's Multi-head Latent Attention stores a low-rank projection of keys and values instead of the full tensors, reaching compression ratios far beyond what quantization alone achieves and representing something close to the architectural ceiling for KV efficiency. The absolute gap this produces is large: a sizable Llama model running long context with grouped-query attention and FP8 needs substantially more KV cache memory than a DeepSeek V4-Pro model handling the same context length with MLA and FP8, itself already a steep drop from a full-precision multi-head attention baseline. None of these techniques wins outright across every situation. The Dell survey is explicit that no single method dominates across all settings, and the right strategy depends on the context length in use, the hardware available, and the shape of the workload being served. Newer approaches try to exploit the structure of language itself rather than pushing precision lower still: ChunkKV compresses semantic chunks instead of isolated tokens, aiming to preserve linguistic structure even under aggressive compression, while RocketKV runs a two-stage process, first a coarse eviction pass, then fine-grained sparse attention selection among the survivors.
What happens to the tradeoff when the KV cache no longer fits in GPU HBM

All of the techniques above assume the KV cache still fits somewhere inside the GPU's own high-bandwidth memory. Once concurrent long-context requests push past that capacity, the tradeoff is no longer about scheduling or precision but about where the overflow lives and how quickly it can be brought back, which turns NVMe and remote storage into core serving infrastructure rather than an emergency overflow valve.
On an H100 serving a large Llama model at very long context length, a single user's KV cache already takes up a large share of the GPU's entire HBM, and eight concurrent users at that same context length would require several times more HBM than one H100 physically has. Serving even a small handful of long-context users from a single H100 means some of that cache has to move off the GPU entirely.
The storage hierarchy that has emerged to handle this runs in three tiers, and each step down trades bandwidth for capacity. GPU HBM holds the hottest blocks at around 3.35 TB/s. CPU DRAM holds warm blocks, data from completed turns likely to be needed again in a multi-turn conversation, over PCIe 5.0 at roughly 63 GB/s. Cold blocks move to NVMe SSD, at around 7 GB/s. Every step down that hierarchy costs latency, which is the price paid for capacity the GPU itself doesn't have.
NVIDIA moved to standardize this hierarchy with its Inference Context Memory Storage Platform, announced at CES 2026 and now called CMX, a four-tier design using NIXL and DOCA Memos to move data directly between GPU and NVMe, with BlueField-4 DPUs handling the transfer so it doesn't compete with the GPU's compute path.
That hierarchy is not free to use indiscriminately. For a single-shot chat exchange running under a few thousand tokens of context, the overhead of reaching out to NVMe adds latency without saving anything, since there was never enough data to justify the trip. NVMe bandwidth runs two orders of magnitude below HBM, and for workloads built around continuous decode, reloading cold blocks from that tier can push inter-token latency high enough to break a production service-level agreement. For very long single-turn RAG contexts, the improvement from loading cached KV blocks rather than recomputing is roughly 11×. For a long context with high prefix overlap, loading cached KV blocks from NVMe instead of recomputing the prefix from scratch cuts time-to-first-token by roughly 13×. The storage hierarchy extends what fits, but it does not erase the cost of reaching for it, and knowing when a request needs that reach is itself part of the allocation decision this entire tradeoff has been about from the start.


