TensorRT-LLM Paged KV Cache Configuration for Multi-LoRA Serving
How to allocate GPU memory efficiently when serving multiple LoRA adapters alongside KV cache.

KV cache is what lets a transformer skip recomputing attention over every prior token at each decode step: it stores the key and value tensors from past tokens once, then reuses them. That's the whole trade. You spend memory so you don't spend compute, over and over, for the life of a generation. The exchange rate on that trade gets steep fast: for Llama 3.1 70B running at BF16 with a 128K context and a single concurrent user, the KV cache alone runs around 42.9 GB. That's larger than the model's own weights would be at FP8 precision. One user, one long conversation, and the cache already outweighs the model.
The math doesn't stay still, either. KV cache size scales with model size, batch size, and sequence length all at once, and in long-context serving those three multiply against each other rather than just adding up. Pushing context length and concurrency at the same time steepens the memory curve quickly.
Multi-LoRA serving stacks another variable on top of all this. Each adapter a request routes to carries its own per-request memory demand, and that demand has to sit in the same GPU memory as the base model's weights, its activations, and the KV cache pool. None of these four gets to expand into space the others aren't using. They compete directly for the same finite pool of VRAM, and TensorRT-LLM's management of that competition determines the outcome: a multi-LoRA deployment runs fast, runs unstable, or runs out of room.
TensorRT-LLM's paged KV cache structure: blocks, pools, and the memory pre-allocation contract
TensorRT-LLM handles this with paged KV cache, an idea borrowed pretty directly from how operating systems handle virtual memory. Instead of demanding one long contiguous slab of memory for each request's KV state, the cache gets carved into fixed-size blocks that don't need to sit next to each other in memory. That one design choice is what makes serving variable-length requests at scale practical: no request has to wait for a perfectly-sized contiguous chunk to open up.
Each block holds the KV state for a fixed number of tokens, and it packs in multiple transformer layers at once rather than allocating layer by layer. But that packing comes with a hard constraint, not a suggestion: every layer sharing a block has to have the same number of attention heads and the same attention window size. Mixed-window or mixed-head-count layers can't share a block, structurally.
That constraint is why TensorRT-LLM builds a separate memory pool for every distinct combination of attention window size and KV head count, rather than one pool for the whole model. It's a direct consequence for architectures using grouped-query or multi-query attention: since GQA models define fewer KV head-groups than a standard multi-head model would, their blocks come out smaller per block, because there are fewer head-slots to fill. GQA and MQA aren't just faster to compute. They're structurally cheaper to store.
Block size selection and its effect on fragmentation, reuse granularity, and adapter coexistence
Block size does two jobs simultaneously, and those two jobs pull against each other. It sets the smallest unit the cache can allocate, and it sets the smallest unit the cache can reuse. Tuning for one means fighting the other.
Bigger blocks mean fewer kernel launches and better raw compute efficiency, which sounds like a straightforward win until you look at what happens with short requests. A request that only fills part of its last block still has that whole block reserved, and the unused remainder just sits there, wasted, until the request finishes. Run a workload full of short generations and large blocks start bleeding memory on padding alone.
Smaller blocks fix that: reuse gets finer-grained, waste drops, and utilization improves on short, varied requests. The cost is metadata. Smaller blocks mean more block-table entries per request and more bookkeeping overhead for the runtime to track, which eats into some of the efficiency gain you were chasing.
Multi-LoRA traffic makes this genuinely harder to tune, not just annoying. Different adapters tend to serve different tasks, and different tasks produce different output-length distributions. An adapter fine-tuned for short classification-style responses and one fine-tuned for long-form generation share no length profile. A block size dialed in for one adapter's typical output can end up wasteful for the other, and a single global block-size setting is, by definition, a compromise across every adapter sharing the deployment. There's no block size that's simultaneously optimal for a chatbot adapter and a summarization adapter unless their length distributions happen to line up, which they usually don't.
Configuring the KV cache pool allocation when adapter GPU cache competes for the same VRAM
In a multi-LoRA deployment, model weights, activation tensors, the KV cache pool, and the GPU-side adapter cache all need to fit into the same GPU VRAM budget at once. None of them is optional, and none of them can borrow space from another mid-request without something else getting squeezed.
TensorRT-LLM gives you two knobs for adapter caching. The host-side cache gets a max size you set directly. The GPU-side cache gets sized as a percentage of whatever GPU memory is free after the engine loads. Both are configurable, but the defaults on both assume the KV pool isn't competing hard for the same memory, which in a busy multi-LoRA deployment, it usually is.
That competition has a latency cost attached to it. When a request calls for an adapter that isn't already sitting in GPU cache, loading it in takes something like 28 to 45 milliseconds, depending on PCIe bandwidth and whatever else is moving across that bus at the same time, enough on its own to blow a tight SLO. That's not trivial in a latency-sensitive serving setup. A cold adapter load landing on the critical path is enough to blow a tight SLO on its own, even before the request has done any actual inference work.
Adapters get loaded and evicted continuously as traffic shifts across the population of lightweight fine-tuned modules, and that churn fragments GPU memory over time, which produces a second, sneakier failure mode. Even when the total free memory a... Adapters get loaded and evicted continuously as traffic shifts across the population of lightweight fine-tuned modules, and that churn fragments GPU memory over time. Even when the total free memory adds up to enough, if it's scattered across gaps too small to hold a new adapter individually, the load fails anyway. That's a different animal from a plain out-of-memory error: the memory technically exists, it's just fragmented, and standard OOM monitoring won't necessarily catch it before it causes a stall.
Configuring KV cache reuse policies for multi-adapter traffic patterns
Reuse works through a radix search tree. As blocks fill up, they get indexed by token ID and stored in that tree, and when a new request comes in, TensorRT-LLM searches for matching prefixes and reuses those blocks instead of recomputing attention from scratch. Shared system prompts, common prefixes, repeated instructions: any of that gets to skip recomputation entirely when reuse finds a match. It saves memory and compute in the same move.
The flag controlling this is enableBlockReuse in KvCacheConfig, and it's on by default in TensorRT-LLM. Turning it off has to be done explicitly. But there's a build-time prerequisite that's easy to miss: reuse requires the model to be built with paged context attention support. You can't flip reuse on after the fact for an engine that wasn't built with that flag set. That decision gets locked in at build time, not at serving time, so it needs to be right before the engine ever ships to production.
When memory pressure forces evictions, TensorRT-LLM defaults to a priority-based eviction policy. Within that priority structure, blocks are evicted according to their assigned priority, with recency used to break ties within a given priority level. In multi-adapter serving, that priority structure is exactly the lever for deciding whether a heavily-used adapter's cached prefixes survive memory pressure or get evicted alongside everything else.
KV cache quantization as a compression strategy to recover pool capacity under adapter memory pressure
KV cache quantization and weight quantization are two separate decisions, and it's worth keeping them separate in your head. One compresses the model's parameters. The other compresses the stored key and value tensors sitting in the cache. You can run a full-precision model with a quantized KV cache, or the reverse, and the choices don't have to match.
On Hopper (H100) and Ampere (A100) hardware, running the KV cache at FP8 instead of BF16 cuts cache memory in half. On those architectures, FP8 is the format to reach for on those architectures.
Blackwell (B200) hardware pushes this further with NVFP4, which cuts KV cache memory by another 50% on top of what FP8 already saves. That's tied to one specific generation of GPU hardware, though. There's no equivalent path on H100 or earlier hardware, so the ceiling on cache compression is tied directly to which generation of GPU is under the workload.
This isn't just theoretical either. NVIDIA's MLPerf v6.0 open-division Blackwell submission ran on HGX B200 (8-GPU), using TensorRT-LLM with tensor parallelism of 4 and expert parallelism of 4, FP4 weights, and FP8 for the KV cache specifically. FP4 weights paired with FP8 KV cache, not FP4 across the board. That pairing is a real signal about where the accuracy-versus-memory line gets drawn in production-grade benchmark work: weights can push further into low precision than the KV cache can, at least for now, on that hardware generation.
For multi-LoRA deployments squeezed for VRAM, the practical read is straightforward: cache quantization is capacity you get back without touching your batch size or trimming your context window. It's memory recovered from a different part of the stack entirely, which puts less pressure on the block-size and reuse-policy tradeoffs described above.
The role of storage infrastructure beneath the serving stack in making KV offloading and adapter loading viable
That 28 to 45 millisecond adapter cold-start cost is fundamentally a PCIe bandwidth number. It's fundamentally a PCIe bandwidth number. How fast data can move from host memory to GPU memory sets a hard ceiling on how fast the adapter cache can refill without adapter loads showing up on the request's critical path and eating into the SLO.
KV offloading runs into the same physics from a different direction. Offloading evicted blocks to CPU DRAM keeps them alive in the reuse tree instead of throwing them away outright, which sounds like a clean win: cheaper storage, same reuse benefit. But offload bandwidth decides whether a block actually makes it back to GPU memory before the request that needs it hits a timeout. Slow offload storage doesn't just add latency. Past a certain point, it turns reuse into a liability instead of a saving, because the request ends up waiting on a fetch that costs more time than just recomputing the block from scratch would have.
Research from 2026 on predictive multi-tier memory management points at a different design. Rather than reacting to evictions after they happen, these systems intercept KV allocation requests up front, size them based on the model's own architecture, and hand back block references that are already aware of which memory tier they belong in. That kind of system works alongside existing serving infrastructure rather than requiring a wholesale replacement of the runtime. The same research projected a substantial cost advantage over reactive offloading schemes.
Reactive offloading, in the style of one prior approach, comes out worse on tail latency specifically, not just on average throughput; 2026 research found this. The paper found reactive approaches underperform predictive tier management on throughput, because the system isn't scrambling to move data only after it's already needed. For any deployment thinking about tiered storage beneath TensorRT-LLM, the practical distinction comes down to this: the storage layer beneath the serving stack either anticipates KV movement or reacts to it after the fact.

A configuration checklist for multi-LoRA paged KV cache deployments, from engine build through runtime tuning
Start at the engine build stage, because some of these decisions can't be revisited later. Confirm the engine is built with paged context attention enabled if reuse matters to the workload at all, since that flag can't be added retroactively once the engine is compiled.
From there, work through block size against the actual length distribution of the adapters going into production, in addition to the base model's typical output. A size tuned for one adapter's tendencies can waste real memory against another's, and if the adapter population spans short-form and long-form tasks, that mismatch should be measured directly rather than assumed away.
Size the GPU-side adapter cache with the KV pool's footprint already accounted for, rather than leaving it at a default that assumes the pool isn't competing for the same VRAM. Check that eviction priority tiers reflect actual adapter traffic patterns too. High-frequency adapters deserve protection from eviction that low-frequency, rarely-called adapters don't need.
Turn on KV cache quantization where the hardware supports it: FP8 on H100 and A100 hardware, NVFP4 on Blackwell if the deployment has it. That's memory recovered from a different layer of the stack, which eases pressure on every block-size and reuse-policy tradeoff made upstream. And if the deployment plans to offload KV blocks to a secondary tier, weigh the storage bandwidth underneath the serving stack against actual request timeout budgets before assuming offloading is a net win. A system with paging and reuse tuned well can still lose the SLO if the storage beneath it can't keep pace with how fast the workload needs blocks reloaded.
None of these settings work in isolation. Block size shapes fragmentation, fragmentation shapes adapter coexistence, and adapter coexistence shapes how urgently quantization and offloading start to matter. Multi-LoRA paged KV cache configuration is a system of interlocking constraints, and treating any one setting as independent of the rest is how deployments end up fast in isolated testing and starved for memory the moment real multi-adapter traffic hits.


