TensorRT-LLM Paged KV Cache Configuration for Multi-LoRA Serving
How to allocate GPU memory efficiently when serving multiple LoRA adapters alongside KV cache.
Priya Subramanian
Speculative decoding speeds up single requests but can cut throughput under heavy load.
How to allocate GPU memory efficiently when serving multiple LoRA adapters alongside KV cache.
RadixAttention outpaces PagedAttention when workloads mix prompt lengths and reuse shared text.
Chunked prefill moves prefill and decode from competing priorities to cooperative scheduling phases.
Quantizing the KV cache cuts memory use sharply but slows token generation at long context lengths.
Continuous batching is the real win—the rest is architecture and memory management choices.
Precision format choice, not GPU count, determines long-context concurrency.