KV Cache Memory Pooling and Allocation Strategies for High-Throughput LLM Serving

Pooling and quantization strategies reclaim wasted GPU memory for higher concurrent LLM throughput.

· 8 min read
Cover illustration for “KV Cache Memory Pooling and Allocation Strategies for High-Throughput LLM Serving”
Long-Context Serving · October 25, 2026 · 8 min read · 1,898 words

KV cache memory pooling and allocation strategies turn wasted GPU memory into concurrent request capacity. Efficient production LLM serving starts with on-demand paged allocation, then combines workload-aware eviction and KV quantization with offload where capacity requires it. Naive allocation wastes a substantial share of GPU memory through over-reservation and fragmentation, but reclaiming that space delivers throughput only when the storage and transfer paths can feed decode efficiently.

Why naive KV cache allocation fails before a single concurrent request is added

A request that generates only a few hundred tokens can still hold a buffer sized for its maximum possible context length. Static allocation sets aside that buffer when the request arrives, before it is known how much the request will need. Most of the reservation may remain unused. Yet other requests cannot use those reserved bytes, so maximum context limits rather than active token counts determine how much GPU memory the service commits.

Fragmentation wastes more space when requests of different lengths leave gaps between reserved regions that other requests cannot fill. Raw free VRAM therefore overstates usable capacity. Over-reservation traps memory inside allocations, while fragmentation strands memory between them. As requests accumulate, this wasted space limits concurrent request capacity before another serving bottleneck needs to be reached.

How PagedAttention restructures KV memory as a shared virtual pool

PagedAttention replaces contiguous per-request buffers with fixed-size blocks drawn from a shared pool. It borrows the operating-system virtual memory model, allocating blocks as tokens are generated rather than reserving a maximum-length buffer upfront. Physical blocks do not have to sit next to one another. A block table maps each request’s logical KV sequence to its physical locations, making that placement transparent to attention computation.

Blocks return to the pool immediately when a request completes, so another request can use them without inheriting a large contiguous reservation. The supplied large-model comparison reports doubled concurrent users and nearly halved KV VRAM consumption at the same serving quality after replacing naive pre-allocation with PagedAttention. In vLLM, paged allocation provides the foundation on which cache reuse and eviction policies operate.

Non-contiguous blocks produce fragmented, blockwise reads during autoregressive decoding, reducing compute and memory bandwidth utilization. The cited XFormers profiling shows only partial utilization under this access pattern, although the brief supplies no utilization percentage. The capacity gain has a bandwidth cost. That cost creates pressure to reduce bytes per token and to design any larger memory hierarchy around the speed at which decode consumes KV blocks.

What eviction policy design adds once fragmentation is solved

A shared pool still needs a rule for which reusable blocks to keep when it fills. Least-recently-used eviction, the default in vLLM, favors the blocks that were used most recently. That fits conversation better than every serving workload. In multi-tenant deployments, unrelated requests can flush one another’s cached prefixes, including document prefixes that many users repeatedly query. Losing those hotspot blocks discards opportunities to avoid substantial prefill computation.

“Recency/Frequency Adaptive KV Caching for Large Language Model Serving,” presented at the ICML 2026 AdaptFM Workshop, addresses this mixed access pattern with Adaptive Replacement Cache. ARC divides space between recently accessed and frequently accessed blocks. Ghost lists track misses and change how much space each of those groups gets as demand changes. It is built as an open-source extension to vLLM.

The paper reports up to 10.8% higher cache hit rate on synthetic document question-answering workloads, with a meaningful reduction in time-to-first-token compared with naive vLLM. Real-world conversational workloads show smaller improvements in both measures because LRU already fits their recency pattern reasonably well. Eviction policy preserves reusable computation, but its benefit follows the request mix as well as the pool’s physical layout.

How quantizing KV tensors trades precision for capacity within the same pool

Quantization addresses a different question from eviction, namely how many bytes each retained token consumes. Moving from BF16 to FP8 reduces bytes per element from 2 to 1, while FP4 reduces them to 0.5. Those formats halve or quarter tensor storage per token without changing the allocation logic or number of blocks. The pool remains structurally the same. More requests fit because each cached token takes up less space.

On NVIDIA B200 GPUs with FlashInfer, FP8 substantially reduces the rate at which decode inter-token latency grows relative to BF16. The reported loss in accuracy is at most a fraction of a benchmark point, with very high recovery even on reasoning benchmarks that are hard. That is a favorable production trade. FP8 and INT8 remain different from the original BF16 tensors even when described as nearly lossless.

Further reductions encounter a practical precision limit. Pushing KV quantization to 2-bit causes major accuracy degradation, so reduced storage cannot be treated as an unqualified gain. Lossless compression offers a separate route when exact tensor preservation is required. BF16 KV activations contain redundancy concentrated in the exponent field, typically only 3–4 bits of entropy. Exploiting that redundancy becomes especially useful when the cache must move between workers or memory tiers.

Why disaggregating prefill and decode forces KV transfer onto the critical path

Prefill and decode place fundamentally different demands on hardware. Prefill parallelizes well and is compute-bound, while sequential decode depends on the bandwidth available to read KV blocks. Keeping both on one GPU forces a compromise. Separating them allows hardware specialization, but the prefill worker must then deliver its KV tensors to the decode worker before generation begins. At long context lengths, transfer latency can dominate time-to-first-token.

Mooncake, published in ACM Transactions on Storage in 2026, makes KV movement an explicit infrastructure function through a Transfer Engine supporting RDMA and TCP. NVIDIA Dynamo supports disaggregated prefill and decode with GPU KV transfer through the NIXL library, using RDMA, NVMe-oF, and S3. CrossPool, implemented on SGLang v0.5.10 in 2026, extends the separation to multi-LLM, multi-tenant serving by dividing consolidated FFN weights from multiple cold MoE models and dynamic KV state into distinct pools. Its KV pool uses virtualized paging, with hidden-state transport between pools and persistent GPU-side dispatch for layer-wise execution.

These designs make the data path consequential. The early CPU-driven RDMA path traverses six hops, staging KV data through pinned host DRAM via cudaMemcpy before posting an RDMA verb. That introduces queuing variability and a mandatory host-DRAM copy whose cost grows with context length. Staging, copies, and dispatch form part of the latency paid before decode can start.

What CXL and NVMe tiers add to the memory hierarchy when GPU capacity runs out

When paging and FP8 cannot fit active KV state in GPU memory, offload extends the pool into CPU memory, NVMe, or CXL-attached capacity. LayerKV offloads selected layers’ KV tensors during prefill, retaining the minimum layers on GPU needed to overlap communication with computation. LMCache extends prefix caching into host memory and persistent storage. CPU memory is the first extension tier. How much a deeper tier helps depends on how it is accessed as well as how much it can hold.

LM-CXD, from Sogang University and Samsung in September 2026, identifies three costs in the block I/O path, including CPU cache contention, mandatory host-DRAM staging, and raw NAND latency. Its ramdisk comparison isolates the interface penalty. Even when DRAM replaces the storage medium, using the block interface is 1.8× slower than direct DRAM. GPUDirect Storage bypasses CPU and host DRAM, but data still enters a VRAM staging buffer and requires a serialized reshape into the KV layout. Those steps are still needed when the media is faster.

CXL.mem exposes NAND-backed capacity as byte-addressable memory in the host address space, eliminating block I/O and host-DRAM staging. A stock CXL-SSD nevertheless remains roughly 3× slower than local DRAM and no faster than NVMe for KV prefix caching. LM-CXD makes KV chunks device-visible units, exposes NAND-to-DRAM progress, and uses device DRAM as a GPU-accessible buffer. Coordinated scheduling and prefetching produce up to 2.6× TTFT reduction with compute-asynchronous prefetching and up to 4.03× with layerwise prefetching, reaching TTFT within 1.5× of local DRAM on average across five LLM models.

Shared CXL memory also offers a different transfer topology. TraCT, a rack-scale system from 2025, uses it for KV transfer and a rack-wide prefix-aware cache, removing the extra copies and network hops found in RDMA-based designs. It reports up to 1.6× higher peak throughput and up to 9.8× lower average TTFT. Deployment scope limits the comparison, however. CXL shared memory currently operates at rack scale, while RDMA spans datacenters, so neither is universally superior.

CXL-SpecKV, presented at FPGA '26 in February 2026, expands GPU memory capacity through CXL disaggregation and uses accurate speculative prefetching to reduce access latency. INF2 instead moves attention computation to Computational Storage Devices, reducing the need to move KV data to the GPU. It reports throughput gains and reduced KV I/O overhead.

How lossless compression on KV tensors accelerates transfer without touching model accuracy

BF16 and FP8 KV tensors have a structure that purpose-built lossless codecs can exploit. Their exponent fields typically contain only 3–4 bits of entropy, while their mantissas are nearly incompressible. That distinction directs compression toward actual redundancy. General-purpose CPU codecs aimed at offline weight compression are too slow for live KV production during prefill. Lossy quantization avoids that problem differently, but cannot provide bitwise preservation and encounters a practical accuracy floor around 4-bit.

SplitZip, introduced by Guo and Joshi in 2026, is a GPU-native lossless compressor built for KV transfer in disaggregated serving. On real BF16 activation tensors, it achieves 613.3 GB/s compression throughput and 2181.8 GB/s decompression throughput. The faster decompression rate is especially important operationally. Decompression lies on the hot path feeding decode, so restoring tensors must not erase the transfer savings. The codec reports up to 1.32× faster BF16 KV transfer and up to 1.30× faster TTFT.

SplitZip also provides up to 1.14× compression over native FP8 E5M2 representation. It preserves tensors bitwise and integrates into existing serving frameworks without changing model execution. Lossless compression therefore complements quantization. In a transfer-bound system, sending fewer bytes increases effective link capacity without adding physical interconnect bandwidth. The resulting concurrency benefit comes from relieving data movement.

Choosing the right combination of strategies for the workload you actually have

For practitioners, the selection starts with access pattern and ownership. Conversational workloads can retain LRU, while mixed RAG and document QA traffic with hotspot prefixes benefits from ARC-based adaptive eviction. Single-model, high-concurrency serving favors deep paging within a unified pool. Multi-model MoE environments favor CrossPool’s separated weights and KV pools.

Context length, concurrency, and hardware then determine the memory tiers. GPU-only serving with FP8 is feasible for moderate contexts at moderate concurrency, while CPU offload through LayerKV or Oneiros suits bursty decoding pressure when CPU–GPU bandwidth is high. Sustained load is a less favorable fit. Million-token prefix caches or very large concurrent reuse pools require CXL or NVMe capacity, with LM-CXD-style semantic integration needed to approach DRAM latency competitively. INF2’s attention-near-storage approach is appropriate when KV I/O dominates and moving computation is cheaper than moving data.

Finally, co-located prefill and decode remain simpler to operate until their resource profiles diverge enough to justify specialized workers. At that point, Mooncake or Dynamo with NIXL provides the disaggregated transfer layer, and SplitZip becomes a useful lever when transfer constrains service. Capacity alone is not the final criterion. The storage layer must be engineered around KV chunk access, placement, prefetching, and transfer semantics so that reclaimed memory becomes usable throughput with timely access to data in the larger pool.

Sources

  1. Recency/Frequency Adaptive KV Caching for Large Language Model Serving
  2. Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
  3. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  4. Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  5. CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

More in Long-Context Serving