Multi-Round Conversation State Management at Scale for Long Contexts
Managing KV cache across memory tiers prevents latency from becoming the bottleneck.

Serving LLMs with long context windows at scale requires a memory hierarchy. Multi-round conversation state management is a KV cache lifecycle problem that spans retention, compression, offload, reload, and reuse. The infrastructure must keep conversation state that later turns need through those stages without letting data movement take up most of the latency.
Why multi-turn conversations create a structurally different KV cache problem than single-turn inference
Each conversation turn is a separate request whose prompt has the earlier turns and the new user input. The model has no intrinsic memory mechanism that carries the conversation forward outside that supplied context. As a result, most of a new request repeats material already processed during the previous turn. The requests are separate, but their state overlaps.
At every turn boundary, the system must decide which existing KV blocks to retain, offload, or evict before the conversation resumes. A framework that handles each request from scratch repeats work on a prefix that grows longer but stays mostly the same. Conversely, a framework that retains everything in GPU memory accumulates state as the session grows.
Prefix caching addresses the redundant computation, but the cached state still consumes memory. The shared history has value because later requests need it again. Preserving that value requires managing its residency between turns, rather than treating the end of a response as the end of the cache’s useful life.
How the KV cache grows turn by turn: prefill, decode, and the compounding footprint
Within a turn, prefill processes the prompt in parallel and populates the KV cache for every layer. Without prefix reuse, that prompt includes the full conversation history. With reusable history already cached, prefill needs to process only the new input. This phase is compute-intensive and produces the state that decode consumes.
During decode, each generated token attends over the cached context and adds its own key-value state. TTKV identifies KV cache access, rather than computation, as the primary determinant of per-token latency as context grows. The cache thus expands during both the arrival of new input and the generation of a response. Those generated tokens are part of the history sent with the next turn.
The uncompressed memory requirement follows the formula KV cache memory = 2 × layers × KV heads × head dimension × sequence length × bytes per element. Sequence length grows as the conversation goes on, while precision sets the bytes per element. Before eviction or compression, the footprint grows monotonically through the session. Request concurrency compounds the total capacity required to hold that state.
What infrastructure is needed when long-context KV state exceeds HBM.
At long context lengths and realistic batch sizes, exceeding HBM capacity is a predictable result of the memory formula. Moving the excess into CPU DRAM solves the immediate capacity shortage but changes the decode path. TTKV describes PCIe bandwidth as typically an order of magnitude narrower than HBM bandwidth. If attention has to keep fetching spilled state through that connection, the time spent on transfer becomes the bottleneck.
The available hierarchy extends through GPU HBM, CPU DRAM, CXL 3.0 pools, NVMe with GPUDirect Storage, RDMA fabric, and a parallel filesystem. The tier gap is substantial, with GPU HBM at 3.35 TB/s and sub-microsecond latency, compared with NVMe with GPUDirect Storage at 12 GB/s and ~10 µs. Lower tiers supply capacity, but they cannot be treated as equivalent extensions of HBM. Their usefulness depends on how much state crosses the boundary and when.
Open Lake’s infrastructure addresses this GPU-memory and NVMe-tier problem by treating KV cache residency as part of decode performance. This holds across the hierarchy, however it is built. Once active state outgrows HBM, it needs space in lower tiers. The system must use that space without putting costly reloads on every latency-sensitive step.
Hot, warm, and cold: a principled tiering policy for conversation-turn KV blocks
Within a session, the most recent turn’s output is hot, prior turns’ outputs are warm, and a shared system prompt or long document prefix that has not been accessed recently is cold. These labels set priorities for access and placement, while semantic importance can change over time. An old fact can still become relevant to a later question.
TTKV formalizes temporal tiering through heterogeneous capacity and precision. Recent KV states occupy faster, higher-precision tiers, while older states move to slower, lower-precision tiers. Its block-wise streaming attention overlaps computation with tier transfers, so placement policy and execution work together. ICMSP provides another route through NVMe tiering when HBM capacity is exhausted.
Putting this policy into production requires tracking eviction decisions, measuring tier-crossing latency, and rebalancing state as conversations lengthen. For infrastructure such as Open Lake’s GPU-memory and NVMe tiers, these are requirements for coordination between the inference engine and storage layer. The system must also account for the cost of making that block available when attention or the next turn needs it.
Eviction is not deletion: how learned and heuristic policies differ in what they preserve
Moving a KV block out of HBM preserves the possibility of reloading it. Discarding its contents does not. The accuracy tradeoff begins when a memory policy removes information that future decoding may need. No placement policy can recover state that has been permanently lost.
H2O ranks tokens by the attention scores they have built up and evicts the lowest-scored entries when its cache budget is full. SnapKV observes attention weights during prefill to identify important tokens and construct a compressed context snapshot. Both depend on proxy scores for future usefulness. IndexMem identifies limitations in those proxies, including local-window bias toward nearby tokens and static aggregation that may not adapt to the inference-time query distribution.
IndexMem, presented at ICML 2026 by HKUST, addresses selection and preservation together. Its learnable indexer predicts token importance, while a lightweight latent memory module compresses evicted tokens into a fixed-size, online-updated state. Those tokens can still add residual readouts to attention, though their original KV entries are no longer kept. MomentKV addresses a complementary directional gap by accounting for directional structure in attention.
For multi-turn serving, the consequence is concrete. A fact stated in turn 1 can matter to a query in turn 10. Recency alone cannot distinguish obsolete history from a distant dependency.
Compression before offload: reducing what has to move across the memory hierarchy
Agrawal & Mayer’s PVLDB 2026 benchmark found that compression ratio is a poor predictor of end-to-end performance. The benchmark evaluated KIVI, TurboQuant, SnapKV, and CaM on Llama-3.1-Instruct and Mistral-Instruct variants. Its tasks included multi-document QA, single-document QA, few-shot learning, and summarization. Compression must be assessed against the workload, transfer path, and resulting cache size.
KIVI uses per-channel quantization for keys and per-token quantization for values, reaching 2-bit compression. That distinction accounts for channel-level outliers in keys and token-level outliers in values. CommitKV takes a different approach to the decision itself. Designed for multi-turn agents, it ties compression to the KV lifecycle stage.
Smaller blocks reduce the volume that must cross tier boundaries, but that benefit depends on how the compressed state is used. TTKV identifies host-to-GPU traffic as a reason compression gains can disappear when caches are offloaded without memory-hierarchy integration. A smaller cache can still incur costly repeated reads. Compression therefore belongs inside the offload design, where its effect on capacity, transfer volume, and workload accuracy can be considered together.
Cross-turn prefix reuse: why the shared conversation history is a caching asset
The KV blocks computed for conversation history in one turn are the blocks needed for the shared prefix of the next. Reusing those blocks eliminates prefill computation for that prefix. It does not eliminate the cost of retrieving them if they have left HBM.
SwiftCache’s June 2026 work makes that connection concrete. Models with low KV cache demand donate idle GPU memory to store prefix caches for high-demand models, with sharing over NVLink. SwiftCache reports substantially lower P99 TTFT and a longer maximum context length compared with vLLM and SGLang. The mechanism is heterogeneous memory sharing, with each model retaining its own computed KV values.
A warm block representing stable conversation history has a different future value from a one-off generated passage. That distinction also matters for Open Lake’s storage infrastructure, where preserving and relocating reusable blocks bears directly on reload cost. Prefix caching loses part of its latency advantage when repeated spilling makes those blocks expensive to recover.
Disaggregated prefill-decode serving and why multi-turn exposes its blind spot
Prefill-decode disaggregation places compute-intensive prefill and memory-bandwidth-intensive decode on separate GPU pools. This separation makes better use of resources by giving each phase more of what it needs most. However, single-turn designs do not fully account for the repeated alternation that conversations introduce. The session goes back to prefill each time a new user message arrives.
In multi-turn inference, incremental prefill for that message is followed by another decode phase. The KV state must move between prefill and decode workers at turn boundaries, making transfer a recurring part of the session. Phase specialization alone does not resolve the state handoff.
He et al.’s AMPD, presented at ICML 2026, addresses this interleaved pattern with a multi-round-aware disaggregated framework built atop NVIDIA Dynamo. It reports substantially improved SLO attainment compared with Dynamo and vLLM baselines. Its experiments used 4 servers, each with 8 NVIDIA H20 (96 GB) GPUs, with NVLink inside servers and InfiniBand between them. The result makes scheduling relevant, but scheduling still depends on a transport layer that can deliver the required KV blocks within the latency budget.
Moving KV blocks between GPUs at speed: NIXL and RDMA as the transport layer
Once prefill and decode occupy different workers, KV placement extends across GPU boundaries. Eviction and compression determine what must move and how large it is. The transport layer determines how quickly the receiving worker can use it. These are separate jobs, but their costs add up on the same request path.
NIXL uses one-sided RDMA for the transfer mechanism described here. Each decode worker registers a region of GPU memory into which prefill workers can write directly. Workers exchange memory-registration metadata, called NIXL descriptors, at startup. The control messages for each request then carry a small buffer identifier in place of a full address structure.
This makes buffer registration, CPU staging, and data movement part of the infrastructure that the serving design must cover. The RDMA and NVMe-tier engineering involved also bears on Open Lake’s work across storage and interconnects. In each case, transfer overhead can decide whether disaggregated multi-turn serving is worth its cost.
vLLM’s push-based NIXL connector, published October 14, 2026, provides a concrete implementation. It moves KV blocks over RDMA-capable interconnects including AWS Elastic Fabric Adapter (EFA). On supported instance types, AWS EFA supports GPUDirect RDMA, allowing NIXL to transfer blocks directly between GPU memory without staging them through host memory.
Choosing a management strategy: matching technique to workload and hardware
The first decision axis is context length relative to HBM capacity at the intended concurrency. For short-to-medium contexts under ~32K tokens that fit in HBM, focus on GQA/MQA architecture selection and quantization, including KIVI4 for stability, to maximize batch size. Once long contexts exhaust HBM, ICMSP-style NVMe tiering or TTKV-style temporal tiering becomes necessary. GPUDirect Storage takes CPU staging overhead out of that storage path.
The second axis is prefix reuse rate. Multi-turn agents, shared system prompts, and RAG favor prefix caching, while mixed-model deployments can consider SwiftCache’s cross-model NVLink sharing. With low prefix reuse, eviction quality carries more weight. IndexMem is relevant where heuristic policies mishandle long-range token dependencies.
The third axis is serving architecture. Monolithic serving can use TTKV or ICMSP to manage the hierarchy within a node. Disaggregated serving needs multi-round-aware scheduling, as in AMPD, and a transfer layer such as NIXL. The backend choice, including InfiniBand, RoCE, or NVMe-oF, sets the floor on transfer latency.
GroupKV illustrates the gains available from coordinated techniques in long-context diffusion LLM inference. Its hierarchical sparse selection, predictive prefetching, and streaming prefill extend maximum serviceable context length by up to 48× under constrained GPU memory and improve end-to-end performance by up to 3.73× in offload-based settings. Those results are specific to its setting, not a universal serving guarantee. The general principle is to manage prefill, decode, eviction, compression, offload, transfer, and reuse as one lifecycle. Optimizing one stage in isolation leaves the next stage to impose the limit.
Sources
- [2609.17573] GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference
- TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
- Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
- IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
- MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
- SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing
- [2602.14516] Efficient Multi-round LLM Inference over Disaggregated Serving


