Storage-First Architecture Decisions for LLM Serving Systems Targeting Million-Token Contexts
Organizing KV cache across memory tiers unlocks efficient million-token LLM serving.

Serving LLMs with long context windows at scale requires a storage-first architecture that places KV cache across GPU HBM, CPU DRAM, local NVMe, and networked storage according to workload needs. At million-token scale, request serviceability, speed, and cost depend on cache placement and movement. Compression, reclaimable memory, and interconnect bandwidth must therefore enter the design before hardware sizing.
Why million-token context turns KV cache into a storage architecture problem
KV cache grows linearly with context length and batch size. At extreme context lengths, naive multi-head attention with FP16 KV cache can need more memory for cached context than for model parameters. The constraint is physical. When weights and context cannot fit together in GPU HBM, the cache must take up less space or sit partly elsewhere for the request to be served.
Recomputing the prefix on every request avoids retaining that state, but increases computation to reduce memory use. At million-token scale, recomputation can consume the GPU’s entire compute budget and make interactive latency impossible. HBM capacity grows more slowly than the demand from longer contexts, so a longer context window calls for choices about storage even when the model weights already fit. The architecture must account for both the model itself and the state needed to serve it.
How the prefill/decode asymmetry shapes which storage tier does what work
Prefill processes the prompt in parallel through large matrix multiplications. Tensor cores are saturated while memory bandwidth is underutilized, making this phase compute-bound. Decode has the opposite resource profile. It generates one token at a time, rereading model weights and the full KV cache at every step, so memory bandwidth is saturated while compute is underutilized.
This difference gives the cache two distinct jobs. During prefill, it must accept writes quickly. During decode, it must support repeated reads with low latency. Co-locating both phases on the same hardware creates resource contention and latency interference, while separating them allows compute-dense accelerators to handle prefill and memory-bandwidth-optimized hardware to handle decode.
That separation introduces a further storage obligation. The hardware running decode must be able to read the cache made during prefill, and moving it takes time. Hardware specialization is therefore useful only in relation to the cost of delivering the state between phases. The resource profiles explain why disaggregation can help, but do not establish that it will help every workload.
What infrastructure is needed to place KV cache across four storage tiers?
GPU HBM offers the lowest latency and highest bandwidth, but the smallest capacity. An H100 has 80 GB, which is quickly exhausted at million-token scale. HBM is appropriate for active KV blocks in the decode window because those blocks face repeated reads. CPU DRAM can hold most of a very large cache, especially for long-context requests at low concurrency. The split between them depends on both capacity and the latency added by moving blocks back toward decode.
Local NVMe serves a different part of the working set. With capacities of 10–30 TB at high drive writes per day targets, it is a good fit for cold sessions and long-tail cache. Under extreme memory pressure, NVMe offloading can recover throughput by retaining state that would otherwise require recomputation. PCIe 5.0 NVMe narrows the latency gap further, but does not erase the distinction between storage and HBM. The relevant comparison is the reload penalty against recomputation or the cost of more higher-tier capacity.
Networked storage, including NVMe-oF and object storage, extends capacity beyond a node and makes KV cache sharable across nodes. That enables cluster-wide prefix reuse. Its latency depends on the interconnect, so remote capacity cannot be evaluated separately from the path used to reach it. A tiered working set promotes hot blocks toward HBM and evicts cold blocks toward NVMe or networked storage. At each boundary, access time must be weighed against capacity and cost, and the preferred media can change with that trade-off.
How compression and quantization change the economics of every storage tier
FP8 KV quantization halves memory relative to FP16/BF16, changing which tier can hold a request before any hardware changes. Its accuracy regression is minimal for many production workloads, whereas INT8 incurs meaningfully greater losses on long-context retrieval benchmarks such as multi-needle tasks. Precision is therefore a placement decision too. FP8 is a workload-dependent practical default for moderate accuracy requirements.
One model’s attention mechanism compresses KV cache through the attention architecture. Combining MLA at FP8 instead of naive MHA at FP16 produces an order-of-magnitude difference in KV memory at million-token scale. Prefix caching offers another source of savings by reusing KV blocks for shared system prompts or document context. How much it saves depends on the cache hit rate, so the same model formats can need different amounts of storage when request patterns differ.
The ENEC analysis shows that the BF16 exponent field contains far less entropy than its bit allocation, creating an opportunity to compress weights without numerical loss. ZipServ demonstrates bit-exact numerical precision and measurable end-to-end speedup over vLLM. KARA instead compresses KV cache during decode, using bidirectional attention to select informative pairs within a recent window. It preserves nearly all full-KV accuracy while retaining a small fraction of the cache and improving throughput under memory-constrained concurrent serving.
These methods need not be mutually exclusive. Together, they can reduce effective KV memory by an order of magnitude, potentially moving a workload from networked storage into local NVMe through FP8 and prefix reuse. But lossless weight compression, numerical KV quantization, and selective KV retention have different accuracy implications. Their storage savings cannot be treated as interchangeable.
Why block-level KV management leaves memory stranded and what token-level virtualization does about it
PagedAttention allocates KV cache in fixed-size blocks, reducing allocator-level fragmentation. Token-level eviction exposes a different kind of waste. When a block has both retained and evicted tokens, the retained tokens keep it from being freed. The resulting holes remain allocated but unavailable for other requests. Compression may shrink the live data without freeing the physical memory that held it.
The vToken paper’s preliminary evaluation finds that, across eviction policies, most allocated blocks run at no more than 50% utilization. This intra-block fragmentation reduces effective concurrency and throughput below what nominal hardware capacity suggests. Smaller blocks add metadata overhead and make address-mapping more complex, moving the cost elsewhere.
vToken introduces token-level virtualization to separate logical token liveness from physical block placement. Token-table indirection maintains a stable logical view while asynchronous repacking gathers live tokens and releases physical blocks. The design preserves PagedAttention kernels and CUDA Graph compatibility, allowing integration into existing vLLM deployments without a runtime rewrite. It keeps fewer blocks per request and raises throughput within service-level constraints while cutting the code needed to integrate each policy from 500+ lines to under 50.
For tier sizing, the consequence is direct. With token-level eviction, only part of the allocated capacity may be usable. Intra-block waste must enter capacity estimates, and a virtualization layer makes reclamation assumptions more reliable before DRAM or NVMe provisioning is set.
What disaggregated architectures actually require from the storage layer
Disaggregated prefill and decode place cache movement on the request’s critical path. Each request requires transferring the full KV cache from the prefill node to the decode node. Local capacity is no longer sufficient by itself. The storage layer must make that state available across engines and nodes without allowing transfer time to overwhelm the benefit of specialized hardware.
Mooncake implements a global scheduler named Conductor that dispatches requests based on KVCache distribution. Its ACM TOS publication describes a disaggregated KVCache built from the cluster’s CPU, DRAM, SSD, and RDMA resources. Placement becomes an input to scheduling. LMCache provides a KV cache layer for vLLM and SGLang with NVMe offloading for prefix reuse and cross-engine transfers for prefill–decode disaggregation. Cache hits can skip prefill entirely.
The interconnect determines how much of that reuse is practical. RDMA provides CPU-bypass transfer latencies of 10–20 microseconds, while NVMe-oF achieves 20–30 microsecond latencies by avoiding SCSI emulation layers. These figures do not describe full million-token cache transfers. Even with RDMA, per-NIC bandwidth limits total throughput, making concurrent long-context sessions a provisioning constraint.
MLPerf Storage v3.0, dated September 1, 2026, extends storage benchmarking to inference workloads. One submitted configuration moves file payload traffic over two independent 400 Gb/s NDR InfiniBand/RDMA links on the client and each storage node. That configuration provides a concrete benchmark baseline. The architectural requirement is to treat transport bandwidth as part of serving capacity.
The disaggregation paradox: when long context breaks the specialization argument
The usual specialization argument assigns H100s to prefill because prefill is compute-bound, with cheaper hardware handling decode. Aggressive prefix caching changes the dominant operation. At million-token context, a cache hit may make the phase almost entirely I/O, with cached KV reads replacing most matrix multiplication.
In that case, compute-dense hardware loses the advantage it was chosen for but still costs more. The premise of the split has changed. A system that benefits from disaggregation on uncached prompts may have a different economic profile as reuse increases. The crossover depends on the model, context-length distribution, and cache hit rate, and there is no universal threshold at which specialization stops paying.
The MLCommons working group says that object-based storage may work as an alternative to filesystems, and may even be preferred, as context reaches into the trillions. POSIX still has lower latency for hot-path KV reads, so the case for capacity at extreme scale leaves open which serving path to use now. These are distinct pressures on the design.
The practitioner implication is to revisit disaggregation as context distributions and hit rates shift. A fixed architecture can outlive the workload assumptions that justified it. The value of phase separation depends on the current balance of computation, cache loading, and transfer.
A decision framework for placing KV cache across tiers given workload characteristics
Where to place the cache should depend first on context-length distribution, request concurrency, cache hit rate, and accuracy requirements. No single architecture is optimal across all four. For long-context requests at low concurrency, HBM serves active decode blocks, CPU DRAM holds the bulk of a very large cache, and local NVMe retains cold sessions. Recomputing million-token prefixes is not a viable substitute for this hierarchy when interactive latency is required. Increasing concurrency then makes both aggregate capacity and cache-transfer throughput more restrictive.
Multi-turn agentic workloads place greater value on persistence between turns. NVMe or networked storage can retain session state, while LMCache-style disk-backend prefix reuse avoids inter-turn recomputation. Edge deployments face a different constraint when neither HBM nor limited DRAM can absorb the cache. Dual-Blade’s dual-path NVMe-direct architecture addresses that case, with adaptive pipeline parallelism managing I/O efficiency. These setups fit different workloads, so each deployment can use the one it needs without passing through them all.
Accuracy requirements limit the choice of compression before any capacity is bought. For accuracy-critical reasoning, ZipServ-style lossless BF16 compression reduces the weight footprint while preserving bit-exact precision. INT8 KV quantization is off the table for that scenario, with FP8 KV as the floor and acceptable accuracy still uncertain. KVQuant combines per-channel quantization, pre-RoPE processing, non-uniform quantization, and dense-and-sparse quantization with outlier preservation for extreme context lengths. At the longest context lengths, schemes like these are needed.
Deployments using token-level eviction should adopt a virtualization layer before sizing DRAM or NVMe to account for partially utilized blocks. The choice of tier must then account for the compressed working set, how it is reused, and the bandwidth needed to move it. The governing decision is not which storage tier is best, but which state must be close to compute, for how long, and at what transfer cost.
Sources
- vToken: Token-Level Virtualization for Reclaimable KV Caches
- KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
- KV CACHE OPTIMIZATION STRATEGIES FOR SCALABLE AND EFFICIENT LLM INFERENCE
- Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs


