KV Cache Memory Scaling From 32K to 1M Token Contexts
Tiered memory strategies keep long-context LLM inference tractable as token windows scale.
NVMe offloading overcomes GPU memory limits for serving multiple long-context requests concurrently.
Tiered memory strategies keep long-context LLM inference tractable as token windows scale.
Recent systems show that pipeline bubbles stem from two separate problems requiring different fixes.
Quantization and pruning trade memory for speed in different ways depending on your workload.
Mixtral's sparse activation saves compute but demands full expert weights in VRAM.
Optimizing for throughput often means sacrificing the latency users actually feel.
Storage I/O, not GPU compute, limits inference throughput.
Understanding how prefill and decode phases behave separately under server load.
Naive peak-memory tests miss the churn that actually hurts latency.
How to test NVMe drives against real KV cache offload I/O.
Official MLPerf scores reflect vendor co-design, and reproducers face a real gap.
A practical guide to running MLPerf Storage v3.0 for inference.