Tokens per Second vs Latency Tradeoffs in Throughput-Optimized Serving Configs
Optimizing for throughput often means sacrificing the latency users actually feel.
Mei-Ling Tsao
Section
7 stories in Benchmarking.
Optimizing for throughput often means sacrificing the latency users actually feel.
Storage I/O, not GPU compute, limits inference throughput.
Understanding how prefill and decode phases behave separately under server load.
Naive peak-memory tests miss the churn that actually hurts latency.
How to test NVMe drives against real KV cache offload I/O.
Official MLPerf scores reflect vendor co-design, and reproducers face a real gap.
A practical guide to running MLPerf Storage v3.0 for inference.