Reproducing MLPerf Inference Benchmarks on Open-Source Serving Stacks
Official MLPerf scores reflect vendor co-design, and reproducers face a real gap.

Reproducing MLPerf Inference Benchmarks on Open-Source Serving Stacks.
MLPerf Inference and Open-Source Reproducers as a Natural Stress Test
MLPerf Inference measures how fast a complete system, the hardware, the runtime, and the serving stack layered on top of it, can run a fixed, pre-trained model while staying inside strict latency and accuracy limits. It is not a raw speed contest. The Closed division locks the model and the preprocessing pipeline in place so results are actually comparable side by side, while the Open division lets submitters swap in different models, at the cost of that comparability.
The metrics that matter are time to first token (TTFT) and time per output token (TPOT), both measured under service-level latency budgets, not throughput alone. Recent rounds added interactive, low-latency scenarios specifically to capture what a real user waiting on a chat response actually feels. MLCommons has kept up a steady release cadence: four rounds since early 2025, v5.0 in April 2025, v5.1 in September 2025, v6.0 in April 2026, and v6.1 on September 16, 2026 HPCwire MLCommons Releases New MLPerf Inference v5.1 Benchmark Results. The v5.1 round alone crossed 90,000 cumulative submitted results and pulled in a record 27 submitting organizations HPCwire MLCommons Releases New MLPerf Inference v5.1 Benchmark Results.
That scale and rigor is why open-source reproduction is such a demanding exercise. The official submissions come from hardware vendors doing full-stack co-design, tuning silicon, interconnect, and software together for one specific benchmark run. Open Lake, a storage infrastructure company whose KV cache and NVMe systems are built specifically around LLM inference throughput, has published benchmark work grounded in exactly this kind of full-stack analysis. Someone trying to reproduce those numbers on vLLM, SGLang, or TensorRT-LLM is working against the same scenario constraints with a fraction of the control over any single layer. Measurement, client design, and storage behavior are where that control gap appears, and understanding it there is what separates a reproduction that means something from a number that just happens to land in the right range.
The workloads a reproducer must now contend with across recent benchmark rounds
The interactive scenario itself got wider coverage in v5.1, now applied across multiple models with tighter latency limits than the standard server scenario MLCommons Releases New MLPerf Inference v5.1 Benchmark Results. Llama 2 70B pulled 24 submitters that round, which makes it probably the single most useful workload for comparing systems apples to apples. v6.0 kept expanding: GPT-OSS 120B arrived for math, scientific reasoning, and coding tasks, DeepSeek-R1 got an interactive scenario that allows speculative decoding, and DLRMv3 introduced the first sequential recommendation test in the suite MLCommons Releases New MLPerf Inference v5.1 Benchmark Results. Five of the eleven datacenter tests in that round were either new or substantially updated.
v6.1 pushed further still, adding Edge Agentic Inference and an End-to-End RAG benchmark. The VLM per-accelerator server scenario result jumped 2.99x over its v6.0 number in just six months. Storage isn't background infrastructure anymore in this benchmark; it's a measured surface in its own right.
For anyone trying to reproduce results, this diversity forces a choice up front. Reasoning workloads, long-context interactive scenarios, and mixture-of-experts architectures each stress a serving stack differently, and picking a target scenario without understanding which bottleneck it's actually testing is the fastest way to a meaningless result. The v5.0 round added Llama 3.1 405B, Llama 2 70B Interactive (low-latency), RGAT, and Automotive PointPainting as new benchmark workloads MLCommons Releases New MLPerf Inference v5.0 Benchmark Results. The v5.1 round added DeepSeek-R1 (671B MoE, the first reasoning-model benchmark), Llama 3.1 8B, and Whisper Large V3, with the reasoning benchmark introducing longer output lengths than prior rounds because reasoning models produce extended chains of tokens before a final answer, yielding a qualitatively different throughput/latency profile from conventional generation.
Official Top Results and the Gap a Reproducer Faces
The numbers coming out of the top submissions show what full-stack optimization buys. Llama 3.1 8B offline crossed 18,000 tokens per second per GPU, and Whisper reached 5,667 tokens per second per GPU HPCwire MLCommons Releases New MLPerf Inference v6.0 Benchmark Results. Those aren't incremental gains from a software patch; they reflect a new hardware generation paired with serving software built specifically around it.
To meet the Llama 3.1 405B interactive benchmark's tighter latency constraints, NVIDIA employed disaggregated serving, separating context and generation phases across different GPUs using its Dynamo framework HPCwire MLCommons Releases New MLPerf Inference v5.0 Benchmark Results. That architectural choice alone delivered a 1.5x throughput-per-GPU gain over conventional serving on Blackwell silicon, and more than 5x over Hopper-based systems HPCwire MLCommons Releases New MLPerf Inference v5.0 Benchmark Results. In v6.1, the frontier moved again: Vera Rubin NVL72 delivered up to 3.7x the throughput of GB300 NVL72 in its debut submission, and GB300 NVL72 itself hit 99% scaling efficiency across a 288-GPU run spread over four racks NVIDIA: MLPerf AI Benchmarks HPCwire MLCommons Releases New MLPerf Inference v6.0 Benchmark Results. That round also included the largest system ever submitted to MLPerf Inference, 512 accelerators, plus the first geographically distributed submission, spanning the Pacific Ocean. Neither of those configurations gets reproduced on a single node running an open-source stack, and nobody should expect it to.
What these results actually reveal is that the official leaderboard is a record of co-design: silicon, memory, interconnect, and serving software tuned together for one benchmark run. A reproducer running a general-purpose open-source framework against the same workload is measuring a genuinely different system, not a smaller version of the same one, and the resulting gap between the reported number and a naive reproduction attempt can be substantial. The honest goal of reproduction was to confirm that a given stack meets the scenario's accuracy and latency bar on the hardware actually available, and to understand precisely why it does, or doesn't. It's to confirm that a given stack meets the scenario's accuracy and latency bar on the hardware actually available, and to understand precisely why it does, or doesn't.
Choosing a serving framework and understanding what each one optimizes for
Three open-source frameworks matter most for production LLM inference today: vLLM, SGLang, and TensorRT-LLM, and each one makes a different bet about where the engineering effort should go. vLLM is serving-first FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving MLCommons Releases New MLPerf Inference v5.0 Benchmark Results. PagedAttention alone cuts KV cache fragmentation waste to under 4%, which unlocks throughput gains of multiple times over naive allocation schemes.
SGLang is also serving-first but leans harder into runtime-level optimization. Its RadixAttention mechanism reuses shared prefixes efficiently, which matters a lot for chat templates, retrieval-augmented generation pipelines, and agent workflows where many requests share a common opening sequence FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving. On H100s, SGLang tends to beat vLLM on throughput for prefix-heavy workloads or smaller models, though the margin shifts depending on model size and prompt distribution FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving. It also supports speculative decoding, delivering up to 3.6x faster generation in some configurations, which lines up directly with v6.0's expanded speculative decoding support for DeepSeek-R1 FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving.
TensorRT-LLM takes the opposite approach: engine-first. The model gets compiled into an optimized TensorRT engine ahead of time, which opens up more room for NVIDIA-specific tuning but makes the setup and iteration loop considerably more involved. It's the natural pick for anyone trying to squeeze the last bit of per-GPU throughput out of NVIDIA silicon. Sitting alongside these is NVIDIA Dynamo, an open-source distributed serving framework built for multi-node, factory-scale deployment; it handles disaggregated serving, routing, and memory extension, and it's what powered the disaggregated Llama 3.1 405B interactive submission in v5.1, marking Dynamo's first appearance in the benchmark MLCommons Releases New MLPerf Inference v5.0 Benchmark Results.
HuggingFace TGI is now officially in maintenance mode, and HuggingFace itself points users toward vLLM or SGLang instead. Anyone starting a reproduction project from TGI should migrate first, rather than benchmark a framework that's no longer where active development happens. In practice, framework choice decides which MLPerf scenario is even within reach: TensorRT-LLM is the more credible path to Closed division per-GPU throughput records, while vLLM and SGLang are the more accessible route for practitioners reproducing server and interactive scenarios on hardware they can actually get their hands on.
The benchmark client's measurement error and its correction
A less obvious failure point sits in the load-testing client itself, not the server being tested. Research published in 2026, "Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks" by Chandrasekar and Kramberger at Google, found that widely used benchmarking tools rely on single-process, asyncio-based architectures that build in client-side queuing bottlenecks once concurrency climbs. The mechanism is specific: Python's Global Interpreter Lock throttles how fast the client itself can service requests, and as the event loop approaches saturation, the client-side wait time trends toward infinity, a behavior the paper models as an M/G/1 queue.
The failure mode this produces is genuinely deceptive. Standard benchmarking tools timestamp latency at the moment a token gets processed rather than when it actually arrives over the network, so queuing delay sitting inside the client gets misattributed to the model server as if the server itself had slowed down. What looks like infrastructure degradation is, in these cases, an artifact of the measuring instrument.
The paper's proposed fix is a multi-process evaluation framework that spreads client-side load across processes so queuing overhead becomes negligible, paired with a new metric called Normalized Time Per Output Token (NTPOT). NTPOT amortizes end-to-end latency, prefill and scheduling delays included, across the full sequence length rather than isolating decode time alone. That distinction matters directly for MLPerf reproduction, because prefix caching makes initial response times swing widely, and a standard TPOT measurement quietly hides heavy prefill computation by only looking at decode.
The practical consequence is blunt: a reproduction run using a single-process load generator at high query rates will overstate server-side latency, systematically and predictably. The reported number might pass or fail the MLPerf constraint for reasons that have nothing to do with the serving stack under test. Anyone building a reproduction pipeline around SGLang, incidentally, can look at the benchmarking utilities the project maintains directly in its own repository, a framework-native alternative worth checking against whatever generic client gets used. Among the tools named as affected, LLMPerf is a single-process asyncio architecture that exhibits this behavior under production-scale request rates (hundreds to thousands of QPS), while k6 and Locust, though also subject to client-side measurement distortions for LLM workloads, are not strictly single-process, since Locust natively supports distributed master-worker mode and k6 uses Go goroutines rather than asyncio.
How parallelism configuration determines which benchmark scenarios are reachable
Most serving stacks make a parallelism decision once, at deployment time, and then live with it. Tensor parallelism (TP) shards model weights across GPUs, which cuts per-request latency and makes long contexts workable under tight service-level agreements. Neither one serves the offline scenario and the interactive scenario equally well at the same time.
This isn't a theoretical concern. Both traces show bursty arrival patterns and wildly uneven request lengths, the kind of workload no single fixed parallelism strategy handles well across the board FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving. A research system called Flying Serving, out of Oregon State and Oak Ridge, built directly against this problem: a vLLM-based system that switches between DP and TP configurations online, without restarting engine workers FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving. Flying Serving, a vLLM-based system enabling online DP-TP switching without restarting engine workers, improves performance by up to 4.79× under high load and 3.47× under low load compared to static parallelism FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving.
For a reproducer, none of this is a deployment detail to sort out later. It shows whether the interactive scenario's latency constraint is reachable. MoE models add expert parallelism (EP) as a third dimension, directly relevant to the DeepSeek-R1 (671B MoE) and GPT-OSS 120B benchmarks in v5.1 and v6.0. For reproducers, a stack can meet the latency constraint of the interactive scenario at all only with the right parallelism choice, and choosing DP for an interactive scenario on a 405B model is likely to fail because the parallelism strategy inflates TTFT, not because of the serving framework MLCommons Releases New MLPerf Inference v5.0 Benchmark Results.

KV cache behavior as the hidden variable between your serving config and your benchmark result
KV cache memory is where a lot of reproduction attempts quietly go wrong, mostly because it grows in ways that aren't obvious from the model size alone. Memory consumption scales linearly with both sequence length and batch size, and at long context lengths the KV cache itself can outweigh the model's own weights in total memory footprint. A 70-billion-parameter model running an 8K context window needs roughly 20GB of KV cache per request, and scaling that up to a realistic batch size is enough to exhaust even high-memory GPUs.
Running the formula for Llama 3.1 70B at FP8 precision with an 8K context and a batch of eight requests comes out to roughly 10GB per batch, a number that sets the floor for whatever NVMe offload path or network bandwidth the system needs to sustain FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving. PagedAttention brings fragmentation waste in that memory down to under 4%, which is a real gain, but eviction and offload decisions still introduce latency variability that appears directly in TTFT measurements during a reproduction run.
Prefix caching complicates this further. Both SGLang's RadixAttention and vLLM's own prefix caching reuse KV blocks across requests that share a common prefix, and a benchmark run against a warm cache will report a very different TTFT than the same run against a cold one. Combined with the earlier point that standard TPOT measurements already obscure prefill cost, this means a reproducer needs to control cache state explicitly between runs, not treat it as background noise.
Disaggregated serving puts KV cache transfer directly on the critical path: the KV tensors generated during prefill have to physically reach the decode worker before generation can start, and at long context lengths, that transfer time increasingly dominates TTFT. Splitwise, DistServe, and Mooncake all address this problem from different angles, and AMD documented its own version of prefill-decode disaggregation on Instinct MI300X GPUs using SGLang in August 2025. LMCache offers one practical answer: a KV cache engine that plugs into vLLM, SGLang, and NVIDIA Dynamo, and serializes KV blocks out to local NVMe, Redis, or remote object storage. Mooncake (Qin et al., 2025, USENIX FAST '25) is a KVCache-centric disaggregated architecture that separates the cache store from compute and pools it across nodes, with KV eviction routed to local NVMe SSDs through a P2P DMA path.
Sources
- Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks
- FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
- MLCommons Releases New MLPerf Inference v5.1 Benchmark Results - MLCommons
- insidehpc.com
- MLCommons Releases New MLPerf Inference v5.0 Benchmark Results - MLCommons
- MLCommons Releases New MLPerf Inference v6.0 Benchmark Results - MLCommons
- Benchmark MLPerf Inference: Datacenter | MLCommons V6.1
- mlcommons.org


