FP8 vs BF16 Inference Throughput and Memory Tradeoffs for Llama and Qwen Models
FP8 cuts memory but batch size and hardware determine whether throughput gains are real.

FP8 versus BF16 inference for Llama and Qwen is a choice about memory, batch size, serving software, and arithmetic speed. FP8 pays off when it removes the constraint limiting deployment, whether that is weight capacity, KV cache pressure, or compute throughput. Storage infrastructure such as Open Lake belongs in that assessment because smaller tensors leave more room for serving depending on where the data lives.
Why FP8's theoretical 2× advantage doesn't survive contact with production decoding
FP8 offers a theoretical 2× Tensor Core throughput advantage over BF16, but production decoding rarely resembles the dense matrix multiplications behind that ceiling. Autoregressive serving makes one token at a time for each sequence, so its thin matrix operations are often memory-bandwidth-bound. The useful comparison is between complete serving configurations.
The format and hardware path also matter. Serving usually uses E4M3 for weights and activations and often keeps BF16 for sensitive layers, while training uses E5M2 for backward-pass gradients. Hopper supports FP8 with per-tensor scaling, and Blackwell adds hardware-native MXFP8 block scaling. Ampere A100 has no hardware FP8 path, while Ada Lovelace supports FP8 operations but lacks dedicated runtime amax-tracking hardware. These differences shape which throughput rates can be compared, even before batch size comes into play.
What real throughput measurements on Llama models show, and why batch size is the deciding variable
Real serving measurements on Llama models put FP8 speedups at roughly 1.15× to 1.84×. Model size and batch size together explain much of that range. Larger batches make matrix operations denser, giving the GPU more useful arithmetic relative to the memory traffic it must sustain. As a result, a large-batch measurement can be accurate yet still be a poor forecast for a latency-sensitive deployment.
Batch size also changes the energy comparison. Chung et al. (2026), from the ML.ENERGY Initiative, found that lower precision is not always faster or more energy efficient, and that differences in GPU utilization led to large gaps in energy use. The batch regime affects both how much work the GPU completes and how effectively it uses its available resources. Measurements at the intended serving batch are more informative than the best number a benchmark can produce.
How hardware generation changes the math for Qwen 3.5 on B200 and B300
Qwen 3.5 on B200 and B300 illustrates why the Llama results cannot become a universal FP8 multiplier. On Blackwell, the FP8 advantage over BF16 is materially larger than on Hopper, but it still varies with the operating point. The gap widens as interactivity requirements rise. Precision choice consequently depends on the latency target as well as the model and GPU.
Tighter latency budgets push serving toward smaller effective batches per chip, yet FP8’s smaller memory footprint can help sustain concurrency within HBM. In that regime, capacity contributes to throughput even when matrix shapes do not approach the compute ceiling. The B200 and B300 comparison is therefore not simply a progression from one arithmetic rate to another. It shows why operators must calibrate FP8 against their own hardware and SLA rather than carry a speedup ratio between generations.
Why VRAM is the constraint that FP8 actually solves for large models
For large models, the first question can be whether the weights fit at all. A large set of BF16 weights can take up more than the H100’s 80 GB HBM or leave too little room on an H200 for KV cache and concurrent requests. FP8 halves the weight footprint, which can move a model across the boundary from out-of-memory to single-GPU deployment. That is a capability change rather than a marginal speed improvement, though the remaining memory still limits serving capacity.
Blackwell changes the baseline. The B200 provides 192 GB and the B300 provides 288 GB, allowing large BF16 weight sets to fit with meaningful headroom. FP8 then becomes a throughput and cost optimization on top of a viable BF16 deployment. On H100 fleets, by contrast, capacity can force the choice before throughput is measured. The same fixed-memory problem extends into KV cache management, where infrastructure such as Open Lake addresses how larger models and more requests share the available memory hierarchy.
KV cache memory is where FP8 pressure compounds at long context

For Llama 3.1 70B at 128K context, one user’s BF16 KV cache takes roughly 40 GB. Eight concurrent users require roughly 320 GB, four times an H100’s entire HBM capacity before accounting for weights. At that scale, keeping the cache in memory requires offload to serve more than one user. Weight quantization alone cannot resolve the constraint because KV cache is a separate memory consumer.
FP8 KV quantization halves that per-user pressure. In vLLM, --kv-cache-dtype fp8 enables the reduction, allowing roughly twice as many concurrent users on a B200 without offload compared with BF16 KV. Architectural compression, including GQA and MLA, can compound with precision reduction, while prefix caching adds reuse. These mechanisms address different sources of memory demand.
Reasoning workloads make the pressure especially concrete. A single 30,000-token reasoning chain from DeepSeek-R1-Distill-Llama-70B uses roughly 9 GB of KV cache at FP16. FP8 cuts that to roughly 4.5 GB, so one request takes up less memory and leaves more room for others.
How the serving stack manages KV cache across the memory hierarchy
When KV cache exceeds HBM, the solution extends beyond the GPU. Cache blocks can be managed through a hierarchy of GPU HBM, CPU DRAM, local NVMe, and networked storage rather than discarded solely because the fastest tier is full. Lower precision cuts the amount of data, while where it is kept and how it is moved determine how it can be used.
NVIDIA NIXL, the NVIDIA Inference Xfer Library, was published in 2026 as part of the ai-dynamo project and provides an open-source transfer layer for this hierarchy. It is the software surface through which serving systems interact with storage. Open Lake addresses the storage side through KV cache management across GPU memory, local NVMe, and networked storage, with infrastructure designed for high-throughput, low-latency LLM serving. It works between the transfer layer and physical media. GPU-resident caching, precision reduction, and storage-backed offload thus solve different parts of the same capacity problem.
What the accuracy tradeoff actually looks like, and where it concentrates
The type of task tells more about the risk to FP8 accuracy than parameter count alone. Math and code generation are generally more tolerant of precision reduction than knowledge-heavy reasoning. For Qwen3-32B on a single H100, FP8 scores 69.64% on MMLU-Pro against 70.24% for BF16, a 0.6-point gap. On HumanEval, both formats score 39.02%. For operators, this difference says more than one average across tasks that have little in common.
KV quantization deserves a separate evaluation because long-context retrieval concentrates a different risk. FP8 KV results show regressions of roughly 0.3–0.7 points on long-context retrieval benchmarks, despite attention’s relative tolerance of small rounding errors. Operators serving knowledge-heavy reasoning should therefore validate against their own evaluation set, with particular attention to retrieval across long contexts. For math, code, and general instruction-following on Hopper or Blackwell with per-tensor scaling, the accuracy case for FP8 is strong. Throughput remains more dependent on the workload.
One systems-integration failure mode operators miss when FP8 is slower than BF16
Qwen 3 Coder 480B A35B provides a concrete counterexample to assuming that lower precision wins. At benchmarking time, its FP8 path in vLLM required attention data parallelism, while BF16 could use tensor parallelism. Straggler GPUs created load imbalance, leaving other GPUs waiting while consuming static power. FP8 was consequently slower and more energy-hungry across every batch size tested.
Chung et al. (2026) identify GPU utilization as a key driver of inference time and energy, which explains why a parallelism mismatch can overwhelm format-level savings. Before committing to FP8, operators should verify that the framework’s parallelism strategy for the target model is equivalent to or better than the BF16 path. End-to-end throughput and energy measurements reveal this mismatch more reliably than per-operation speed.
What MLPerf v6.0 and v6.1 results reveal about where the precision frontier is moving
MLPerf Inference v6.0 and v6.1 results show the highest-throughput production-representative submissions combining FP8 KV caching with FP4 or MXFP4 weights. In that tier of the benchmark record, FP8 is a foundation used alongside lower precisions. A submission can reduce them differently because they serve different roles and impose different accuracy and systems constraints.
A production-representative submission shows what one configuration can do under benchmark conditions. Other serving frameworks or workloads may need more work to match that result. For operators, these results make FP4 part of planning, while the current FP8-versus-BF16 choice still depends on the stack in use.
FP4 on the horizon and what it means for the FP8 vs. BF16 decision today
Blackwell’s MXFP4 and NVFP4 formats are beginning to deliver end-to-end speedups beyond FP8, but they require format-specific quantization. Naively applying FP8 or INT4 techniques can cause meaningful accuracy loss. NVFP4 uses roughly 4.5 bits per element on average, compared with roughly 4.25 for MXFP4, so it gives up some compression to gain precision. MXFP4’s power-of-two scale quantization can severely degrade accuracy without mitigation.
The ICLR 2026 paper Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization introduces MR-GPTQ to address these format-specific problems. Its block-wise Hadamard transforms and specialized optimizations show that the accuracy work is substantive. FP8 remains the validated, lower-risk production choice on Hopper and Blackwell, subject to workload and integration checks. Teams on Blackwell should monitor MXFP4 and NVFP4 support in vLLM and TensorRT-LLM.
Lossless compression as a complementary lever for BF16 and FP8 storage
Lossless compression addresses storage and transfer costs without introducing another accuracy tradeoff. Cloudflare’s Unweight research (April 2026) demonstrated lossless MLP weight compression by separating and entropy-coding BF16 exponent bytes. BF16’s 8 exponent bits cluster around a narrow range in practice, creating redundancy that compression can exploit without changing tensor values.
Intel research (August 2025) extended exponent-based techniques to KV caches, finding skewness similar to that in model weights. FP8 E4M3 exponent streams are also compressible, while mantissa streams offer less redundancy. Quantization and lossless compression can therefore stack at the NVMe and networked storage tiers. FP8 first reduces the representation, and compression then reduces the space needed to store that representation.
For Open Lake, this stacking guides compression at the storage layer that uses tensor structure without changing model accuracy or the serving framework. The research supports the mechanism, but it does not establish a universal production compression ratio. A smaller stored cache also need not cut active GPU memory by the same amount.
A decision framework for choosing between FP8 and BF16 based on hardware, model size, and workload
Start with hardware and weight capacity. On H100 and H200, FP8 may be needed to fit large models, while B200 and B300 more often have room for BF16, making it a useful baseline for comparison. A100 has no hardware FP8 path, with INT8 representing a different alternative and accuracy tradeoff. Next, measure at the batch regime and latency target that will be used, checking energy use and how parallelism works as well as peak throughput.
Then separate task quality from cache capacity. Math, code, and general instruction-following usually present a strong accuracy case for FP8, while knowledge-heavy reasoning calls for workload-specific evaluation of FP8 KV quantization. Long-context deployments that exceed HBM should pair cache precision reduction with offload infrastructure when the reduced cache still does not fit. Plans for Blackwell should also allow for FP4’s progress and the work needed to use it.
Purpose-built NVMe and networked storage, including Open Lake, adds tiers where the memory saved by FP8 can make room for more serving work. The storage layer determines how far that benefit can reach beyond the GPU.


