FP8 vs BF16 Inference Throughput and Memory Tradeoffs for Llama and Qwen Models
FP8 cuts memory but batch size and hardware determine whether throughput gains are real.
Tobias Wrenner
Section
8 stories in Model-to-Hardware Fit.
FP8 cuts memory but batch size and hardware determine whether throughput gains are real.
Eliminating the CPU bottleneck in GPU storage access unlocks faster inference on longer contexts.
Four sequential hardware layers determine whether serving works or merely looks feasible on paper.
Bandwidth, not storage capacity, determines how many NVMe drives you need for KV cache offloading.
Sparse expert routing requires keeping all weights accessible, not just active ones.
NVLink keeps tensor parallelism fast; PCIe makes pipeline parallelism necessary.
H100's compute advantage shrinks dramatically when KV cache fills up.
Mixtral's sparse activation saves compute but demands full expert weights in VRAM.