Llama 3.1 405B Multi-Node Serving Hardware Requirements
Four sequential hardware layers determine whether serving works or merely looks feasible on paper.

Serving Llama 3.1 405B at production scale is a layered hardware problem, not a single-number specification. KV cache demand scales with context length and concurrent users, parallelism strategy determines how the memory and compute load distribute across nodes, and interconnect and storage dictate whether data actually moves fast enough to keep the GPUs busy. Each layer has a different owner in the hardware stack, and a failure at any one of them caps throughput regardless of what the others provide.
Why Llama 3.1 405B is a layered hardware problem, not a single-number spec
A model with 405 billion parameters exposes every assumption a generic GPU cluster has inside it. From that first fact, every subsequent engineering decision becomes a trade-off with a measurable throughput cost. HBM holds weights and active cache, CPU DRAM and NVMe absorb the offload tiers, and InfiniBand or NVLink carries the traffic between them. No single engineering choice is free at this scale.
The practical consequence is that a hardware requirement statement for 405B must address at least four distinct layers in sequence. Weights establish the minimum GPU count and precision tier. KV cache introduces a dynamic ceiling that grows with every user and every token of context. Parallelism strategy distributes those demands across nodes in a way that respects the physical network topology. Interconnect and storage then determine whether the distribution can sustain throughput under load. Understanding the constraints in this order, weights then KV cache then interconnect then storage, is the only way to produce a specification that works in production rather than one that merely looks viable on paper.

How model precision determines the minimum GPU count before anything else is considered
Precision is the first decision an operator makes because it establishes the physical lower bound on GPU count. A single 8×H100 80GB node holds approximately 640 GB of addressable HBM, which falls short of the FP16 weight footprint alone. At that precision tier, multi-node deployment is not optional. The arithmetic forecloses any single-node configuration.
FP8 changes the calculus. At roughly 405 GB for weights, an 8×H100 node can hold the model, and Meta provides a mixed BF16/FP8 checkpoint that preserves numerical stability by keeping non-linear operators such as layernorms in higher precision while compressing the major linear weight operators to FP8. Q4 quantization at roughly 245 GB fits within a single high-memory node as well, but the quality trade-offs make it a last resort for production serving rather than a first recommendation. Each tier either forces multi-node deployment or changes which single-node configuration is viable, and you make that choice before any other hardware decision has meaning.

KV cache: the memory cost that grows with every user and every token
KV cache is the constraint that practitioners actually encounter in production, because it is not a fixed cost like weights. It scales with context length and concurrent batch size, and at 128K tokens with even a modest number of concurrent users it can rival or exceed the weight footprint in GPU memory demand. Llama 3.1 405B mitigates this with Grouped Query Attention, using 128 query heads but only 8 KV heads, which delivers a 16× KV cache reduction versus standard multi-head attention. Without GQA, the cache problem at this scale would be substantially worse.
Even with GQA, the per-token KV cost in BF16 runs roughly 0.516 MB per token. The Kempner Institute benchmark on 16×H100 with vLLM 0.8.5 records 123.05 GB of KV cache VRAM for the full 128K token context. Scaling to a batch of concurrent users multiplies the cache requirement linearly, and a small batch fills GPU HBM for cache alone before model weights are accounted for. The optimization stack that vLLM exposes shows how much each technique contributes: naive pre-allocation in BF16 is the baseline, while PagedAttention, vLLM's block-sparse KV management, doubles to quadruples concurrent users on the same GPU footprint by eliminating internal and external fragmentation.
Parallelism strategy: how to distribute 405B across nodes without losing throughput to communication overhead
Once the memory requirements are established, the question becomes how to decompose the model across the nodes required to hold it. The canonical production pattern for multi-node 405B serving is 2D parallelism: tensor parallelism within a node and pipeline parallelism across nodes. The two strategies have fundamentally different communication profiles that match the two fabrics available in a GPU cluster, which is precisely why the pairing works. Tensor parallelism alone across nodes meets aggregate memory requirements but incurs high communication overhead, because every transformer layer requires two allreduces across all participating GPUs, one for the attention block and one for the MLP block. Inter-node bandwidth is far lower than intra-node NVLink bandwidth, and the cost of those allreduces compounds across layers.
The common NVIDIA NIM configuration for two 8-GPU nodes sets TP=8 and PP=2. Each node handles tensor parallelism within its own NVLink domain, and only pipeline-stage boundary activations cross the inter-node fabric. A scatter-allgather optimization, drawn from Megatron-LM work, lets each GPU send only a fractional slice of the activation over the slower inter-node link while the full tensor is recovered within the node via intra-node allgather. Snowflake AI Research demonstrated the scale of what this approach buys: a stack combining hardware-agnostic FP8 ZeroQuant with optimized tensor and pipeline parallelism implementations achieved up to 3× lower end-to-end latency and more than 1.4× higher token throughput compared to their baseline for Llama 3.1 405B on common enterprise workload patterns of 2K input and 256 output tokens, with the stack upstreamed to vLLM and DeepSpeed.
Interconnect requirements: what NVLink, InfiniBand, and RoCE each contribute and where each fits
Multi-node 405B serving depends on two separate network fabrics, and under-provisioning either one creates a throughput ceiling that no amount of GPU memory solves. Within a node, NVLink, and NVSwitch in HGX configurations, provides the bandwidth that makes 8-way tensor parallelism practical. Running without NVLink and relying on PCIe for intra-node GPU-to-GPU traffic results in significantly slower processing, because the allgather operations in tensor parallelism are bandwidth-bound. The intra-node fabric is not a performance nicety. It is the substrate on which the entire tensor-parallel decomposition rests.
Across nodes, InfiniBand and RoCEv2 are both viable for pipeline-parallel activation transfers, but the choice has downstream implications. NDR InfiniBand at 400 Gb/s per port is the high-performance path. Beneath both fabrics sits GPUDirect RDMA, the enabling technology for both storage offload and KV cache disaggregation. NVMe-over-Fabrics with RDMA extends this to remote NVMe storage at near-local speeds for cold KV cache retrieval, but NVMe/RDMA requires RDMA-capable NICs such as InfiniBand or RoCE. The storage retrieval path, the third fabric in the system, inherits its performance characteristics directly from these interconnect choices.
Storage requirements: loading 810 GB of weights, fast NVMe, and the three-tier KV cache offload hierarchy
Storage is not a background concern for 405B serving. The weight loading time alone is operationally significant, and the KV cache offload hierarchy is the mechanism that makes long-context, high-concurrency serving economically viable. FP16 model weights occupy roughly 810 GB on disk, and fast NVMe SSDs with at least several terabytes of free space are required to hold the weights and handle intermediate data. The Kempner Institute's deployment documentation notes that loading 810 GB of weights onto 16×H100 GPUs can take up to two hours, a startup latency that directly affects deployment operations, rolling restarts, and failover planning. Aggregate NVMe read bandwidth matters more than individual drive IOPS.
The sustained serving path is governed by a three-tier KV cache offload hierarchy. Hot KV blocks, those actively generating tokens, remain in GPU HBM where latency is lowest. Warm KV blocks, completed sequences likely needed for multi-turn conversations, spill to CPU DRAM, which is orders of magnitude cheaper per byte than HBM. Cold KV blocks, historical context and prefix cache candidates, land on NVMe, where the latency penalty for a cold load is large in absolute terms but far smaller than recomputing a 128K-token prefix from scratch. GPUDirect RDMA and NVMe-oF together allow cold KV blocks to be fetched from a networked NVMe pool directly into GPU HBM without CPU involvement, which keeps the storage retrieval path from becoming a CPU bottleneck at scale. The ceiling on what well-provisioned storage can deliver is high: the MLPerf Storage v3.0 benchmark recorded a three-node AIMesh system delivering 379.9 GiB/s of load bandwidth for 5.29 TB of Llama 3 405B model state, reading the full checkpoint in approximately 14 seconds. GPUs that wait on data are not doing inference, and the gap between that benchmark and what slow NVMe imposes is the difference between a viable deployment and an idle capital expense.
What MLPerf results reveal about which hardware configurations actually deliver at 405B scale
MLPerf results for Llama 3.1 405B are the closest thing to a controlled cross-system comparison the industry has, and they reveal that hardware generation and system-level integration, not just GPU count, determine throughput at this scale. The MLPerf Inference v5.0 server scenario holds systems to latency targets of 6 seconds for time to first token and 175 ms for time per output token before throughput numbers count. The generation-over-generation gains are substantial. GB200 NVL72 delivered up to 3.4× higher per-GPU performance on Llama 3.1 405B compared to H200 8-GPU systems. In MLPerf v5.1, Lambda AI on 8×B200 achieved throughput of 1,648 tokens/s in Offline and 1,246 tokens/s in Server for Llama 3.1 405B with FP4 weights, a meaningful server-scenario gain over the prior round's best result. At the extreme end, NVIDIA set a time-to-train record of 10 minutes on the Llama 3.1 405B MLPerf Training v5.1 benchmark using 5,120 Blackwell GPUs, a scale that underscores how far purpose-built infrastructure diverges from a minimally viable deployment.
How precision and compression choices propagate through every layer of the stack
Precision is the master variable that sets every hardware requirement at once. The same choice that determines GPU count from the weight footprint also determines KV cache memory density, since FP8 KV halves cache pressure, and interconnect traffic volume, since smaller tensors move faster, and storage footprint, since FP8 weights halve the disk requirement and load time. The decision made at model load time reverberates through every layer analyzed so far.
Weight quantization and KV cache quantization are separable decisions with different accuracy profiles. NVFP4 KV, available only on Blackwell hardware, carries the compression further still, reaching roughly 8× the baseline concurrent user count in benchmarked configurations, but the hardware dependency is absolute and excludes H100 and A100. Lossless compression research represents a third path that has not yet reached 405B-scale production deployment: research published in 2025 demonstrated a lossless dynamic-length float compression scheme achieving no accuracy or perplexity loss versus BF16 originals on tested models at roughly 70% of original file size, a result that matters for storage footprint and load time if it scales to 405B, though no source confirms 405B-scale results. A100 operators face a distinct constraint: the A100 lacks native FP8 compute, so FP8 weight compression requires the GPTQ-Marlin FP8 kernel, which achieves memory savings but may not recover the full compute throughput benefit that native FP8 hardware delivers.
Building a complete hardware spec for production 405B serving: minimum viable, reference, and high-throughput configurations
A complete hardware spec for multi-node 405B serving must specify GPU count and HBM, intra-node interconnect, inter-node fabric, host-side DRAM for KV offload, NVMe capacity and throughput, and parallelism strategy simultaneously. The minimum viable H100-based configuration for full 128K context starts at 12×H100 80GB to hold weights plus full-context KV cache, with 16×H100 across two 8-GPU HGX nodes as the verified production reference that provides headroom. NVLink handles intra-node traffic, and NDR InfiniBand with eight ports per server carries inter-node pipeline-parallel traffic. The spec assumes FP16/BF16 weights, at least several terabytes of fast NVMe for weight storage, and substantial CPU DRAM for warm KV offload. Weight load time must be budgeted at up to two hours, and rolling restart and failover plans must account for that window explicitly.
The single-node H200 path challenges the premise that 405B is inherently a multi-node problem. Eight H200s with 141 GB HBM3e per GPU accommodate 405B in FP8 on a single node, eliminating inter-node InfiniBand latency entirely and simplifying the parallelism stack to TP=8 only. For teams that can provision H200, single-node FP8 is a legitimate alternative to two-node H100. The high-throughput Blackwell tier is where the benchmark results point next. GB200 NVL72 or B200 8-GPU configurations unlocked substantially higher per-GPU throughput in MLPerf benchmarks and enable NVFP4 KV cache compression, which reaches the highest concurrent user counts of any tested configuration. Disaggregated prefill and decode serving over NVLink becomes viable at this tier and is the architecture that delivered the largest throughput gains in MLPerf v5.1. The storage and interconnect decisions are as important as GPU selection, and that is the through-line of every layer a production 405B deployment must get right.


