Tensor Parallelism Degree Selection for Llama and Mixtral Multi-GPU Deployments

Tensor parallelism degree selection for Llama and Mixtral is an interconnect-constrained optimization. The right degree balances synchronization cost, KV cache headroom, and the deployment’s latency or throughput objective. NVMe and tiered storage enter this decision when GPU memory cannot sustain the required cache, extending workload feasibility while preserving the role of fast GPU interconnects.
Why tensor parallelism degree is an interconnect-constrained optimization, not a model-size lookup
The intuitive approach is to divide model weight memory by GPU capacity and select the smallest tensor parallelism degree that fits. That calculation establishes only a memory floor. It does not show whether the GPUs can exchange intermediate results quickly enough to make the configuration efficient. The optimization is therefore over usable memory and computation after synchronization costs.
Synchronization is built into the execution path. Every transformer layer in a TP deployment requires two all-reduce collective operations, so Llama 3.1’s 80 layers produce 160 synchronization points per forward pass. The Ohio State University study Characterizing Communication Patterns in Distributed Large Language Model Inference finds that sequential autoregressive decoding limits computation-communication overlap. Communication consequently consumes a larger share of execution time in inference than in training. Even a configuration that handles those exchanges efficiently still needs memory beyond its weights, because long-context KV cache demand can become the binding constraint.
How all-reduce communication scales with TP degree and why interconnect bandwidth determines the ceiling
Increasing TP degree spreads computation across more GPUs, but each layer’s collective operations must coordinate a larger group. Adding GPUs does not remove those synchronization points. The gain from smaller computation shards must outweigh the cost of exchanging results, and the balance depends on both the interconnect and the collective implementation. The Ohio State study finds that NCCL all-reduce is slower than it should be for small messages, which is a major bottleneck in multi-node TP.
NVLink 4.0 on H100/H200 SXM provides 900 GB/s bidirectional bandwidth per GPU in an 8-GPU domain, making TP=8 within a node viable with manageable overhead. PCIe Gen 5 x16 has more bandwidth than Gen 4 but is still well below NVLink. NVLink 5.0 on B200/B100/GB200 reaches 1800 GB/s, further reducing synchronization stalls. These distinctions prevent GPU count alone from serving as a useful performance predictor.
Cross-node TP introduces another link into the collective path. All-reduce must traverse both NVLink and InfiniBand, making the slower link the bottleneck. A degree that performs well within one node therefore cannot be assumed to perform similarly across nodes. The hybrid most often used in production keeps TP within the node and uses pipeline parallelism across nodes. That arrangement preserves fast communication for tensor shards while moving the network boundary between pipeline stages.
KV cache growth as the second axis of the TP selection problem, separate from weight fit
KV cache demand depends simultaneously on layers, KV heads, head dimension, sequence length, batch size, and bytes per element. Weight memory alone cannot predict that demand. Extended context with a moderate batch in FP16 can consume tens of gigabytes beyond the weights, and a single concurrent user’s cache on a large model can rival the weight footprint. A configuration can therefore load the model but have too little headroom to serve its actual workload.
The practical sizing heuristic is to reserve 40–50% of VRAM for KV cache and runtime overhead beyond weight storage. A per-GPU weight share of 35 GB consequently calls for GPUs with 60 GB or more to sustain meaningful batch sizes, exceeding the capacity of 40 GB GPUs. Higher TP can create room by shrinking each weight shard. The supplied Llama examples leave roughly 45 GB free per GPU on 4× A100 80 GB and roughly 62.5 GB on 8× H100 80 GB after BF16 weights. Those figures describe memory headroom.
Cache allocation and precision can also be changed. When GPU HBM still cannot hold the cache, storage enters the design. NVIDIA announced its Inference Context Memory Storage Platform, or ICMSP, at CES 2026 to standardize offload to NVMe SSDs. NVIDIA Dynamo’s distributed KV cache management, announced at GTC 2025, orchestrates movement across GPU HBM, DRAM, local NVMe, pod-level shared NVMe through G3.5 via NVIDIA CMX, and enterprise storage. This tiered architecture extends long-context feasibility, but the supplied evidence does not quantify a cost-per-token improvement from NVMe tiering.
Latency versus throughput as the workload axis that determines which TP degree is correct
The latency-SLO engineer and the throughput/cost engineer optimize different outcomes. The first needs an individual request to meet its response-time target. TP helps by parallelizing computation within that request, and the Ohio State study finds superior response times for brief sequences despite substantial network overhead. For a mid-sized Llama model across 8 GPUs, pure TP=8 with PP=1 occupies the low-ITL end of the configuration trade-off. That makes high intra-node TP relevant when inter-token latency, or time per output token, is binding.
TTFT requires a separate assessment from decode latency. The supplied research places TP=8 at the low-ITL/high-TTFT end and pipeline-dominant execution at the low-TTFT/high-ITL end. High TP is therefore not a universal answer to every latency SLO. Pipeline parallelism minimizes data transfer but can increase total latency, and the supplied scaling example shows substantial increases in end-to-end latency and TTFT from PP=2 to PP=4. The useful comparison is between configurations against a named latency metric, rather than between parallelism methods against an undifferentiated promise of speed.
The throughput/cost engineer instead aims to get more tokens per dollar. Assigning more GPUs to each request-handling group leaves fewer groups available for concurrent replicas, while synchronization prevents speedup from scaling proportionally. At moderate request rates, smaller TP groups with concurrent replicas can outperform one larger group on that objective. Pipeline parallelism remains a cross-node throughput option, but its latency penalty limits its appeal for response-time-sensitive service.
Concrete TP degree selections for Llama and Mixtral across real hardware configurations
For an 8-GPU H100/H200 SXM node, NVLink 4.0 makes TP=8 a viable intra-node choice. The relevant Llama latency example uses 8 GPUs with TP=8 and PP=1 when low ITL is the priority. A throughput-oriented deployment must weigh that group against smaller TP replicas that still meet weight and cache requirements.
For memory planning, 4× A100 80 GB and 8× H100 80 GB provide the Llama headroom reference points established above. They do not establish a hardware-independent winning degree. Even with the same weight footprint, the limits on serving can change with context length, batch size, and cache precision. On PCIe-only systems, the supplied framework instead sets TP=2 as the maximum before communication cost dominates.
For a large model served through vLLM on 2 nodes with multiple GPUs each, the concrete pattern is to set --tensor-parallel-size to the per-node GPU count and --pipeline-parallel-size to the node count. TP then stays within the NVLink domain. PP crosses the network boundary. For Mixtral, the supplied decision framework favors a TP+EP hybrid rather than treating the dense-model choices as directly transferable, because expert placement changes the communication problem.
Why Mixtral's MoE architecture changes the TP selection logic and when expert parallelism should lead
Mixtral’s mixture-of-experts architecture separates weight residency from active computation. Every expert’s weights must stay in VRAM, but each token uses only a small subset. Total parameters therefore determine the weight-sizing requirement, while routing and expert activation shape execution rather than matrix computation across every stored weight. A dense-model calculation captures the first constraint but not the second.
Expert parallelism assigns complete, non-overlapping experts to different GPUs and routes tokens to the appropriate GPU. That avoids TP all-reduce overhead for expert FFN blocks. When GPU count allows, EP is therefore the preferred strategy for those layers, with a TP+EP hybrid addressing the model’s broader parallelization needs. The principle is to account separately for non-expert computation and expert execution. At scale, EP-only execution also remains conditional on balanced expert demand, because uneven routing can leave the expert workload unevenly distributed across GPUs.
Where communication overhead in TP can be addressed by the collective library rather than by reducing TP degree
The Ohio State study locates a specific problem in NCCL’s small-message behavior in multi-node deployments. That distinction separates the cost inherent in synchronization from the additional cost of a particular implementation. Reducing TP can lower both costs, but it can also give up the lower latency that led to the larger group.
NVRAR provides a concrete collective-layer alternative. Built on NVSHMEM, this hierarchical all-reduce achieves up to 1.14×–3.57× lower latency than standard NCCL implementations for 256 KB–2 MB messages on InfiniBand. Integrated into YALIS and vLLM, it improves multi-node TP inference for large dense and MoE models. Those results hold for the stated message range and setting and do not show that NCCL should be replaced in all cases.
MLCommons released MLPerf Storage v3.0 in September 2026 with a KV Cache test for LLM inference cache read/write operations. The test makes storage I/O a benchmarkable dimension of serving infrastructure, distinct from GPU collective latency. For operators, the practical implication is to identify which path is limiting performance before changing TP degree. Small-message synchronization can call for a collective-library audit, while an offloaded cache requires attention to its storage path.
A decision framework for selecting TP degree given interconnect, workload shape, and model architecture
Selection should begin with the weight-fit floor at the chosen precision. The minimum GPU count is a constraint. Then the 40–50% VRAM reserve heuristic should be used to check whether there is enough cache headroom for the planned batch size and context length. If the implied weight share leaves too little room, the available responses are higher TP or reduced KV cache precision, including FP8 and NVFP4 where available.
NVLink supports TP up to 8 within the node, while the supplied PCIe-only rule stops at TP=2 before communication cost dominates. If the model needs to scale beyond the NVLink domain, the default is to keep TP within that domain and add PP across nodes. Model architecture further narrows the choice. Dense Llama deployments follow the standard TP calculation, while Mixtral deployments should default to a TP+EP hybrid that accounts for expert FFN execution.
The workload objective selects among the remaining candidates. Low TPOT or ITL favors higher TP within the NVLink domain, whereas moderate-rate throughput-per-dollar favors the lowest degree that satisfies cache headroom and permits more replicas. TTFT must be assessed separately because the supplied comparison does not support maximizing TP for that metric. If multi-node TP is unavoidable and communication overhead remains high, the next action is to audit NCCL in the small-message regime before reducing degree.
When high TP and lower-precision cache still cannot satisfy KV demand within GPU HBM, NVMe offload through ICMSP, Dynamo, and GPUDirect Storage extends the available design space. The relevant architecture spans HBM, DRAM, local and shared NVMe, and enterprise storage rather than treating SSD capacity as an isolated purchase. Its role is to make the required cache workable within the serving configuration, with storage read/write performance evaluated separately from all-reduce. A TP degree is operationally viable only when its synchronization path, cache capacity, and workload objective all align.


