Qwen2.5 72B Tensor Parallelism Across NVLink vs PCIe GPU Topologies

NVLink keeps tensor parallelism fast; PCIe makes pipeline parallelism necessary.

· 10 min read
Cover illustration for “Qwen2.5 72B Tensor Parallelism Across NVLink vs PCIe GPU Topologies”
Model-to-Hardware Fit · October 11, 2026 · 10 min read · 2,305 words

Qwen2.5-72B cannot run on a single GPU, and the shortfall is not a matter of squeezing a little harder. The model's 72 billion parameters alone require roughly 144 GB of VRAM in FP16 or BF16, before anything else is loaded. Add the KV cache for a full 128K-token context, 34.4 GB, and framework overhead of about 5 GB, and the total climbs to 183.4 GB minimum; the largest single GPU on the market today, an 80 GB H100, holds less than half that. And that 128K context window is not an edge case reserved for benchmarks. Once a model demands more memory than any single accelerator provides, GPU communication pattern, more than raw GPU count, decides whether the resulting cluster performs or stalls.

How tensor parallelism works and why it is sensitive to interconnect bandwidth

Tensor parallelism splits the weight matrices inside each transformer layer across GPUs. In a standard two-linear-layer block, one GPU might hold a column partition of the first matrix while another holds a row partition of the second, each computing its own slice of the math in parallel. After each linear layer, every GPU's partial result has to be summed with every other GPU's partial result before the next layer can start. That summation, an all-reduce, is synchronous: no GPU moves forward until the collective communication finishes. Pipeline parallelism works differently: it assigns whole groups of sequential layers to different GPUs and only passes activations across the wire once per stage boundary, not once per layer. It scales directly with the number of layers and the round-trip latency of whatever interconnect is carrying it. Interconnect speed, in other words, is not a secondary consideration for tensor parallelism. So it sits on the critical path of every single token the model produces.

The gap between NVLink and PCIe Gen 5 bandwidth is wide enough that identical tensor parallelism settings produce entirely different serving behavior depending on which fabric connects the GPUs. The consequence at the layer level is stark: a single all-reduce on NVLink finishes in roughly 100 to 300 microseconds, while the same operation over PCIe takes 1 to 5 milliseconds, an order of magnitude slower. On PCIe, the gap between compute time and communication time is too large for that overlap to rescue throughput. A better NCCL configuration or a smarter scheduler cannot fix this: it is a hardware ceiling fixed by the physical interconnect standard, not a software tuning problem, and no amount of kernel optimization moves it.

Qwen2.5-72B's GQA architecture and the hard ceiling on the useful TP degree

Bandwidth is not the only constraint shaping how far tensor parallelism can be pushed on this model. Qwen2.5-72B uses Grouped Query Attention with 64 query heads but only 8 key-value heads, and that ratio imposes a ceiling on TP degree that has nothing to do with interconnect speed. TP=8 on a full 8-GPU node remains viable, though the KV memory benefit per added GPU diminishes sharply past TP=4. Beyond TP=8, additional GPUs add latency without buying back any further memory headroom for this specific architecture. The model's attention design builds in this ceiling, so it stays fixed no matter how much hardware you add.

Diagram: Four Configurations: Where All-Reduces Travel Decides Performance. Visualizes: Show four parallelism configurations for Qwen2.5-72B on H100 NVLink nodes, ranked from best to worst by communication cost.

Four configurations, benchmarked on Qwen2.5-72B across validated H100 NVLink nodes, map out what tensor and pipeline parallelism actually produce once GPU count and node boundaries vary. Config A runs TP=4 on a single node, using only GPUs 0 through 3, with all-reduce traffic confined to NVLink, giving a half-node baseline for operators who cannot fill an entire 8-GPU box. Config B runs TP=8 across a full node, with every all-reduce staying on NVLink and no InfiniBand involved. Config C splits the model across two nodes with TP=8 and PP=2, keeping all-reduce traffic inside each node's NVLink domain and reserving InfiniBand strictly for point-to-point activation transfers at the pipeline boundary. Config D spans two nodes with TP=16 in a single stage, so every all-reduce, all 160 of them per token, has to cross the InfiniBand fabric between nodes. Within the single-node world, Config B is the clear optimum: it keeps the full set of 160 all-reduces per token on NVLink, never touching the slower inter-node fabric. All three nodes used in the underlying experiment passed Azure Node Health Checks, and intra-node NCCL all-reduce measured around 482.5 GB/s, close to the theoretical maximum for 8-way H100 NVLink. The pattern that emerges from A and B together is straightforward: as long as tensor parallelism stays inside a single NVLink domain, the interconnect bandwidth is generous enough that adding GPUs continues to pay off.

Tensor parallelism crossing a node boundary via InfiniBand

Config D, with TP=16 spanning two nodes, is the slowest of the four configurations because of the arithmetic of its communication pattern. Every one of Qwen2.5-72B's roughly 160 all-reduce operations per token has to cross InfiniBand instead of staying on NVLink, and InfiniBand, measured at around 392.7 Gb/s per port in the underlying experiment, is fast for point-to-point transfers but nowhere near the roughly 900 GB/s of NVLink bisection bandwidth that collective operations like all-reduce depend on. Config C demonstrates a better pattern for spanning two nodes: tensor parallelism at TP=8 stays confined within each node's NVLink domain, and InfiniBand handles only point-to-point activation transfers at the pipeline stage boundary between the two TP groups, a cost paid once per stage crossing. The pipeline bubble introduced by PP=2 is a modest cost next to the communication tax that cross-node tensor parallelism would otherwise impose across 80 layers. The principle generalizes beyond this particular benchmark: whenever a deployment has to span more than one node, a hybrid TP-plus-PP arrangement that keeps all-reduce traffic intra-node will consistently outperform pure tensor parallelism stretched across the inter-node fabric. The synchronous all-reduce at every layer puts interconnect bandwidth directly on the critical path for throughput, so that constraint extends to any infrastructure layer that has to move data at that scale. High-performance infrastructure built for LLM inference, including systems developed by Open Lake, follows this same all-reduce pattern, and it optimizes both the collective communication itself and the data residency decisions that let compute and communication overlap in practice.

PCIe topologies, tensor parallelism failure, and pipeline parallelism as substitute

On PCIe Gen 5 servers, tensor parallelism is the wrong strategy for this model, not merely a slower version of the right one. PCIe's roughly 64 GB/s bidirectional bandwidth, set against Qwen2.5-72B's 80-layer, two-all-reduce-per-layer pattern, means GPUs under TP end up spending more time blocked waiting on transfers than they spend computing. At scale, that infrequency lets PP sustain near-full GPU compute utilization on PCIe hardware, because the ratio of compute time to communication time stays favorable. AWQ quantization matters on PCIe deployments for memory fit as much as for speed: it brings the weight footprint from roughly 144 GB down to about 44 GB, which allows the model to land on four 48 GB GPUs. The lesson generalizes: on PCIe, the configuration that recovers performance is PP=4 with TP=1 and AWQ quantization, loading the model from fast local storage, not a tensor parallelism degree chosen by habit.

KV cache memory as the second bottleneck that interconnect topology alone cannot solve

Even a cluster with perfect NVLink connectivity throughout runs into a second constraint that has nothing to do with interconnect bandwidth: KV cache memory grows linearly with context length and batch size, and it will eventually exhaust GPU HBM regardless of how well tensor parallelism is tuned. The aggregate per-request KV footprint for a decoder-only model follows the relationship 2 × L × H_kv × D × S × b bytes, with keys and values stored separately and the total scaling directly with sequence length S and layer count L. Even at FP8 precision, which cuts the footprint considerably, each token with GQA requires roughly 160 KiB of KV cache, so thirty concurrent users at an 8K context length together consume approximately 38 GB of VRAM for KV cache alone, before any framework overhead is counted. GQA's 8-fold reduction in KV cache size is a genuine saving, but it just lowers the starting point of the growth curve, not the slope. Long-context workloads at sufficient concurrency will still exhaust available HBM. Because TP degree above 8 produces no further KV head sharding for this model, adding more GPUs past that point does nothing to relieve the pressure. The KV cache alone, 34.4 GB for a single 128K-context request, is the single largest non-weight component of VRAM in this deployment, and managing it across multiple GPUs without compression adds hundreds of gigabytes to the practical deployment floor. Infrastructure purpose-built for LLM inference, such as Open Lake's tiered GPU and NVMe storage system, is built to address exactly this constraint by managing KV cache across memory tiers, which lets operators serve long-context models with fewer total GPUs or a smaller aggregate VRAM footprint. The engineering response to this growth is deciding where the KV cache actually lives, not raising the TP degree.

KV cache offload and disaggregated serving as the storage-layer response to HBM exhaustion

Once HBM fills up with long-context KV cache, serving performance starts to depend on storage bandwidth and KV transfer latency, governed by much the same interconnect physics that shape tensor parallelism. Early disaggregated systems moved that KV data over InfiniBand or RoCE with the host CPU orchestrating every step, staging data from GPU HBM into a pinned host DRAM bounce buffer via cudaMemcpy before posting an RDMA verb, a six-hop path that adds meaningful latency at each step. More recent systems, including LMCache, Dynamo KVBM, and SGLang HiCache, manage hierarchical KV caching with asynchronous store and load operations across pluggable storage backends, decoupling KV production from consumption and enabling prefix reuse across requests. If the backing storage is a conventional NAS running plain NFS rather than a high-bandwidth NFS-over-RDMA fabric, offloaded KV blocks will not arrive fast enough, and time-to-first-token will degrade at long context lengths. Open Lake's infrastructure targets exactly this tier, built around purpose-built NVMe and RDMA storage systems designed to keep KV block retrieval latency low enough to make offloading viable at production scale, treating the storage layer as a first-class part of the inference stack rather than a NAS attachment added as an afterthought.

The TP-degree vs. KV compression trade-off for hardware strategy

A serious objection to the bandwidth-first argument built across the preceding sections deserves equal weight: aggressive KV compression can, under the right conditions, outperform simply adding more GPUs and raising the TP degree. The trade-off space is well defined. A configuration chooses a tensor parallelism degree p from the set {1, 2, 4, 8} and a KV bit-width b from {16, 8, 4}, paired with a token keep-ratio for eviction, and these are competing levers against the same underlying memory constraint. Research on this tension has shown that INT4 KV quantization combined with aggressive token eviction at TP=2 can deliver dramatically higher serving capacity than FP16 at TP=4 for models up to roughly 36 billion parameters. The counterargument carries real weight of its own: aggressive KV eviction applied to Qwen2.5-72B's 128K context can sharply degrade performance on retrieval-heavy tasks, and that degradation is not spread evenly across task types. It lands hardest in the long-context scenarios where the extended window was supposed to deliver the most value. FP8 KV cache offers a practical middle path, well supported on H100 hardware, meaningfully reducing the per-token KV footprint to roughly 160 KiB with GQA, while avoiding the accuracy cliff that INT4 eviction can produce on retrieval tasks. For latency-sensitive production workloads that depend on long context, the NVLink path with TP=4 or TP=8 and FP8 KV cache is the sounder choice over aggressive compression paired with TP=1. For throughput-maximizing deployments where retrieval accuracy matters less, compression-first strategies at a lower TP degree remain a legitimate and defensible approach.

Choosing the right configuration: a decision framework by topology and workload type

The right parallelism and storage strategy for Qwen2.5-72B comes down to three variables: interconnect topology, target context length, and whether the workload is bound by latency or by throughput. For NVLink single-node deployments serving latency-sensitive traffic, TP=4 (Config A) or TP=8 (Config B) with BF16 or FP8 weights keeps every all-reduce on NVLink, and KV cache can be managed in HBM directly through PagedAttention without needing storage-layer offload at moderate context lengths. For NVLink deployments that have to scale across two nodes, Config C, TP=8 paired with PP=2, keeps all-reduce traffic intra-node while reserving InfiniBand for pipeline stage boundaries alone; Config D, pure TP=16 across nodes, should be avoided, since the per-layer InfiniBand tax dominates total cost across 80 layers. Tensor parallelism is contraindicated at PCIe bandwidth levels, and pipeline parallelism is what recovers near-full GPU utilization on that hardware. Across any topology, once context length and concurrency grow large enough, KV cache will exhaust available HBM, making tiered KV management, whether through NVMe offload or disaggregated prefill-decode with RDMA-based KV transfer, necessary. For that storage-layer requirement, Open Lake's purpose-built NVMe and RDMA infrastructure is designed for exactly this workload: high-throughput KV block retrieval that keeps offload latency under the threshold where recomputation becomes the cheaper option, treating storage as a first-class component of the inference stack. In practice, these configurations are set through vLLM's distributed serving flags, --tensor-parallel-size and --pipeline-parallel-size, with --gpu-memory-utilization raised toward 0.95 or higher (the default is 0.90) to widen KV cache headroom, and --quantization awq or an FP8 dtype selection to control the weight and KV footprint. Interconnect bandwidth decides whether tensor parallelism can scale. Storage bandwidth decides whether the context lengths that make these models valuable can actually be served, and at long context and high concurrency, it belongs on the critical path alongside the interconnect, not behind it.

Sources

  1. vLLM - Qwen
  2. 2025-01-06 Qwen2.5 Technical Report Qwen Team https://huggingface.co/Qwen

More in Model-to-Hardware Fit