Multi-Node Inference Cluster Network Topology for AI Workloads

Network topology for LLM inference must optimize KV cache movement across tiered storage.

· 8 min read
Cover illustration for “Multi-Node Inference Cluster Network Topology for AI Workloads”
Distributed Serving · October 29, 2026 · 8 min read · 1,881 words

Multi-node inference cluster network topology must keep tensor-parallel collectives within NVLink domains while providing inter-node transport for pipeline stages and KV cache transfers. For NVMe and tiered storage for LLM inference throughput, the governing architecture places active KV blocks in GPU HBM and cold blocks in CPU DRAM or NVMe. Keeping latency and cost per token in check takes both storage capacity and the transfer bandwidth, compression and workload placement to make that hierarchy work.

Why inference clusters cannot simply inherit training network assumptions

Inference now makes up most of the ongoing hardware amortization and the cost of running the cluster. A mismatched network therefore imposes a recurring unit-economics penalty. Training networks primarily support synchronous, bulk, east-west collectives such as ring-AllReduce, with large static batches absorbing latency variance. Inference instead serves stochastic request arrivals under strict per-token P99 latency SLOs, while moving and retaining dynamic KV-cache state. Network stalls directly raise time-to-first-token or generation latency because there is no long batch operation to hide them.

This changes what the network must deliver, from predictable collective throughput to a service that keeps latency low when demand is irregular. Decode is constrained by both memory bandwidth and compute. Where training favors homogeneous accelerators performing dense matrix multiplication, inference can use heterogeneous hardware with bursty point-to-point transfers. A fabric that works well for training's main traffic pattern may struggle with those transfers. Its adequacy depends on where the serving engine places computation and how much state must cross each physical boundary.

How parallelism strategy determines which network boundary matters most

Tensor parallelism requires frequent All-Gather and Reduce-Scatter collectives, making the NVLink domain its natural placement boundary. These exchanges take place as tokens are generated. Intra-node NVLink provides hundreds of GB/s per direction, while inter-node Ethernet or InfiniBand provides a small fraction of that bandwidth. Matching parallelism to these network tiers therefore improves aggregate network utilization and is a prerequisite for high Model Flops Utilization in inference.

Consider a large model split across several nodes. If tensor-parallel shards cross ordinary inter-node links, their repeated collectives incur that slower boundary on every generated token. Pipeline parallelism instead divides model layers into stages and transfers activations at stage boundaries, reducing communication frequency to a pattern inter-node links can sustain. Data parallelism has looser coupling still because inference replicas route independent requests. The NVLink domain thus limits the tensor-parallel degree, while inter-node latency and bandwidth determine where pipeline and data parallel workloads can go.

The prefill/decode split as a topology-forcing function

Prefill and decode impose different demands within that hierarchy. Dedicated pools separate those demands. Colocation avoids an inter-node handoff but forces a continuing compute-memory compromise, whereas disaggregation introduces a transfer that must complete before generation begins. Neither arrangement removes the underlying resource constraint.

NVIDIA Dynamo, released GA on 12 March 2026, sits above vLLM and routes prefill and decode to separate GPU pools. It moves KV cache through NIXL, the NVIDIA Inference Xfer Library, which supports RDMA or TCP and is a primary transfer mechanism in both systems as of 2026. The handoff can be substantial. At long context lengths, one user's KV cache can reach tens of gigabytes, and concurrency multiplies the transfer demand across the pools. The fabric connecting the pools must do more than provide a generic east-west connection, since it has to keep transfers flowing with low latency and high throughput.

What storage architecture reduces LLM inference latency at scale?

KV cache grows linearly with context length and constrains concurrent users more directly than raw FLOPS. At long contexts, one user's cache can exceed a high-end GPU's entire HBM capacity. Cold blocks must then leave the GPU. Attention design reduces the footprint before tiering begins, with Multi-Query Attention sharing keys and values across all heads and Grouped-Query Attention sharing them within head groups. DeepSeek introduced Multi-Head Latent Attention, which uses linear projections of low-rank vectors to represent keys and values and cuts the storage needed per token further.

PagedAttention, introduced by Kwon et al. in 2023 and used in vLLM, addresses fragmentation through OS-inspired paging, achieving near-zero space waste within HBM. Efficient allocation does not eliminate capacity pressure. At scale, the resulting hierarchy uses GPU HBM for active KV blocks, CPU DRAM accessible through PCIe for overflow, and NVMe SSDs for the largest-capacity tier. HBM is vastly faster than DRAM, which is vastly faster than NVMe, so these tiers cannot be treated as interchangeable memory. NVIDIA's Inference Context Memory Storage Platform, announced at CES 2026, standardizes this hierarchy with BlueField-4 DPUs handling I/O scheduling to prevent storage waits from stalling the GPU.

Why lossless compression of KV cache tensors changes the economics of inter-node transfer

BF16 KV tensors contain statistical redundancy in their exponent fields that lossless compression can remove without changing the original values. The redundancy appears across dense, MoE and hybrid architectures, across calibration datasets and through nearly all transformer layers. It therefore offers an infrastructure lever for both inter-tier movement and the prefill-to-decode handoff.

SplitZip encodes frequent exponent values with fixed short codes and sends rare values to a small escape buffer, enabling GPU-friendly parallel decoding and bit-exact reconstruction. Speed is essential to the benefit. Existing lossless compressors can achieve high ratios yet operate too slowly for online inference, at tens-of-GBps or sub-GBps for CPU-based implementations, against multi-hundred GBps transfer requirements. SplitZip's authors report the highest encoding and decoding throughput among the lossless floating-point compressors they evaluated. Their SGLang integration over Mooncake transport delivers faster BF16 KV transfer, lower time-to-first-token and higher request throughput. Compression thus changes the bandwidth requirement for a throughput target, or permits more concurrency on the same fabric.

RDMA as the required transport layer for inference-scale KV movement

Compression cuts the bytes sent over the network, but moving them still has transport overhead. Traditional TCP/IP moves data through the OS networking stack with multiple memory copies and substantial CPU intervention, making it unsuitable for latency-sensitive KV handoff at inference scale. RDMA lets NICs transfer directly between memory regions on different nodes. GPUDirect RDMA extends that access to GPU HBM through compatible NICs. NIXL's RDMA path connects this hardware capability to the serving software's KV movement.

The OCP Open Cluster Designs Aligned AI Inference Fabric Reference Architecture, dated April 2026, specifies RoCE on a SONiC-based back-end, with a Switch CRD toggle activating PFC and ECN. Fabric behavior remains part of the requirement. Packet loss and retransmissions disrupt latency guarantees, while collective synchronization stalls translate into idle compute and slower tokens. Chelsio announced its seventh-generation AI Interconnect Platform on July 16, 2026, with native 400Gb Ethernet and Unified RDMA supporting iWARP and RoCEv2. In October 2026, Meta released its AI-focused MetaRoCE specification through the Open Compute Project, along with a reference software implementation and a test suite to check compliance. These developments give clusters more transport choices, but the network still has to be lossless and keep latency low under real traffic.

The two-profile topology model the OCP April 2026 Reference Architecture formalizes

The OCP reference architecture sets out two profiles that meet these requirements for 64–1,024 xPU clusters, both of which use SONiC-based switches and a Kubernetes-native control plane. The distinction follows workload placement. Minimal Inference converges scale-out and storage traffic without a back-end network, serving single-server models whose traffic flows north-south through a gateway. It fits deployments where tensor parallelism stays within the node's NVLink domain and no cross-host KV transfer is needed. Distributed Inference retains the converged frontend but adds a lossless RDMA back-end for cross-host pipeline parallelism or disaggregated serving.

Both profiles use a distributed x86 gateway with stateful ingress filtering and horizontal scaling. Tenant boundaries are explicit as well. Per-tenant VRFs and overlay-backed subnets provide isolation by default, with sharing permitted by policy. Connectivity maps, bills of materials and Kubernetes CRDs provide deployment artifacts, while the companion VLAB supports topology and overlay validation on virtual SONiC switches before physical deployment. Because switches are continuously reconciled from versioned intent, wiring, tenant overlays and external connectivity become artifacts compatible with version-controlled deployment workflows, reducing Day-2 operational drift rather than leaving the topology as an unmanaged diagram.

Topology-aware scheduling as the operational mechanism that enforces the hierarchy

A declared topology cannot protect latency if placement ignores it. The scheduler must distinguish NVLink domains, rack-level InfiniBand or RoCE switches, and cross-rack paths through spine switches. Bandwidth falls and latency rises across those boundaries. Tensor-parallel shards belong within one NVLink domain, while pipeline stages should share a rack switch to minimize activation-transfer delay. When cross-rack placement is unavoidable for very large models, stage boundaries should fall at layers with small activation tensors.

Process pinning must align GPU affinity with the correct NUMA domain and PCIe switch, because cross-socket memory traversals reduce the bandwidth available to the GPU. NVIDIA GB200 NVL72 illustrates a Multi-Node NVLink system whose domain spans multiple nodes. A scheduler that treats every node boundary as a tensor-parallel boundary would miss that extended domain.

What MLPerf Storage v3.0 benchmarks reveal about real inference storage throughput

MLPerf Storage v3.0, released in September 2026, adds KV cache and vector database workloads for the first time, together with a new S3 layer complementing POSIX. The storage tier is now directly benchmarkable for inference. One submission achieved single-node throughput of 14,579.22 tok/s in the KVCache inference scenario, providing a concrete reference for NVMe-backed KV storage. That result applies to the storage path tested, while performance on other NVMe paths can differ.

The same benchmark round measured peak write bandwidth for checkpointing across dozens of hosts, dozens of data nodes and hundreds of simulated accelerators. That workload is distinct from latency-sensitive KV retrieval. LLM, vision and video workloads occupy different NVMe bandwidth ranges, so a strong result in one does not substitute for inference-specific validation. GPU Direct Storage addresses the data path by moving storage data directly into GPU HBM through DMA, removing CPU RAM from the traditional storage-to-CPU-to-GPU route and cutting its two PCIe crossings. This configuration is necessary for the storage tiers to sustain the throughput described by the benchmark results.

Translating the topology argument into concrete cluster design decisions

For practitioners designing or auditing a cluster, the decision sequence begins with model fit within the physical NVLink domain. With Multi-Node NVLink, that check must use the domain's actual boundary. If the model fits and serving needs no cross-host KV exchange, the Minimal Inference profile is enough. If pipeline stages cross ordinary node boundaries, or prefill and decode use separate pools, the Distributed Inference profile supplies the required RDMA back-end. Strict throughput and P99 targets favor disaggregation when colocation's compute-memory compromise is unacceptable, but context length, concurrency and compression ratio must then determine the handoff bandwidth.

The next step is to decide how much KV state can stay in HBM at the maximum supported context length and user count. Capacity determines whether overflow tiers are required. Where NVMe is in the path, GPU Direct Storage and DPU-offloaded I/O scheduling are required configurations, while lossless BF16 compression provides a lever for reducing transfer demand without changing model fidelity. Fabric sizing must then account for KV transfers, pipeline activations or both, with PFC and ECN configured and scheduler placement enforcing NVLink and rack affinity. Storage sizing, fabric bandwidth and KV compression belong among the cluster's first design inputs because inadequate choices can put its latency and throughput targets out of reach.

Sources

  1. Date: April 2026 OPEN CLUSTER DESIGNS ALIGNED AI INFERENCE FABRIC REFERENCE

More in Distributed Serving