Load Balancing Strategies for Heterogeneous Multi-Node LLM Serving Fleets
Choosing the model and the instance together beats splitting the decision.

NVMe and tiered storage improve LLM inference throughput when they preserve reusable KV cache and avoid repeated prefill without making retrieval the bottleneck. Heterogeneous fleets also need scheduling that chooses the model and instance together. The effective architecture joins fused scheduling with fast KV transfer and lossless storage across GPU memory, CPU memory, NVMe, and object storage.
Why heterogeneous LLM serving fleets split scheduling into two layers
Production fleets span a wide range of model sizes across mixed GPU generations. They face two distinct scheduling decisions. A learned model router inspects each prompt and selects a model using predicted quality or token cost, while a downstream dispatcher chooses a replica of that model. Collecting fresh cluster-wide state quickly enough to combine those decisions is difficult at distributed scale, so separating them is deliberate.
RouteBalance identifies this boundary in vLLM’s Semantic Router, which delegates instance balancing to Envoy through round-robin and places per-instance scheduling outside its scope. The same split is used in llm-d, Ray Serve, and the production stacks of major model providers. Light traffic makes that separation look effective. Each layer can optimize its own objective because spare capacity absorbs decisions made without the other layer’s information.
How the two layers produce conflicting objectives under real load
Heavy traffic removes that margin for error. A quality-oriented router keeps choosing a large model even as its queues grow, while a cost-oriented router concentrates requests on smaller models and leaves capable multi-GPU instances underused. RouteBalance documents the former pattern inflating mean end-to-end latency by 2.4 times under high load. The dispatcher cannot repair either outcome because it can choose only among replicas of the selected model.
When quality gaps are normalized independently for each request to [0,1], model selection becomes insensitive to queue depth, and the large model’s traffic share stays flat across load levels. Meanwhile, round-robin takes no account of how requests differ in response length and KV memory demand. Being simple does not guarantee that it will perform well under those conditions.
Better runtime visibility alone is also insufficient. Astrolabe explains that variable response lengths and Paged Attention’s dynamic memory allocation make latency, throughput, and memory poor proxies for a request’s actual end-to-end load. Even a dispatcher that sees every replica can be misled by those measures of load. Making either layer better on its own leaves the other’s limits in place, because the boundary between them blocks corrections that would send requests to another model.
Why KV cache memory pressure makes the layer boundary untenable at long context
KV cache turns this coordination problem into a physical limit. Its size grows linearly with sequence length and concurrency, so a single long-context user on an H100 serving a large model can consume a large fraction of available HBM. Only a few concurrent users can exhaust the card. Queueing or preemption then follows regardless of the router’s quality preference, and discarding KV values imposes additional GPU work when they must be recomputed.
Prefill-decode disaggregation addresses a related mismatch in resource demand. Prefill is compute-intensive, whereas decode depends heavily on memory bandwidth, so combining them wastes whichever resource is not limiting progress. SGLang separates these worker pools and transfers KV through NCCL or NIXL. That separation makes cache location and movement explicit scheduling concerns.
SMETRIC, from SJTU and Alibaba Group, uses production traces from the Bailian cluster to show why cache awareness becomes especially valuable in agentic serving. Most tokens in these workloads are served from cache. A dispatcher that ignores cached-block locations therefore forfeits substantial reuse even when its load estimate is sound. The field’s first response was to expose KV state within the existing scheduling hierarchy.
KV-cache-aware routing as the first partial fix and where it stalls
Ray Serve LLM’s alpha-stage KVAwareRouter balances KV overlap against token load. Consistent-hashing session affinity already supports reuse across multi-turn conversations, but affinity alone can concentrate traffic. Combining reuse with load improves time to first token, time per output token, and throughput for heterogeneous workloads. Cache locality helps, but cannot be the only objective.
SMETRIC addresses this tension with a two-tier KV store comprising local GPU memory and global CPU memory. A local miss can retrieve cached state globally, avoiding full prefill recomputation. Its session-centric scheduler balances first-turn requests, then sends follow-ups to the instance with the highest local KV hit rate. If that instance cannot serve within the SLO, the session moves to the least-loaded instance. The global tier cuts the cost of moving that session.
Accuracy requires a separate qualification about how offloading preserves context. Low-rank key projection and unreliable landmark approaches degrade precision on text-to-structured-data tasks across Llama 3 and Qwen 3. Alongside storage speed, it matters whether retrieval keeps the context exact or uses methods that approximate it. Even exact, fully KV-aware dispatch is still limited by the model chosen upstream, so removing that limit requires making both decisions together.
Prediction-guided one-shot dispatch as a middle path before full fusion
Astrolabe improves dispatch without first replacing the model router. It combines response-length estimation, per-instance simulation-based latency prediction, and a power-of-two choices policy to place requests before imbalance develops. Randomization has a specific purpose here. Deterministic selection of the apparent best instance can send nearby correlated requests to the same destination, whereas sampling two candidates breaks that herding pattern.
The scheduler queries only those two instances’ predictors rather than querying every predictor for every request. Astrolabe reports approximately a 2.8-fold reduction in per-predictor CPU use relative to full fanout. On its Llama-2 / ShareGPT benchmark, it matches the best load-aware baseline’s SLO capacity while lowering P99 time to first token by 16–77% and reducing preemptions at capacity. These gains do not depend on live migration.
Astrolabe also reports a separate migration-enabled A100 comparison that outperforms Llumnix by up to 2.6× in throughput, with orders-of-magnitude lower per-token latency at saturation. The comparison supports prediction-guided placement without making migration a prerequisite. Its remaining architectural limitation is that it cannot redirect a request to a less-loaded model of equivalent quality once the router has committed.
Fused joint assignment: how RouteBalance collapses the two layers into one decision
RouteBalance removes that commitment point by assigning requests across all candidate model-instance pairs. Wei Da and Evangelia Kalyvianaki of the University of Cambridge describe the scheduler in a paper submitted June 2026 and revised October 2026. It holds incoming requests in a buffer for short scheduling rounds that adapt to load, then weighs quality, cost, and estimated latency together.
Round-wide quality normalization is the main way it does this. Per-request normalization gives each request its own quality scale, which can keep the large model ahead even when it is congested. As load increases, rounds grow and cover a broader quality range, shrinking small quality gaps relative to the round’s scale and allowing latency estimates to exert more influence. The large model’s share can then decrease without a separate explicit load signal in the router.
Each round uses one batched k-NN search to estimate the quality and output length of the candidate models. It also takes one snapshot of all instance loads, while local dead reckoning maintains exact in-flight counts between snapshots. Requests are assigned longest-first to the model-instance pair with the highest joint weighted score. The inference engines remain unchanged. Standard vLLM continuous batching runs beneath the fused scheduling layer.
A single weight vector exposes different quality–cost–latency operating points without code changes. RouteBalance evaluates this approach on a 13-instance, 28-GPU cluster using A30, V100, and A100 GPUs with Qwen2.5 models from 3B to 72B. It reports results at least as good on quality, latency, and cost, and better on at least one of those measures, than the overwhelming majority of baseline configurations. The joint decision takes roughly 32 ms at a representative request rate, and assigned requests dispatch immediately, before the round closes.
What storage architecture reduces LLM inference latency at scale through KV transfer and offloading
A correct assignment can still stall on data movement. In disaggregated or tiered serving, the storage architecture must retrieve reusable KV quickly enough to avoid replacing prefill work with retrieval delay. GPU memory holds active state, while CPU memory, NVMe, and object storage extend the capacity available for reuse. NVMe tiering improves the economics of serving when cache hits avoid repeated GPU computation, but its cost-per-token benefit depends on the retrieval path.
NVIDIA NIXL, announced March 2025, unifies RDMA/GPUDirect data movement across GPU memory, CPU memory, NVMe, and cloud object stores, including S3 and Azure Blob. vLLM v0.8+ with the V1 engine supports disaggregated prefilling through NixlConnector, linking a prefill KV producer to a decode consumer through NIXL backends including RDMA, UCX, and LIBFABRIC. Small transfers are especially sensitive to this path. RDMA Write has a structural latency advantage over the CUDA copy engine for small chunks, which suits frequent incremental KV movement.
GPUDirect Storage addresses a different part of the path by providing direct DMA between NVMe SSDs and GPU VRAM. It bypasses CPU-managed bounce buffers and their added copying overhead. vLLM’s contiguous-block KV layout, upstreamed in late 2025, complements that transport by placing all layers’ KV data for one logical block into one physical block. Efficient transfer depends on layout as well as bandwidth.
LMCache’s disk backend moves cold KV blocks to NVMe so hits can skip prefill, while ObjectCache, introduced in May 2026, adapts S3-compatible object storage for runtime KV reuse. NVIDIA’s ICMSP, announced at CES 2026, uses BlueField-4 DPUs to manage KV movement off the GPU compute path. MLPerf Storage v3.0, with results announced September 1, 2026, adds a KV Cache benchmark for inference-cache reads and writes, providing a workload-specific basis for comparing these storage paths.
Lossless compression as the missing piece between scheduling correctness and transfer efficiency
Fast transport still has to move the bytes. SplitZip, introduced in May 2026, reduces that volume by exploiting redundancy in the BF16 exponent field while preserving sign and mantissa exactly. Its compression and decompression throughput are high enough to avoid becoming the KV transfer bottleneck in the reported setting. Decompression is faster than compression, which also suits offloading’s write-once, read-many cache-hit pattern.
The precision failures associated with low-rank projection and landmark methods arise from approximating context. Lossless compression avoids that class of failure while reducing traffic over RDMA links and writes to NVMe. When output quality is a service-level requirement, saving bandwidth is no help if application-level accuracy workarounds wipe out the latency gain.
What operators need to change to move from two-layer to fused scheduling today
Operators can retain the serving stack while selectively changing the scheduling, transfer, and storage layers. Teams ready for fusion can deploy RouteBalance above unmodified vLLM backends and configure its weight preset. Teams retaining the hierarchy can use KVAwareRouter or SMETRIC’s session-centric approach to improve locality and balance. Prediction-guided dispatch gives teams another step they can take while keeping their model-selection policy.
For disaggregated serving, operators can use vLLM’s NixlConnector or SGLang’s separate prefill and decode pools. NIXL with RDMA/GPUDirect backends supports fast KV movement on InfiniBand or RoCE without serving-engine code changes. llm-d v0.9, released in 2026, adds GPU-utilization-aware routing, router high availability, bounded flow-control defaults, event-driven autoscaling foundations, and end-to-end tracing. Those controls make the distributed fleet more manageable, but do not themselves eliminate the model-instance decision boundary.
Storage validation should precede further scheduling optimization when retrieval may be the bottleneck. Operators should use the public MLPerf KV Cache benchmark to assess the storage layer against inference-cache reads and writes rather than infer suitability from generic storage performance. The migration can remain incremental. Open-source scheduling and transfer components, tiered KV storage, and public benchmarks provide a practical path toward joint assignment without assuming that any single tool resolves the whole system.
Sources
- RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving
- [2606.17949] RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving
- Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Scheduling
- RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving
- SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling


