DeepSeek-R1 MoE Routing and NVMe Bandwidth Requirements
Sparse expert routing requires keeping all weights accessible, not just active ones.

DeepSeek-R1 is routinely described through a single efficiency statistic: 671 billion total parameters, only 37 billion activated per token. That figure is accurate, and it explains why the model computes cheaply relative to its size. It says nothing about what the storage layer underneath it has to do. Sparse activation lowers the arithmetic cost of producing a token, but it does not shrink the set of weights the system needs to keep ready to reach. All 256 experts have to sit in addressable memory at once, because the router picks a different combination of experts for every token, and there is no way to issue a load after the fact for a decision that has already been made. A model built to minimize floating-point operations per token ends up maximizing the number of distinct weight tensors that must be reachable at any given moment, and that inversion is the premise the rest of this analysis rests on.
Fine-grained expert segmentation and shared-expert isolation in the routing pattern
Two architectural choices in the DeepSeekMoE design determine the shape of this problem, and together they make the resulting I/O pattern both wider and harder to anticipate than in conventional MoE systems. The first is fine-grained expert segmentation: rather than building a smaller number of large experts, DeepSeek-V2's technical report describes splitting each expert's feed-forward hidden dimension, producing many smaller experts within the same total parameter and compute budget. More, smaller experts means the router has a much larger combinatorial space to draw from. The number of distinct expert subsets that can be active across a batch grows exponentially as expert count rises, even though each expert's weights are cheaper to move once selected.
The second choice is shared-expert isolation. A fixed group of experts activates for every token, absorbing the common, general-purpose knowledge the model needs regardless of input, which frees the routed experts to specialize more narrowly, as the DeepSeek-V2 paper describes. That shared block is active on every forward pass, so it is a predictable, high-frequency target for any caching system. The eight routed experts selected per token are the opposite: their selection depends on token content, sequence position, and learned routing weights that shift from one token to the next. No fixed structural regularity exists for an infrastructure system to exploit unless it actually observes the routing decisions as they happen. This architecture trains efficiently because it balances load across many narrow specialists, but that same property is what makes its serving-time memory access pattern resistant to prediction.
How MLA compression affects the storage problem
Multi-head Latent Attention is often cited as the fix for DeepSeek's memory footprint, and it is a genuine advance, but it solves a different problem than the one expert residency creates. MLA caches a single compressed low-rank latent vector per token instead of full per-head key and value vectors, reconstructing the full KV set through an up-projection at attention time, as described in the DeepSeek-V2 technical report. DeepSeek-V2 reported roughly a 93 percent reduction in KV cache size relative to its comparable dense predecessor. In practice, DeepSeek-V3 runs at a per-token footprint far smaller than the 192 to 328 KB per token seen in other attention-based models of comparable scale, a sizable reduction against that class of model.
None of that touches the other side of the memory ledger. The 653 billion routed expert parameters still have to be resident in addressable memory regardless of how small the KV cache gets. MLA reduces one of the two large memory consumers in the system and leaves the larger one structurally unchanged. There is also a cost on the other side of the trade: Epoch AI's analysis notes that DeepSeek-V3 and R1 become arithmetic-bound rather than memory-bound during long-context decoding, because the up-projection step adds compute that scales with context length, with attention compute reaching parity with parameter multiply-accumulates around 5,000 tokens of past context, far earlier than dense models such as Llama 3 70B. MLA's memory savings arrive with a compute bill that grows the longer the context runs, and that bears directly on R1's reasoning workloads. R1 generates long chains of intermediate reasoning tokens before producing a final answer, and even compressed per-token KV entries accumulate into a large product once multiplied by trace length. The HuggingFace hardware discussion for DeepSeek-R1 notes that KV cache can exceed the memory footprint of the model weights themselves at extended context lengths and higher concurrency. MLA is real progress on one axis of the memory problem. It was never built to solve the other.

The memory hierarchy under full R1 deployment

Deploying the full R1 model forces a tiered memory architecture, and NVMe sits inside that hierarchy as an active component of serving, not a fallback reserved for operators who ran out of budget. At Q4_K_M quantization, the full 671 billion-parameter model needs roughly 376 GB of VRAM, the HuggingFace hardware requirements discussion states. That requires a multi-GPU cluster of A100 or H100 nodes, or equivalent cloud capacity. That configuration is the happy path: every expert's weights sit in VRAM, and no tier-crossing transfer happens during inference. It also requires hardware that most operators do not have standing idle for a single model.
Below that ceiling, a three-tier degradation path takes over. Full GPU offload keeps the entire model and its KV cache in VRAM and produces the fastest token generation, but it demands the cluster just described. Partial GPU offload splits layers between VRAM and system RAM, and every token that routes to an expert sitting in RAM pays a PCIe bandwidth cost on the way to compute. Below that tier, NVMe enters as a streaming participant that supplies weights throughout inference. When an expert's weights are not resident in VRAM or RAM, the router's activation decision triggers an I/O read that must finish before that expert can execute its share of the computation. That is a fundamentally different job than loading weights once at startup. The same sparsity pattern that keeps compute costs low works against the storage system here: because the router decides which 8 of 256 experts activate based on the content of the token itself, the set of NVMe reads required for the next step of generation cannot be known until that decision is made.
How expert parallelism distributes the routing problem across nodes
Expert parallelism is the production-scale answer to the VRAM ceiling: it spreads expert weights across many GPUs and nodes, so the aggregate pool, not any single device, holds the full model. That solution does not eliminate the underlying difficulty. It converts every routing decision into a data movement event across devices, and the interconnect fabric has to service that movement inside the token generation time budget. When a token's router selects an expert living on a remote GPU, the token's activations travel to that GPU, get processed, and return, a pattern commonly called dispatch and combine. DeepEP, DeepSeek's open expert-parallel communication library, is the reference implementation of that pattern.
Within a single node, NVLink and NVSwitch supply the bandwidth and low latency the routing traffic needs, and this is the preferred path in practice: the LMSYS and SGLang deployment on a 96-GPU H100 cluster achieved over 52,000 input tokens per second running on this fabric. Across nodes, the traffic depends on RDMA over InfiniBand or an equivalent high-speed network, because PCIe-only communication between nodes is too slow for the fine-grained, latency-sensitive routing R1 generates. Two-batch overlap, which hides cross-node communication latency by overlapping it with computation from a separate batch, raised throughput from roughly 12,900 to roughly 17,500 tokens per second in reported configurations, a result that shows both how large the communication overhead can be and how much of it can be reclaimed once hidden. A sufficiently large cluster with a well-built RDMA fabric can, in principle, remove NVMe from the hot path entirely by keeping every expert in aggregate VRAM across the cluster. That outcome depends on a GPU count and a network fabric that most operators cannot dedicate to a single model, which leaves NVMe offload as the realistic serving path for the majority of deployments running R1 today.
NVMe bandwidth and latency requirements imposed by the routing pattern
The requirements R1's routing pattern places on NVMe are governed by latency and access randomness, not by the peak sequential throughput figure that dominates most SSD datasheets. The access unit in this workload is a single expert tensor, and the eight experts selected per token are drawn from a pool of 256. Across a batch of concurrent requests, the union of activated experts grows, but it remains a shifting fraction of the full set, changing from one token to the next in a way no static prefetch schedule can fully anticipate. A sequential bandwidth rating of N gigabytes per second says nothing useful about a workload where the next computation step cannot begin until one specific small read completes. What matters here are 4K random read latency and queue-depth-one IOPS, and modern NVMe drives can vary by more than an order of magnitude on those numbers at low queue depths. That variance translates directly into variance in how long a token takes to generate.
GPUDirect Storage addresses a significant piece of this latency budget by giving NVMe a direct path into GPU memory that bypasses the CPU and system RAM entirely, removing two PCIe hops and their associated copy overhead from the critical path, as described in work on NVMe-direct KV-cache offloading for inference. At long context lengths, where GPU HBM cannot hold the full KV cache, a second I/O stream enters the picture: KV cache reads run concurrently with expert-weight reads, and the two compete for the same NVMe bandwidth. These streams behave differently. KV cache reads accumulate sequentially within a sequence as tokens generate, while expert-weight reads are random and content-dependent. A storage configuration tuned for one pattern will starve the other. WiSP's framing of MoE expert management (arXiv:2606.21868) treats this as a working-set paging problem, in the operating-system sense: the expert working set shifts with every token, eviction decisions carry latency consequences, and the page size, the expert weight tensor, is fixed by the architecture rather than chosen by whoever designs the storage system. Expert weight tensors in BF16 or FP8 are not random bytes but follow learned distributions with exploitable structure, so lossless, format-aware compression applied before data reaches the NVMe bus reduces the bytes moved per read, cutting latency and bandwidth consumption together without changing the logical access pattern the router imposes.
Design principles for speccing NVMe in an R1 serving stack
The access pattern traced through the previous sections leads to a specific set of configuration choices, and they depart from conventional storage guidance and from standard AI training storage advice. Evaluation should prioritize random read latency over sequential bandwidth: the benchmark that actually predicts serving performance is 4K random read at queue depth 1 and queue depth 4, not the rated sequential read speed most marketing material leads with. Topology matters as much as the drive itself: a direct-attached NVMe device on the GPU host's own PCIe bus removes a network hop that a shared storage system would introduce, and for latency-sensitive expert streaming, local NVMe is structurally better suited to the hot path than networked storage. GPUDirect Storage should be a requirement, not an optional tuning step, at production scale, because the CPU-bypass path from NVMe to GPU memory is what keeps NVMe latency inside the token generation budget; if you run without it, you accept avoidable overhead on every single expert-weight read. Speculative prefetch deserves real engineering investment even though the routing decision cannot be known in advance: patterns from a request's prior token, from sequence-level routing history, and from broader request context can inform a prefetch policy that warms the NVMe read queue before the router finalizes its choice, and the fact that this exact problem, forecasting data movement for large-scale MoE inference, earned a Best Paper Award at ISCA 2026 under the title "Patterns Behind Chaos" is a signal that the problem is both real and actively being worked. If you compress BF16 and FP8 expert weights before writing them to NVMe, you cut the effective bandwidth demand on reads, and because the read workload is the binding, random-access constraint, any proportional cut in transfer size gives you a proportional cut in read latency at the same device throughput. If the controller or device supports namespace isolation or independent I/O queues, run the KV cache stream and the expert-weight stream through separate paths, because if the sequential KV stream shares a queue with the low-latency random reads the expert-weight stream needs, it will degrade the one workload that can least afford it. Finally, caching policy for experts should be built around bandwidth consumed by misses rather than raw miss count: evicting a large expert that is rarely used costs less than evicting a smaller one that gets hit moderately often, and WiSP's working-set framing (arXiv:2606.21868) gives the right objective function for that trade-off, which plain LRU does not capture.
The MoE storage problem as context windows and model scale grow
The NVMe bandwidth demands traced through R1 are a structural consequence of fine-grained MoE design at scale, applying broadly beyond this one model, and both of the forces driving them, model scale and context length, are moving in directions that increase the pressure rather than relieve it. Fine-grained MoE is becoming a standard architectural choice across open-source models, extending well beyond DeepSeek. Infrastructure built to handle NVMe serving demands for R1 carries forward to the next generation of MoE models built the same way. The serving infrastructure needed to manage it is still young. Prefill-decode disaggregation, expert parallelism, speculative prefetch, and NVMe-direct KV offload are all active research areas with production implementations that are less than two years old, and the design space around them has not settled into a single accepted approach.
That immaturity is why storage engineering for LLM inference, NVMe selection, GPUDirect Storage integration, compression tuned for AI-native formats, bandwidth-aware caching policy, counts as a durable investment rather than a stopgap to be abandoned once GPU memory gets cheaper. GPU memory capacity keeps growing, but model scale and context length grow alongside it, and the gap between what fits in VRAM and what these models actually require has not closed across the MoE generation so far. Nothing in the architecture gives reason to expect that gap to close with the next one. The engineers who get the NVMe layer right for R1 are building infrastructure that will carry the MoE models that come after it.


