Mistral 7B and Mixtral 8x7B GPU Memory Footprint Comparison at Scale
Mixtral's sparse activation saves compute but demands full expert weights in VRAM.

Mixtral 8x7B's architecture makes two promises that pull in opposite directions: compute cost that tracks its active parameters, and memory cost that does not. The compute savings from sparse activation reduce active-parameter cost, but using that savings as a proxy for total VRAM demand means budgeting against the wrong number.
The trap sits in what the mixture-of-experts design actually guarantees. Near-large-model output quality at small-model compute cost is a genuine architectural property of Mixtral 8x7B, but it says nothing about how much VRAM the model needs sitting in a GPU. Teams that size hardware off active-parameter count alone end up underprovisioned, because the router can send any token to any expert, and nothing in that routing logic shrinks the pool of experts that has to stay loaded. Three layers of memory pressure produce this gap, each behaving on its own terms: static weight storage, the per-request growth of the KV cache, and the VRAM spikes that batch inference produces. Treating those three as one undifferentiated "memory cost" produces a hardware budget that is wrong in ways visible only once the deployment is live.
Weight Memory: Mistral 7B vs. Mixtral 8x7B
Mistral 7B is a dense model, so its weight memory is easy to reason about: VRAM scales with parameter count in a straight line. At BF16 full precision, that is around 14 GB, and if you quantize it, the number drops substantially from there. Planning for Mistral 7B at the weight layer is close to arithmetic.
Mixtral 8x7B breaks that straight line. The model runs eight separate feed-forward expert sets, each shaped like the FFN in Mistral 7B (roughly 5.25 billion parameters per expert FFN, not a full 7 billion), and all eight have to sit in VRAM at load time even though only two activate on any given forward pass. The router's selectivity happens after the hardware already paid for the whole set. At BF16, that full expert set pushes weight-only VRAM up toward roughly 90 GB, and no single high-end consumer GPU can hold that much on its own.
Quantization narrows the gap but doesn't close it. Q4 brings Mixtral 8x7B down into a range you can spread across two or three consumer GPUs, and Q8 roughly halves the full-precision requirement. None of that quantization, though, brings the weight floor down to anything resembling "7B-class," because the full set of eight experts is still resident regardless of bit width. And weight memory is only the floor. KV cache and runtime overhead stack on top of it, and those are the layers that cause the real planning failures.
Why the KV cache grows the same way for both models regardless of expert sparsity
The assumption that trips up most planners is that sparse activation ought to extend some kind of efficiency advantage into runtime memory the way it does into compute, but it doesn't. Every token in every request writes key-value pairs across all attention layers, and it does this no matter which experts the router picked for that token.
The mechanics explain why. During autoregressive decoding, the serving system has to hold onto the attention keys and values for every token already generated, so it never has to recompute them at each step. That retention has nothing to do with the MoE router: it's a function of attention itself, not of which feed-forward expert processed a given token. As concurrent requests and sequence length grow, the KV cache turns into one of the largest consumers of VRAM in the whole deployment, scaling with sequence length and the number of attention heads in direct proportion.
Mixtral 8x7B and Mistral 7B share the same attention configuration (32 layers, 32 query heads, 8 KV heads), so the KV cache cost per token is about the same for both models. Mixtral's larger context window changes the practical outcome: it accumulates more KV state over long sequences because it holds a longer sequence. A single long-context request against Mixtral 8x7B can eat a significant fraction of available VRAM in KV state alone, and the MoE architecture gives you no structural way around that cost.
This is where weight-only provisioning estimates fall apart. If a team sizes hardware purely off the static weight footprint, it finds out, usually under load, that KV cache is a second and independent budget that keeps growing with every concurrent session and every added token of context, and it can consume whatever headroom the weight-only math assumed was free.
How batch inference creates VRAM spikes that neither weight nor KV cache estimates predict
Single-request decoding keeps the router's behavior simple: two experts per token, predictable and bounded. Batch inference breaks that simplicity. When a batch holds many concurrent requests, many tokens get processed at once, and each one routes on its own to its own top-2 experts. If the batch is big enough, nearly all eight experts end up activated somewhere in it, even though the per-token rule never changed.
This doesn't inflate the weight footprint, because you already have all eight experts resident in VRAM regardless of batch size. What changes is compute footprint and latency: those shift unpredictably as overlapping expert sets collide during matrix operations, and that produces memory pressure during the computation itself, which a weight-only estimate never accounts for. The result at modest batch sizes: many or all eight experts active simultaneously in a single forward pass, eroding the compute-efficiency advantage sparsity was supposed to deliver, while the underlying weight memory cost stays fixed.
Mistral 7B has no equivalent failure mode. Its compute footprint scales linearly and predictably with batch size, and whatever gets measured at batch size 1 scales proportionally as load increases. Mixtral 8x7B offers no such guarantee. A deployment sized against measured single-request VRAM can spike into out-of-memory territory under production traffic, with no change to the model weights and no change to per-request context length. The only way to catch this before it happens in production is to run load tests at realistic concurrency and measure peak VRAM directly, because static estimates, whether from weights or from KV cache projections, cannot predict a spike that only exists once real concurrent traffic hits the router.

GPU Memory Hierarchies and Expert Swapping
The obvious fix looks simple on paper: only two experts run per token, so hold the other six somewhere cheaper, like CPU RAM, and page them into GPU memory as the router calls for them. The idea fails on contact with interactive latency requirements.
Experts can change from one token to the next, so keeping response times usable means holding all experts in GPU memory, unless the deployment is willing to stall on a model swap between tokens. The PCIe bandwidth connecting CPU RAM to the GPU is the binding constraint, and LLM inference is already memory-bound work. Autoregressive decoding makes the problem worse still: every token you generate needs a full pass through the resident weights, so any paging delay you introduce along the way compounds with every subsequent token instead of staying a one-time cost.
That's why low-bit weight-only quantization does more real work in production than offloading schemes: it cuts HBM traffic directly, without introducing any paging latency, and it does so while keeping every expert resident. Research systems like FlashMoE, which uses SSD and NVMe offloading paired with ML-based caching that blends recency and frequency signals, and MoE-Lightning, which runs a CPU-GPU-I/O pipelining schedule called CGOPipe alongside a Hierarchical Roofline Model for throughput-optimized policy selection, show that offloading can work. But you build both as purpose-built infrastructure engineering projects, not settings a team flips on in a config file. For production planning, you want full expert residency in GPU memory as the safe baseline. Anything less demands custom serving infrastructure and latency benchmarking rigorous enough to prove the tradeoff is worth it.
KV Cache Quantization and Architectural Tuning
KV cache pressure doesn't respond to anything that touches expert sparsity, so cutting it down takes a separate set of tools aimed directly at the cache itself: quantizing KV tensors to FP8 or INT8, pruning attention heads, and using prefix caching to avoid recomputing KV state that requests already share.
Quantizing the KV cache is the most direct lever available. Storing keys and values in FP8 instead of BF16 halves the per-token KV footprint, with accuracy loss that stays manageable across most workloads, and that matters because KV cache, not weights, is frequently the actual constraint once a system is under load. BF16 and FP8 are AI-native numeric formats, and compression strategies built specifically for their statistical distributions outperform general-purpose compression tools, which weren't designed with these formats in mind and don't deliver comparable savings.
Prefix caching attacks the problem from a different angle. When several requests share a common prefix, a system prompt or a shared document context, the KV state for that prefix only needs to be computed once and then reused across every request that shares it. That single pattern addresses one of the largest sources of KV cache growth in production RAG and chat workloads. Open Lake's cache management layer shows this working at production scale: prefix KV entries get shared across dozens of concurrent requests, with residency managed across GPU HBM and local NVMe tiers, and a Mixtral-8x22B entry serving a large context across NVMe illustrates how tiered residency stretches effective cache capacity well past what GPU memory alone could hold. Attention head pruning and grouped-query attention cut the KV footprint at a structural level, and planners should confirm whether the specific Mixtral checkpoint they're deploying uses GQA before finalizing any memory estimate, since that detail changes the math directly.
RDMA, NVLink, and Multi-GPU Mixtral Deployments
Once Mixtral 8x7B's weight footprint pushes past what a single GPU can hold, the interconnect linking the GPUs together decides whether the cluster behaves like one coherent memory pool or like a slow, fragmented distributed system.
Within a single node, NVLink lets GPUs read each other's memory at bandwidth far above PCIe, though still well under native HBM speeds (roughly a quarter of HBM bandwidth). That's still enough to let expert weights on one GPU get accessed by another GPU without running into the same PCIe bottleneck that makes CPU offloading slow. Across nodes, RDMA lets KV cache state and weight shards move between servers without routing through the CPU, which makes KV cache sharing and tensor parallelism possible at network speeds rather than at disk or PCIe speeds.
Without interconnects fast enough to support this, tensor parallelism for a mixture-of-experts model turns punishing fast. Expert routing introduces all-to-all communication patterns, and those patterns amplify the cost of a slow interconnect far more severely than they would for a dense model like Mistral 7B. Infrastructure planners should weigh interconnect bandwidth just as heavily as raw VRAM capacity when they plan. A cluster with plenty of GPU memory but a weak interconnect can end up delivering Mixtral throughput no better than a smaller GPU pool wired together with fast interconnects.
A practical memory provisioning framework for Mistral 7B and Mixtral 8x7B at production scale
Provisioning Mixtral 8x7B correctly means budgeting three memory pools on their own terms, weights, KV cache, and batch headroom, and then picking serving infrastructure capable of managing residency across all three as conditions shift.
Start by establishing the weight floor at the target precision, whether that's BF16, Q8, or Q4. That number is fixed, and it sets the minimum size of the GPU pool. For Mixtral 8x7B, plan on needing multiple GPUs at any precision above Q4.
Next, estimate the KV cache budget at the concurrency and context length the deployment actually expects to see. This figure grows with load, so it has to be measured under realistic traffic rather than at idle, and planners should expect KV cache to consume a meaningful share of whatever headroom remains after the weight floor is accounted for.
Add batch headroom on top of that. Load tests run at realistic concurrency are the only reliable way to catch peak VRAM under batch inference, since static estimates understate this number for Mixtral 8x7B in ways they simply don't for Mistral 7B.
Choose a KV cache management strategy next. Prefix caching, FP8 quantization, and tiered residency across GPU HBM and NVMe each reduce how much KV budget has to live in GPU memory, and the right choice depends on how the workload actually accesses its context, shared prefixes across many requests versus long, mostly unique sessions.
Finally, validate the interconnect. If the weight pool spans multiple GPUs or multiple nodes, benchmark that interconnect under the actual all-to-all communication pattern MoE routing generates, not under a synthetic or best-case test. Managing this dual-layer memory budget, weights plus the per-request overhead stacked on top, is where purpose-built infrastructure stops being optional: systems like Open Lake's KV cache management layer are built to keep high-utilization deployments from swinging between memory pressure and unacceptable latency when serving multiple concurrent requests against quantized expert models. Understanding these three layers on their own terms, static weight storage, per-request KV cache growth, and batch-driven VRAM spikes, is what separates a deployment plan that survives contact with production traffic from one that was never provisioned for it in the first place, and it's why infrastructure teams are increasingly turning to storage systems built specifically for LLM serving rather than adapting general-purpose solutions to a problem they weren't designed to solve.
Sources
- OpenLake
- LLM Inference Sizing and Performance Guidance - VMware Cloud Foundation (VCF) Blog
- Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
- Beyond Capacity: Scalable MoE LLM Inference via High-Bandwidth Flash with Direct GPU and HBM Paths
- Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
- mistralai/Mixtral-8x7B-Instruct-v0.1 · Hardware Requirements


