Llama 3 70B Serving on H100 vs A100 NVMe Offloading Configurations
H100's compute advantage shrinks dramatically when KV cache fills up.

The H100 beats the A100 on every major spec that governs Llama 3 70B inference, and that gap is large enough to matter. But the spec sheet answers a narrower question than most teams think it does, and the gap it describes only tells part of the production story. A100 SXM4 carries HBM2e at 2,039 GB/s, third-generation Tensor Cores, and no FP8 hardware support, connected via NVLink 3.0 at 600 GB/s bidirectional. Both cards ship with 80 GB of VRAM, so capacity is not what separates them: bandwidth is, because LLM decode is a memory-bandwidth-bound operation. Bandwidth, not raw compute, is the binding constraint on how fast tokens come out the other end.
Form factor quietly rewrites the comparison here: if you buy H100 PCIe expecting SXM-level bandwidth gains, you will not get them. The compute gap NVIDIA advertises translates into throughput numbers that look decisive in isolation, holding up only until KV cache pressure and memory capacity enter the picture at the concurrency levels and context lengths production serving actually demands.
How large the throughput gap is under concurrent load, and why it collapses at batch size 1

The throughput advantage H100 holds over A100 is not a constant. It widens sharply as concurrent load rises and shrinks to something far more modest at batch size 1. A100 saturates near approximately 570 TPS at just 50 users, so the advantage across the sweep is roughly 12 to 14 times. With longer inputs of 1,000 tokens in and 200 tokens out, H100 delivered approximately 2,600 TPS at 250 concurrent users, while A100 stayed under approximately 230 TPS at the same load.
At batch size 1, the picture changes substantially. H100 runs single-stream Llama 3 70B inference at approximately 280 tokens per second, against approximately 130 tokens per second on A100, which is a meaningful difference but nowhere near an order of magnitude. Three architectural mechanisms explain why concurrency amplifies the gap so much more than single-stream workloads do. A100 has no FP8 hardware at all, so BF16 sits as its precision ceiling. Tensor Memory Accelerators combine with the Transformer Engine so that H100 throughput scales with batch size far more aggressively than A100 ever can, and NVIDIA's own figures put the Transformer Engine's throughput advantage at 3 to 4 times on models above 30B parameters where memory bandwidth is the primary constraint. Llama 3.3 70B in BF16 requires a minimum of two 80 GB GPUs, and packing the model onto two cards produces this tight fit: almost no headroom remains for KV cache.
KV Cache as the Primary Memory Bottleneck at 128K Context and Real Concurrency

At realistic concurrency and long context lengths, KV cache consumption overwhelms the HBM available on both H100 and A100. NVMe offloading stops being a nice-to-have, because it is the only way to make multi-user, long-context serving possible on either card. Llama 3 70B uses Grouped Query Attention at an 8:1 ratio, so eight query heads share a single KV head, and that shrinks the KV cache footprint relative to full multi-head attention, but it does not remove the underlying problem. At 128K context, a single user's KV cache on H100 serving Llama 3.1 70B takes approximately 40 GB. Eight concurrent users at that context length need approximately 320 GB total, which is four times the entire HBM capacity of the H100.
The memory hierarchy beneath the GPU is tiered, and the bandwidth drop between tiers is severe enough to reshape every design decision that follows. GPU HBM runs at approximately 3.35 TB/s on H100 SXM. NVMe SSD over PCIe 4.0 delivers approximately 7 GB/s. The practical consequence is stark: transferring a 50 GB KV cache takes approximately 15 milliseconds from HBM but approximately 800 milliseconds from CPU DRAM, before NVMe is even reached. These numbers dictate an eviction policy that production systems have converged on. Hot KV blocks, meaning those in active generation, stay resident on GPU HBM. Cold blocks, historical context and prefix cache, move to NVMe. This is the point at which H100 and A100 start to diverge in a way no spec sheet captures directly: FP8 KV cache dtype halves the per-block memory footprint on H100, which increases how many hot blocks stay resident before eviction has to begin. A100 cannot do this natively, and that one gap shapes every decision you make about storage architecture from here on.
NVMe Offloading and the Capacity Equation for H100 and A100
FP8 KV cache on H100, paired with NVMe offloading, opens up multi-user long-context serving that A100 cannot reach even with an identical storage configuration. Running H100 with the --kv-cache-dtype fp8 flag halves KV memory per block on the GPU, leaving more room for hot blocks before eviction starts. That single setting enables NVMe offload as a viable production strategy. With FP8 KV and NVMe offloading at 128K context and a high prefix cache hit rate, a single H100 can serve approximately 12 to 15 concurrent users, versus just 1 to 2 users relying on HBM alone.
A100 hits a harder wall under the same conditions. The NVMe I/O budget drains faster on A100 for the same reason: more blocks must move per request, and the PCIe 4.0 NVMe ceiling of roughly 7 GB/s is shared across every concurrent eviction competing for it. The quality ceiling takes a hit too, because INT4 or INT8 quantization throughout cuts into the model's headroom no matter what storage configuration sits underneath it. Workloads with unique prefixes, such as retrieval-augmented generation over novel documents or streaming agents with no repeated context, may see no benefit from NVMe offloading at all and can suffer worse time-to-first-token than a baseline configuration without it.
The I/O substrate that determines whether NVMe offloading is viable or catastrophic
Whether NVMe offloading gives you near-DRAM performance or multi-second stalls depends mostly on which I/O path you pick, not on how fast the drives are. GPUDirect Storage sets the baseline requirement for any serious offload architecture. GDS eliminates CPU bounce buffers by enabling direct DMA transfer between NVMe and GPU. Without it, every KV block eviction travels a three-hop path from GPU to CPU to NVMe, saturating PCIe twice and adding staging latency at each hop.
A handful of production systems have been built specifically to solve this. Mooncake's Transfer Engine has been integrated into both TensorRT-LLM and vLLM v1 as a KV Connector for production prefill-decode disaggregated inference, and Mooncake now supports SSD offloading directly. Tutti, a 2026 system, takes a GPU-centric approach to SSD-backed KV cache, removing CPU bottlenecks through object abstraction, GPU io_uring, and slack-aware I/O scheduling, and it delivers near-DRAM performance at higher request rates and lower cost than earlier GDS-based systems. LMCache extends vLLM's in-process prefix caching with an NVMe tier and is the source of the throughput figures cited earlier for high-prefix-reuse workloads.
The fastest results reported for disaggregated offload depended on more than NVMe hardware alone. Cutting cold-prefix TTFT from over 11 seconds down to approximately 1.5 seconds at 128K context required a 400 Gbps RDMA fabric, BlueField-3 DPUs, NVIDIA Magnum IO GPUDirect Storage, and a context length long enough to make the trade-off worthwhile, all working together. Teams without RDMA infrastructure will see that offload benefit compressed substantially, since the NVMe ceiling without GDS and RDMA sits much lower. A newer line of research on zero-copy KV offloading removes the intermediate DRAM staging step entirely, collapsing the three-hop GPU-to-CPU-to-NVMe path into a direct GPU-to-NVMe transfer. That approach reduces the latency penalty for A100 configurations specifically, since they lack FP8 KV as a mechanism for keeping blocks hot. Benchmarks that isolate storage bandwidth impact, including those published by one vendor's storage benchmark participation, reveal where the compute gap from the spec sheet actually manifests in production serving, shifting attention away from raw compute throughput and toward the data movement constraints that govern real-world throughput.
KV cache compression as a force multiplier on NVMe offload effectiveness
Compressing KV blocks before they are offloaded reduces the PCIe and NVMe bandwidth tax that offloading otherwise imposes, and the schemes delivering the largest gains are tied closely to FP8 and disaggregated serving architectures.
This advantage compounds asymmetrically in favor of H100 configurations. Compression schemes such as Minima-KV and SplitZip operate on FP8 or sub-FP8 representations, which assumes the serving pipeline can both produce and consume FP8 KV blocks in the first place, a capability A100 does not have natively. On A100, compression has to work on INT8 or BF16 blocks instead, so it limits the achievable reduction ratio and adds extra quantization steps that eat into the model's quality ceiling. Smaller compressed blocks mean fewer NVMe I/O operations per eviction event, higher effective NVMe throughput, and lower cold-prefix fetch latency, and all of these advantages stack directly on top of the benefit FP8 KV already provides on H100. The data distributions found in KV tensors differ fundamentally from the payloads general file or database compression tools were built for, so this stays an active area of research, not a settled engineering problem.
When A100 Configurations Remain Rational
NVMe offloading is not a universal equalizer. The workloads where it delivers the largest gains are structurally different from the workloads where it only adds latency without a corresponding throughput benefit, and A100 remains a cost-rational choice in the latter case. NVMe offloading only earns its keep when prefix cache hit rates run high enough that the TTFT savings from cache reuse outweigh the penalties incurred during eviction. Unique-prefix workloads, including RAG applications over novel documents and streaming agents with no repeated context, may see no benefit from offloading and can end up with worse TTFT than a baseline that skips it.
The disaggregation-versus-colocation choice splits along similar lines. Mooncake-style architectures, which separate prefill and decode into distinct pools, maximize throughput for high-concurrency workloads built on shared prefixes, but they require RDMA fabric and inter-node KV transfer infrastructure that many deployments simply do not have in place. The colocated approach used by PagedAttention in vLLM's default configuration avoids that RDMA dependency and the inter-node transfer overhead that comes with it, and you can run it more simply at moderate concurrency. KVPR, or KV partial recomputation, offers a middle path between these two extremes: overlapping PCIe transfers with partial KV recomputation on the GPU reduces stall time without requiring a full move to disaggregated serving.
Against this backdrop, A100 holds up as a rational choice in specific, well-defined circumstances. Even on pure cost, single-stream inference cost per million output tokens favors H100, but the margin at batch size 1 is not overwhelming, and for workloads that are not constrained by concurrency, A100 can be the more economical option on balance.
Serving Framework Choice, GPU Tier, and NVMe Configuration
The serving framework sitting on top of the hardware is an independent variable in its own right. The same GPU and the same NVMe configuration can produce materially different throughput depending on which stack is managing KV cache and how. On H100, SGLang supports FP8 natively, and TensorRT-LLM's H100 engine defaults to FP8 for transformer inference. SGLang's throughput edge on H100 also comes from RadixAttention, a memory manager built specifically for serving patterns with shared prefixes across requests. LMCache's NVMe tier integrates with vLLM's in-process prefix caching, and the throughput gains cited earlier, running from approximately 900 tokens per second up to approximately 1,800 tokens per second, are specific to that combined stack.
A100 hits a framework ceiling, and no amount of serving-stack optimization can lift it. BF16 remains the practical precision ceiling for A100 workloads, and frameworks that expose FP8 execution paths, including SGLang and TensorRT-LLM, cannot use them on A100 regardless of configuration. Storage bandwidth and KV cache residency occur as memory pressure rises, and they decide whether the architectural edge is visible in production.
Matching GPU Tier and NVMe Configuration to Workload Shape
The right configuration comes down to three workload variables: concurrency, context length, and prefix reuse rate, and how those three interact with GPU memory bandwidth and NVMe I/O throughput together, not any single factor in isolation.
For high concurrency, long context, and high prefix reuse, such as shared system prompts, multi-turn chat, or document QA against repeated documents, H100 SXM paired with FP8 KV and NVMe offload running through GDS is the only configuration that serves this reliably at scale. A100 cannot match it with an equivalent storage setup, because it has no FP8 KV and its per-block eviction cost runs higher throughout.
If you need more capacity here, more GPU HBM, whether through additional cards or a move to H200, works better than more NVMe.
For moderate concurrency at any context length under cost pressure, A100 with INT8 quantization and CPU DRAM offload remains a defensible choice.
For batch or offline inference, H100 costs less overall despite its higher hourly rate, because it throws so much more throughput at large batch sizes that A100 needs substantially more instances to match it.
One warning applies across all four cases: teams deploying H100 PCIe rather than SXM need to account for its HBM2e bandwidth of 2,000 GB/s rather than the 3,350 GB/s available on SXM. The NVMe offload benefit compresses under PCIe, because fewer hot blocks fit in HBM before eviction starts, and the GPU-to-NVMe transfer path is itself PCIe-constrained on top of that.
Open Lake works on the storage and infrastructure decisions for teams running Llama 3 70B and similar models in production, including KV cache tier management, GDS integration, high-throughput NVMe at scale, and lossless compression for FP8 and BF16 KV blocks, which decide whether NVMe offloading helps or hurts. Its work spans the full stack from GPU memory hierarchy through local NVMe to networked storage, which is the exact range the decision matrix above calls for.
Before committing to any of these architectures, characterize the workload's prefix reuse rate first: a cache hit rate measurement on representative traffic is the single most useful benchmark to run before anything else. Validate that GDS and RDMA fabric are actually available in the target environment before assuming the TTFT figures from disaggregated offload research will carry over. And you should benchmark at the concurrency level the deployment expects to see in production, not at batch size 1, because single-request numbers miss the queueing effects that end up dominating real-world outcomes.


