GPU Direct Storage Integration With Open-Source Inference Runtimes

Eliminating the CPU bottleneck in GPU storage access unlocks faster inference on longer contexts.

· 11 min read
Cover illustration for “GPU Direct Storage Integration With Open-Source Inference Runtimes”
Model-to-Hardware Fit · October 16, 2026 · 11 min read · 2,370 words

GPU Direct Storage, or GDS, solves a structural problem that faster GPUs cannot address on their own: the CPU bounce buffer in the classical storage data path. Integrating GDS with open-source inference runtimes is a concrete engineering task with specific mechanics, and understanding those mechanics determines whether the technology delivers its latency and throughput promises in production serving.

Why the CPU bounce buffer is the load-bearing problem GDS is solving

The classical storage data path imposes a structural ceiling on GPU inference performance. Each hop consumes PCIe bandwidth and CPU cycles, and the cumulative cost is not trivial. It represents a fixed tax on every byte that moves.

The CPU is not merely slow in this arrangement. This becomes especially damaging at inference time, where I/O bursts are short, irregular, and latency-sensitive. The long sequential reads where CPU buffering amortizes its cost are not the pattern that dominates serving workloads.

What GDS actually is: the cuFile API, the nvidia-fs driver, and how the DMA path is established

Diagram: The GDS Stack: Three Layers, One Direct Path. Visualizes: Show the contrast between two data paths: the classical CPU bounce buffer path (Storage → CPU staging buffer → GPU HBM, consuming PCIe bandwidth and CPU cycles at each hop) versus…

GDS is not a single toggle but a stack of cooperating components. The stack has three layers: the cuFile user-space API, which is the interface applications call. The nvidia-fs kernel driver, which manages the DMA mapping between GPU memory and the storage device. And the storage layer itself, which must support O_DIRECT and present a compatible block device or filesystem. Each layer carries its own version constraints and configuration surface.

NVIDIA's GDS Overview Guide describes cuFile APIs as providing CUDA applications with the best-performing access to local or distributed file and block storage, with interoperability with traditional OS file I/O. The GDS Release Notes from September 17, 2026 document a meaningful reduction in setup complexity: as of CUDA 12.8, the nvidia-fs kernel driver is no longer required for local NVMe drives or DOCA SNAP. The same notes list recent additions including scatter-gather I/O across multiple buffers, RAID1 and RAID10 on NVMe drives, and Linux kernel 7.0 support for nvidia-fs. cuFile was open-sourced in August 2026 with Google, Intel, Meta, and NVIDIA as co-maintainers, moving the project out of single-vendor control.

Filesystem and hardware compatibility: what the storage layer must provide for GDS to engage

GDS compatibility is narrower than most practitioners assume. The gap between "GDS is installed" and "GDS is active" is where most silent failures originate. Files must be opened with O_DIRECT, and the filesystem must support the direct I/O path without falling back to buffered I/O at the VFS layer. This single requirement excludes a wide range of configurations that appear superficially compatible.

The supported filesystems across release versions are EXT4, XFS, Lustre, GPFS, and VDURA PanFS, with ZFS and BTRFS arriving on the compatibility path in a more recent release. SLES supports GDS in P2PDMA mode. The distinction matters because native mode and compatibility mode are meaningfully different performance tiers.

The deeper constraint is the two-tier ecosystem problem. The full direct path, especially for remote NVMe-oF or distributed storage, requires RDMA infrastructure that purpose-built datacenter and bare-metal AI servers have but most standard cloud GPU instances lack. Engineers must assess whether their environment qualifies before investing in integration work.

The silent fallback trap: how cuFile degrades without telling you

Without explicit configuration enforcement, a GDS integration can run entirely on the CPU staging path and appear to function normally. The benchmark numbers it produces are indistinguishable from no-GDS operation. This is not a hypothetical failure mode. It is the default behavior when any prerequisite is missing. cuFile falls back to POSIX I/O through the CPU staging buffer if the nvidia-fs module is absent, the driver version is wrong, or the filesystem is unsupported. The GDS Release Notes document this as vanilla_posix_mode, a named and intentional fallback path.

The allow_compat_mode: false setting in /etc/cufile.json disables this silent fallback, causing cuFile I/O to fail rather than degrade quietly. Some filesystem-specific settings can still force compatibility mode regardless, but the primary control is there. Without it, there is no runtime signal that the direct path was never engaged.

The gdscheck utility reports which modes libcufile supports on the current system: p2pdma, nvfs, or compat. Its output was updated to specify these modes explicitly, so verification is straightforward. The operational implication is that benchmarking must be done with compatibility mode disabled and gdscheck output verified before any throughput result is interpreted as a GDS measurement.

KV cache eviction as the primary inference integration point

The KV cache eviction path, from GPU HBM to NVMe, is the inference I/O workload GDS is structurally best suited for. It is also the bottleneck that determines whether long-context serving is economically viable. Serving multiple concurrent users requires evicting cold KV blocks off-GPU on a continuous basis.

The storage hierarchy has three tiers. PCIe 5.0 CPU-GPU bandwidth is an order of magnitude below HBM bandwidth, which makes transferring large KV caches through the CPU DRAM tier a meaningful latency cost on the critical decode path. GDS on the eviction write path eliminates the CPU staging copy entirely: KV block to NVMe directly, rather than KV block to CPU staging buffer to NVMe.

NVIDIA's NIXL post explicitly cites long-context KV cache storage as a core use case: "for long context KV, the previous results can be loaded from local SSDs and remote storage, instead of recomputing them as prefill." KV cache offloading turns long-context inference into a storage throughput and latency problem at roughly 320 KB per token for a 70B-class GQA model at FP16, such as Llama 3 70B, though this varies widely by model architecture. At 1M-token contexts, the data path architecture determines whether the feature is feasible at all.

Wiring GDS into vLLM, LMCache, and TensorRT-LLM: where the integration points actually are

The integration depth varies significantly across open-source inference runtimes, from automatic filesystem-level acceleration to explicit custom code requirements. Each runtime presents a distinct integration pattern with a distinct effort level. vLLM's default model loading path is CPU-based. Amazon FSx for Lustre support on EC2 is the path that enables vLLM to benefit from GDS without code changes. That is substantially faster than standard vLLM load without GDS. TensorRT-LLM can similarly leverage the direct storage-to-GPU path automatically when the underlying filesystem supports GDS.

LMCache extends vLLM's built-in prefix cache with an NVMe tier, Redis, and remote object storage. Unlike vLLM's built-in swap-space capability, which is CPU RAM only, per-instance, and non-persistent, LMCache persists KV blocks to NVMe and shares them across replicas through shared backends such as Redis or remote object storage. The persistence survives server restarts when backed by disk or external storage.

PyTorch FSDP checkpointing presents the highest-friction integration point. PyTorch's checkpoint writer does not natively call cuFile, so the tensor serialization step must be wrapped with cuFile Python bindings to write each per-rank shard through the GDS path. This is explicit code work, not a configuration change.

NIXL, the NVIDIA Inference Transfer Library, provides a unified abstraction across these integration points. It is an open-source, vendor-agnostic data movement library with a unified API across GPU memory, CPU memory, and storage tiers. NIXL supports GDS as a backend alongside RDMA, GPU-initiated networking, NVMe, and cloud object stores. It is already integrated into NVIDIA Dynamo and TensorRT-LLM, providing a framework-level path to GDS without each runtime needing to call cuFile directly. NIXL's KVBench tool profiles LLM-aware KV cache transfer with automatic I/O size calculation, which matters for validating that GDS is actually engaged on the KV eviction path.

DeepSpeed's native GDS path and what it means for checkpointing workflows

DeepSpeed offers the lowest-friction GDS integration path for checkpoint workloads. The support is native, through a separate gds_handle abstraction using GDSBuilder in its DeepNVMe module, not through the aio config block. The trade-off is that GDS bypasses the Linux page cache entirely.

Data written via cuFileWrite bypasses CPU DRAM on the primary O_DIRECT path, though it may stage through system memory under compatibility mode, misalignment, or unsupported filesystem conditions. Practitioners who mix GDS and non-GDS readers on the same files will encounter failures that are difficult to diagnose. The throughput benefit comes from eliminating CPU I/O stalls on the write path, and the uplift is workload-conditional: it applies to I/O-bound operations, not compute-bound training steps.

Multi-node tensor parallelism and the RDMA dependency that gates the full GDS path

The highest-performance GDS configurations, covering remote NVMe-oF and distributed storage, require RDMA infrastructure that is present in purpose-built AI clusters but absent from most cloud GPU instances. This creates a genuine two-tier deployment reality. Disaggregated KV cache storage across nodes requires the full RDMA stack.

For local NVMe, the RDMA requirement is removed entirely. The GDS Release Notes confirm that the nvidia-fs driver is no longer required for local NVMe drives or DOCA SNAP as of CUDA 12.8, and local NVMe DMA goes directly over PCIe. NIXL is vendor-agnostic and runs across AWS with EFA networking and Trainium/Inferentia, and Azure with RDMA networking. The deployment-topology decision is the primary variable: bare-metal and purpose-built AI servers qualify for the full path, while standard cloud instances are confined to local NVMe.

FP8 KV quantization and lossless compression as storage-layer multipliers on the GDS path

What flows over the GDS path is as consequential as the path itself. FP8 KV quantization and lossless compression of AI-native formats reduce the volume of data that must traverse the direct path, making storage bandwidth go further without widening the pipe. Other models can fall as low as roughly 94 percent in the BF16-model setting, but the ceiling remains high.

For memory-bound decoding, FP8 KV-cache reduces per-token KV cost to roughly half its BF16 counterpart in the best cases. Lossless ZSTD compression of BF16 model weights achieves meaningful compression ratios, and combining lossless compression with FP8 quantization on LLaMA 3.1 8B achieves a total reduction exceeding half the original size. The mechanism is specific to the format: exponent and mantissa streams are extracted and coded separately, with skewed exponent distributions yielding exponent-stream compression ratios well below 0.3 in specific layers, typically through Huffman coding. On the GDS path, smaller tensors mean fewer DMA transactions, reduced NVMe write amplification, and lower DRAM pressure on the host, all without sacrificing the direct-path benefit.

The recompute-vs-reload trade-off and when GDS does not help

The case for recomputation over NVMe reload is real but narrows sharply at the context lengths that define current production workloads. The conditions that favor recompute are specific: short context, where prefill is cheap. High GPU utilization, where GPUs are not waiting. And constrained NVMe bandwidth, where the reload path is slow. The conditions that favor NVMe reload are the inverse: long context, where prefill is expensive. Multi-turn or agentic workloads, where the same KV blocks are reused across many requests. And provisioned high-endurance NVMe, where the storage tier is fast enough that reload beats recompute.

The honest answer is that it depends on context length, GPU utilization, and NVMe tier quality, not that GDS always wins. GDS is not the right optimization when the workload is compute-bound rather than I/O-bound. Any integration effort should establish which category the workload falls into before attributing performance gains to GDS.

MLPerf Storage v3.0's new KV cache benchmark and what it signals for the field

MLCommons released MLPerf Storage v3.0 results on September 1, 2026, adding a new KV Cache test that measures storage performance for LLM inference cache read and write operations. The addition is the clearest signal that storage performance on the KV eviction path is now a first-class infrastructure metric, not an implementation detail. MLCommons' rationale for adding the test explicitly acknowledges how widely KV caching is adopted for transformer-based AI inference, framing the benchmark as a response to production reality rather than a speculative category.

The checkpointing write test performance scaled from a substantial baseline with 10 data nodes to substantially higher throughput with 30 data nodes for 1,024 simulated accelerators. A benchmark gap remains: MLPerf Storage v3.0 measures KV cache read and write throughput at the storage system level, but it does not yet score the end-to-end inference path including GDS engagement. The field's direction is settled. The verification burden remains with the operator.

A practical integration checklist: the decisions that determine whether GDS delivers in production

Whether GDS delivers its latency and throughput promises in production depends on a sequence of decisions that must all be made correctly. Any gap in the chain produces either silent fallback or genuine unsuitability. The recurring theme is that misconfiguration and silent fallback are the most common failure modes, and the checklist should be treated as a debugging and verification sequence as much as a setup guide.

The first decision concerns the storage layer. Confirm that the filesystem supports GDS natively through EXT4, XFS, Lustre, GPFS, or VDURA PanFS, or in compatibility/P2PDMA mode through ZFS, BTRFS, or SLES. For remote storage, confirm that an RDMA-capable NIC of ConnectX-6 or newer and MOFED are present. For local NVMe on CUDA 12.8 or later, the nvidia-fs driver requirement is removed. The second decision concerns verification. Set allow_compat_mode to false in /etc/cufile.json to force loud failure rather than silent CPU-path fallback, then run gdscheck and verify the mode output before benchmarking.

The third decision concerns the integration path. vLLM on FSx for Lustre requires no code changes, but pre-sharded checkpoints are required for weight loading acceleration. PyTorch FSDP requires implementing a custom StorageWriter with cuFile Python bindings. DeepSpeed uses the gds_handle through the DeepNVMe module, with the caveat that page-cache bypass means GDS-written files cannot be read via mmap. NIXL serves as a unified abstraction layer when operating across multiple backends or building a disaggregated serving architecture.

The fourth decision concerns data format. Apply FP8 KV quantization before sizing the NVMe tier, since it roughly halves the I/O volume on the eviction path with near-lossless accuracy. Lossless compression of BF16 and FP8 tensors further reduces checkpoint and weight-loading I/O, and the combined reduction is meaningful enough to affect NVMe provisioning decisions.

The fifth decision concerns benchmarking. Use NIXL's KVBench for LLM-aware KV cache transfer profiling, and use MLPerf Storage v3.0 KV cache test results as external reference points for the storage tier. Every step in this sequence is a gate. Skipping any of them converts a performance investment into a configuration artifact indistinguishable from the default CPU path.

Diagram: The Integration Checklist: Five Gates to GDS in Production. Visualizes: Visualize five sequential decision gates that must all pass for GDS to deliver in production.

More in Model-to-Hardware Fit