Attention Kernel Efficiency at Long Sequence Lengths in Flash Attention 3
FlashAttention-3 reaches GPU compute limits on long sequences through asynchronous execution.

To serve LLMs with 128K or longer context windows, teams need to design hardware-aware attention kernels, KV cache management, and a storage tier together. FlashAttention-3 shows why this matters by making long-sequence attention compute-bound on Hopper through asynchronous execution and controlled FP8 quantization while retaining the existing tiling principle. That kernel efficiency becomes production throughput only when the memory hierarchy can hold and deliver the attention state.
Why standard attention becomes unworkable as sequence length grows
Standard scaled dot-product attention materializes an N×N attention matrix in high-bandwidth memory, or HBM. Its memory requirement therefore grows quadratically with sequence length. At 128K tokens, the FP16 attention matrix alone takes up tens of gigabytes, even before any other model state is counted. This intermediate matrix strains both capacity and bandwidth, since writing and reading it consumes time that the GPU could otherwise spend computing. Attention consequently stalls well below the throughput its arithmetic hardware could deliver.
Avoiding that matrix is necessary, but it does not by itself close the utilization gap. FlashAttention-3 documents that FlashAttention-2 achieves only a fraction of the utilization of optimized GEMM kernels on H100. At the system level, long-context serving also needs control over where attention state resides and how quickly it moves. Open Lake's KV cache management layer addresses that residency and throughput problem across memory tiers, rather than treating GPU capacity as the only available limit.
What FlashAttention's tiling approach changed, and what it left unsolved
FlashAttention's tiling strategy removed the need to write the full attention matrix to HBM by keeping intermediate computation in SRAM. H100 has a three-level memory hierarchy comprising SRAM, L2, and HBM. SRAM can sustain compute-bound attention if data arrives at the right time, so the remaining constraint is how the kernel coordinates movement and computation.
FlashAttention-2 improved parallelism across sequence length and distributed work more effectively, but its design assumes synchronous execution. Hopper instead provides independently schedulable Tensor Cores and a data-transfer engine that can move data while computation runs. FlashAttention-3 at NeurIPS 2024 identifies the absence of explicit asynchrony and low-precision handling in FA-2 as a design limitation. Hopper's FP8 Tensor Cores also offer nearly double the arithmetic throughput available at BF16. Exploiting either capability requires a structural change to execution beyond adjusting tile size.
The three mechanisms FA3 uses to exploit Hopper hardware
FA3 coordinates three techniques, namely warp-specialized pipelining, softmax hiding, and FP8 block quantization. The first divides warps into producers and consumers. Producers drive TMA transfers from HBM into shared memory, while consumers issue WGMMA instructions to Tensor Cores. With the warps split this way, data can move while arithmetic runs, so the same execution path no longer has to wait through both. It also addresses the instruction-issue latency exposed by synchronous scheduling.
The second technique addresses the dependency between attention scores, softmax normalization, and the weighted sum. A block's scores must be normalized before its weighted sum can proceed, but that does not require all work on other blocks to stop. FA3 runs softmax on one block while the asynchronous WGMMA proxy computes the next block's matrix multiplication. The dependency remains within each block, while overlap across blocks hides its cost. The mathematical attention operation stays the same under this new schedule.
The third technique makes Hopper's FP8 throughput usable without accepting the error of naive FP8 attention. FA3 scales each tile independently and applies incoherent processing through randomized Hadamard transforms, which spread outlier energy across a block. FlashAttention-3 at NeurIPS 2024 reports 2.6× lower numerical error than a baseline FP8 attention implementation. This handling of the format follows the same principle as Open Lake's approach to AI-native formats, which need more than generic compression when cache state moves across tiers: handling built for each format.
Together, these mechanisms let computation occupy hardware that synchronous execution leaves idle. FlashAttention-3 at NeurIPS 2024 measures BF16 utilization reaching 85% on H100, substantially narrowing the gap to optimized GEMM. The coordinated design produces this gain, with FP8 contributing only part of the benefit.

Why the efficiency gains compound at long sequence lengths specifically
Short sequences provide relatively few tiles over which to amortize pipeline setup. Warp specialization and asynchronous scheduling can then take up a large share of the total work. Longer sequences supply more useful matrix multiplication relative to that overhead, allowing the pipeline to remain productive for longer. The advantage is not simply a flat speed multiplier across context sizes. It becomes more consequential for long-context inference, multi-document reasoning, and video workloads because those workloads give the asynchronous machinery more work to sustain.
FlashAttention-3 at NeurIPS 2024 reports a large-sequence pattern in which BF16 outperforms and FP8 is on par in its comparisons. Precision selection must therefore account for sequence length alongside peak Tensor Core rates. Hardware-Efficient Attention frames the underlying constraint as arithmetic intensity, or computation per byte loaded from HBM, which also depends on how many query heads share each KV head. Avoiding quadratic attention-matrix traffic makes long sequences feasible, while maintaining a favorable ratio of arithmetic to data movement keeps the GPU compute-bound.
Where FP4 mixed-precision attention goes next and what it reveals about the precision trade-off
FP4 sharpens the problem that FA3 addresses at FP8. ThriftAttention finds that the output impact of FP4 quantization error is highly uneven across query-key interactions. Functionally important error concentrates in a small number of blocks containing the most important tokens, and the quality penalty worsens with context length. Uniform low-bit computation can therefore distort the interactions that matter most. Per-block scaling alone does not resolve that imbalance.
ThriftAttention responds by using a lightweight heuristic to select high-scoring query-key block pairs for FP16 computation while leaving the remainder in FP4. Online softmax merges the two paths into a single output. ThriftAttention's authors report that a small FP16 block budget recovers 89.1% of the FP4-to-FP16 quality gap on average, with the advantage over uniform FP4 growing at longer sequences. Blackwell's FP4 Tensor Cores offer roughly 4× the arithmetic throughput of FP16, so the incentive to solve this problem is substantial. Precision control may require routing important interactions to a different format in addition to rescaling every block in the same way.
How FA3's compute-bound attention reshapes KV cache pressure and storage requirements
A serving system can still face other limits when its attention kernel is compute-bound. For large models at 128K context, the KV cache dominates HBM consumption. An H100 SXM5 with 80 GB HBM holds only a small number of concurrent long-context users with BF16 KV, even when attention executes efficiently. The constraint moves from kernel throughput toward cache capacity, residency, and delivery.
FP8 KV quantization roughly halves the per-user cache footprint, but extends the precision-quality trade-off from computation to stored state. When HBM fills, cross-tier caching can extend into CPU DRAM and NVMe. LMCache spans those tiers and can avoid repeated prefill computation on shared long prefixes, saving redundant work even with NVMe's lower bandwidth than HBM.
Disaggregated architectures such as Mooncake go further by separating prefill and decode pools and transferring KV blocks over RDMA. Storage then has a role in planning how work runs as well as holding overflow. In systems with a KV management layer such as Open Lake's, residency, compression, and data movement determine how much kernel efficiency reaches concurrent users. The engineering requirement is to make retained state useful without allowing transfer costs to erase the computation saved.
The architectural fork between full-attention kernels and attention variants that reduce the KV load
FA3 optimizes full attention while retaining all query-key interactions. Other architectures cut down on the state that the kernel must fetch. Multi-Head Latent Attention, used in DeepSeek models, reduces its cache by compressing keys and values into a low-rank latent representation before they reach HBM. That reduction also eases downstream capacity and transfer demands.
Hardware-Efficient Attention introduces Grouped Latent Attention with a design that works well in parallel and an optimized kernel. It reports performance up to 2× faster than FlashMLA in speculative decoding settings. The same work's Grouped-Tied Attention matches GQA quality while using roughly half the KV cache. These results make a different trade-off between quality and efficiency from that of full-attention kernel optimization. They address bytes fetched as well as the efficiency of the computation performed on those bytes.
ThriftAttention's analysis finds that sparse methods must omit at least 75% of KV blocks to match FP4 latency during generation, while the information lost through omission cannot be recovered. FA3 with FP8 retains full attention with controlled quantization error, though low-precision arithmetic is not error-free. Conversely, a smaller architectural cache makes HBM capacity, NVMe offloading, and RDMA transfer easier to manage. Practitioners use both kernel optimization and architectural KV reduction because neither strategy settles every workload's trade-off.
What FlashAttention-4 and Blackwell change about the efficiency calculus
FlashAttention-4 extends the hardware co-design principle to Blackwell's SM100 architecture. Its algorithm and kernel pipelining respond to asymmetric scaling, where Tensor Core throughput grows faster than other resources, including SFUs and shared memory bandwidth. As on Hopper, increasing arithmetic capacity exposes delays elsewhere in the execution path. A kernel must address those imbalances explicitly to benefit from the hardware. FA3 is therefore a template for redesign rather than a final schedule that applies unchanged to every GPU.
Modal's FlashAttention-4 integration with arbitrary KV page sizes achieved up to 2.40× throughput improvement for small page sizes. The result ties throughput directly to the way the cache is paged and managed. vLLM uses FlashInfer as the default attention backend on Blackwell B200/B300, covering block-sparse KV attention, MLA, grouped GEMM for MoE layers, and FP4/FP8 quantized attention.
Work on scheduling for specific hardware also includes NVIDIA's GB10 processor, which was available as of October 15, 2025. GB10 was the platform studied for sawtooth wavefront reordering in CuTile FlashAttention. That work carries Hopper's co-design method into warp scheduling on Blackwell.
What infrastructure do teams need to serve LLMs with 128K or longer contexts today
Teams building long-context serving on H100 should treat FA3's measured 85% BF16 utilization and approximately 1.3 PFLOPs/s at FP8 as kernel ceilings. Reaching useful concurrency also requires enough effective KV capacity and a data path that can supply the kernel. Precision selection should be sequence-length-aware because the fastest nominal format and the best quality-throughput balance need not coincide. BF16, FP8, and emerging FP4 attention therefore belong in a workload-specific decision.
The corresponding infrastructure decision spans paged block management, cross-tier caching to NVMe, and disaggregated prefill-decode with RDMA. These mechanisms shape how much higher kernel throughput lets the system serve more users at once and how much it mainly cuts latency for a single session. For contexts measured in millions of tokens, kernel utilization alone is likewise not a sufficient architecture criterion. They still need to weigh the cache footprint, how much they can reuse shared prefixes, what transfers cost, and how well precision is preserved.
Open Lake treats the storage tier as part of this system through KV overflow handling, compression primitives, and data movement across NVMe and RDMA layers. As kernels approach hardware limits, the production gap increasingly lies in where state resides and how it reaches compute. Long-context LLM serving infrastructure must therefore co-optimize the kernel, the KV cache management layer, and storage as one system.


