FlashAttention

Overview

FlashAttention is the official implementation of FlashAttention and FlashAttention-2, described by its authors as fast and memory-efficient exact attention with IO-awareness. FlashAttention-3 is a beta release optimised for Hopper GPUs such as H100 and H800, shipped with FP16/BF16 forward and backward plus FP8 forward and requiring CUDA 12.3 or later, while FlashAttention-4 is written in CuTeDSL for Hopper and Blackwell GPUs and installed with pip. The kernels install with pip or from source and require a CUDA or ROCm toolkit and PyTorch 2.2 and above; the project is BSD 3-Clause licensed and free to use and modify.

Editorial verdict

Fits ML teams training or serving transformers on CUDA or ROCm GPUs that need faster, lower-memory attention without changing model code beyond swapping the kernel. The trade-off is hardware and version coupling: the newest generation is beta for Hopper and needs a recent CUDA toolkit and PyTorch release, Blackwell support is early, and the library provides speed and memory savings rather than any energy measurement. Published throughput gains are vendor-reported, so a buyer's own throughput and power draw need testing. Teams on unsupported accelerators, or without a build toolchain, should check compatibility first.

Written by the GreenAIX editors from the recorded fields above. Vendors do not review or approve verdicts.

Use cases

• Speed up attention computation in transformer models
• Reduce attention memory use for longer contexts
• Accelerate transformer training and inference
• Use FP8 attention kernels on Hopper GPUs

Sources

• FlashAttention GitHub repository
• FlashAttention README
• FlashAttention BSD 3-Clause licence