FlashAttention

Full definition

Introduced by Dao et al. (Stanford, 2022), FlashAttention recognizes that standard attention is memory-bound, not compute-bound: most time is spent reading the N×N attention matrix to and from HBM. FlashAttention tiles the computation, computes softmax online, and never materializes the full matrix — yielding wall-clock speedups of 2-4x on training and inference. FlashAttention-2 (2023) and FlashAttention-3 (2024) extend this to Hopper GPUs with asynchronous tensor cores, hitting 75% of theoretical FP16 throughput on H100.

Why it matters

FlashAttention is the default in every modern training stack (PyTorch SDPA, vLLM, TensorRT-LLM, Hugging Face). Workloads still on vanilla attention are paying 2-4x more for the same compute — a clear signal of an immature serving stack.

Example

A research team retrains a 7B model with FlashAttention-2; training time drops from 14 days to 6 days on the same 64xA100 cluster, with bit-identical losses.

Related terms

• Attention Mechanism
  An attention mechanism is a neural-network operation that lets each output position weigh every input position by learned relevance scores, replacing fixed-window context with content-addressed lookup.
• Paged Attention
  Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.
• Transformer Architecture
  The transformer architecture is a neural-network design built on stacked self-attention and feed-forward layers, with no recurrence or convolution, that processes sequences in parallel.
• GPU (in AI context)
  A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks 10-100× faster than a CPU — and the dominant hardware for both training and inference of modern AI models.