FlashAttention

Full definition

Introduced by Dao et al. (Stanford, 2022), FlashAttention recognizes that standard attention is memory-bound, not compute-bound: most time is spent reading the N×N attention matrix to and from HBM. FlashAttention tiles the computation, computes softmax online, and never materializes the full matrix — yielding wall-clock speedups of 2–4x on training and inference. FlashAttention-2 (2023) improved parallelism and work partitioning; FlashAttention-3 (2024) uses Hopper's asynchronous tensor cores to reach up to 740 TFLOPs/s in FP16, 75% of the H100's theoretical maximum (Shah et al., 2024).

Why it matters

FlashAttention is the default in every modern training stack (PyTorch SDPA, vLLM, TensorRT-LLM, Hugging Face). Workloads still on vanilla attention are paying 2–4x more for the same compute — a clear signal of an immature serving stack.

Example

A research team retrains a 7B model with FlashAttention-2; training time drops from 14 days to 6 days on the same 64xA100 cluster, with bit-identical losses.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Attention Mechanism
  An attention mechanism is a neural-network operation that lets each output position weigh every input position by learned relevance scores, replacing fixed-window context with content-addressed lookup.
• Paged Attention
  Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.
• Transformer Architecture (Components)
  Transformer architecture is the internal design of a transformer model: a stack of identical blocks, each combining multi-head self-attention, a position-wise feed-forward network, residual connections and layer normalization, arranged as an encoder, a decoder or both. For what the transformer is and why it dominates AI, see Transformer.
• GPU (in AI Context)
  A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks many times faster than a CPU — and the dominant hardware for both training and inference of modern AI models.

More infrastructure terms

• Prefix Caching
  Prefix caching is an inference optimization that reuses the KV-cache computed for a shared prompt prefix — system prompts, few-shot examples, retrieved documents — across multiple requests, eliminating redundant prefill compute.
• TPU
  A TPU (Tensor Processing Unit) is Google's custom AI accelerator designed specifically for tensor operations — the dense matrix multiplications at the core of neural networks — and used in production to train and serve Google Search, Translate, YouTube, and Gemini.