Attention Mechanism

Full definition

Attention computes a weighted sum of value vectors, where weights come from the dot product of a query vector and key vectors, scaled and softmaxed. Self-attention applies this within a single sequence so every token can attend to every other token in O(n²) time. Multi-head attention runs the operation in parallel across multiple learned subspaces. Vaswani et al. introduced scaled dot-product attention in the 2017 "Attention Is All You Need" paper, which underpins every modern large language model.

Why it matters

Attention is the single architectural primitive that made trillion-parameter language models tractable. Quadratic cost in sequence length is also the dominant driver of long-context inference price — buyers should ask vendors how attention is computed (vanilla, FlashAttention, paged) when comparing token economics.

Example

When GPT-4 reads "The trophy did not fit in the suitcase because it was too large," self-attention links "it" to "trophy" rather than "suitcase" via learned attention weights — the resolution is computed, not hard-coded.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Transformer Architecture (Components)
  Transformer architecture is the internal design of a transformer model: a stack of identical blocks, each combining multi-head self-attention, a position-wise feed-forward network, residual connections and layer normalization, arranged as an encoder, a decoder or both. For what the transformer is and why it dominates AI, see Transformer.
• Transformer
  The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.

More models terms

• Encoder-Decoder
  An encoder-decoder model is a neural architecture with two stacks: an encoder that compresses input into a representation and a decoder that generates output from it, with cross-attention linking the two.
• Autoregressive Model
  An autoregressive model generates output one token at a time, where each new token is conditioned on every previous token in the sequence, producing text by repeated next-token prediction.