Attention Mechanism

Full definition

Attention computes a weighted sum of value vectors, where weights come from the dot product of a query vector and key vectors, scaled and softmaxed. Self-attention applies this within a single sequence so every token can attend to every other token in O(n²) time. Multi-head attention runs the operation in parallel across multiple learned subspaces. Vaswani et al. introduced scaled dot-product attention in the 2017 "Attention Is All You Need" paper, which underpins every modern large language model.

Why it matters

Attention is the single architectural primitive that made trillion-parameter language models tractable. Quadratic cost in sequence length is also the dominant driver of long-context inference price — buyers should ask vendors how attention is computed (vanilla, FlashAttention, paged) when comparing token economics.

Example

When GPT-4 reads "The trophy did not fit in the suitcase because it was too large," self-attention links "it" to "trophy" rather than "suitcase" via learned attention weights — the resolution is computed, not hard-coded.

Related terms

• Transformer Architecture
  The transformer architecture is a neural-network design built on stacked self-attention and feed-forward layers, with no recurrence or convolution, that processes sequences in parallel.
• Transformer
  The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.