Transformer
The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.
Full definition
Introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., the transformer replaced recurrent and convolutional architectures with a stack of self-attention and feed-forward layers. Self-attention lets each token weigh the relevance of every other token in the input directly, producing parallel training and long-range dependencies. Variants include encoder-only (BERT), decoder-only (GPT, Claude), encoder-decoder (T5), and vision transformers (ViT).
Why it matters
The transformer is the single most consequential architectural choice in modern AI. Almost every frontier AI model today is a transformer or a transformer variant. Investment in transformer-shaped infrastructure — long-context attention kernels, KV cache, GPU memory hierarchy — compounds across language, vision, and multimodal use cases.
Example
GPT-4o, Claude, Gemini, Llama, BERT, Stable Diffusion (text encoder), and Whisper are all transformer-based models — the same architecture, scaled and adapted to different modalities.
Related terms
- Large Language Model (LLM)A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
- Neural NetworkA neural network is a machine-learning model composed of layers of interconnected nodes ("neurons") whose numerical weights are adjusted during training so the network maps inputs to desired outputs.
- Context WindowA context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
- KV CacheThe KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
- Foundation ModelA foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.
Source & further reading
Primary source: Vaswani et al. — "Attention Is All You Need" (NeurIPS) (2017).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/es/glossary/transformer/.