Transformer

Full definition

Introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., the transformer replaced recurrent and convolutional architectures with a stack of self-attention and feed-forward layers. Self-attention lets each token weigh the relevance of every other token in the input directly, producing parallel training and long-range dependencies. Variants include encoder-only (BERT), decoder-only (GPT, Claude), encoder-decoder (T5), and vision transformers (ViT).

Why it matters

The transformer is the single most consequential architectural choice in modern AI. Almost every frontier AI model today is a transformer or a transformer variant. Investment in transformer-shaped infrastructure — long-context attention kernels, KV cache, GPU memory hierarchy — compounds across language, vision, and multimodal use cases.

Example

GPT-4o, Claude, Gemini, Llama, BERT, Stable Diffusion (text encoder), and Whisper are all transformer-based models — the same architecture, scaled and adapted to different modalities.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Neural Network
  A neural network is a machine-learning model composed of layers of interconnected nodes ("neurons") whose numerical weights are adjusted during training so the network maps inputs to desired outputs.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.

More models terms

• Diffusion Model
  A diffusion model is a generative model that learns to reverse a gradual noising process — starting from random noise and iteratively denoising it into a coherent image, audio waveform, or video — and is the dominant architecture behind modern AI image and video generation.
• Multimodal Model
  A multimodal model is an AI model that natively understands or generates more than one type of input — typically text plus images, audio, or video — within a single network rather than via separate task-specific models.
• Mixture of Experts (MoE)
  Mixture of Experts (MoE) is a neural-network architecture in which only a small subset of "expert" sub-networks activate for any given input, cutting compute and inference cost dramatically while preserving large total parameter capacity.