Transformer Architecture

Full definition

A transformer block combines multi-head self-attention, a position-wise feed-forward network, residual connections, and layer normalization. Variants include encoder-only (BERT), decoder-only (GPT family, Llama), and encoder-decoder (T5, original 2017 design). Decoder-only autoregressive transformers dominate generative AI in 2024-2025. Scaling laws from Kaplan et al. (2020) and Hoffmann et al. (Chinchilla, 2022) describe how transformer loss decreases predictably with parameters, data, and compute.

Why it matters

Transformers are the substrate of every frontier model — GPT, Claude, Gemini, Llama. Their parallelism is what turned AI training from CPU-bound to GPU-bound, making the trillion-dollar AI capex cycle possible. Understanding the architecture is required to evaluate any vendor model card.

Example

Llama 3 70B is a decoder-only transformer with 80 layers, 64 attention heads, and SwiGLU feed-forward layers — the same template as GPT-2, scaled and refined.

Related terms

• Attention Mechanism
  An attention mechanism is a neural-network operation that lets each output position weigh every input position by learned relevance scores, replacing fixed-window context with content-addressed lookup.
• Encoder-Decoder
  An encoder-decoder model is a neural architecture with two stacks: an encoder that compresses input into a representation and a decoder that generates output from it, with cross-attention linking the two.
• Transformer
  The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.