Transformer Architecture (Components)

Full definition

A transformer block combines multi-head self-attention, a position-wise feed-forward network, residual connections, and layer normalization. Variants include encoder-only (BERT), decoder-only (GPT family, Llama), and encoder-decoder (T5, original 2017 design). Decoder-only autoregressive transformers dominate generative AI in 2024–2025. Scaling laws from Kaplan et al. (2020) and Hoffmann et al. (Chinchilla, 2022) describe how transformer loss decreases predictably with parameters, data, and compute.

Why it matters

Transformers are the substrate of every frontier model — GPT, Claude, Gemini, Llama. Their parallelism is what turned AI training from CPU-bound to GPU-bound, making the trillion-dollar AI capex cycle possible. Understanding the architecture is required to evaluate any vendor model card.

Example

Llama 3 70B is a decoder-only transformer with 80 layers, 64 attention heads, and SwiGLU feed-forward layers — the same template as GPT-2, scaled and refined.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Attention Mechanism
  An attention mechanism is a neural-network operation that lets each output position weigh every input position by learned relevance scores, replacing fixed-window context with content-addressed lookup.
• Encoder-Decoder
  An encoder-decoder model is a neural architecture with two stacks: an encoder that compresses input into a representation and a decoder that generates output from it, with cross-attention linking the two.
• Transformer
  The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.

More models terms

• Autoregressive Model
  An autoregressive model generates output one token at a time, where each new token is conditioned on every previous token in the sequence, producing text by repeated next-token prediction.
• Masked Language Model
  A masked language model (MLM) is a model trained to predict tokens that have been hidden in the input, learning bidirectional context rather than left-to-right next-token prediction.