Autoregressive Model

Full definition

During training, an autoregressive language model learns to predict token t+1 given tokens 1..t, using a causal attention mask so future tokens cannot leak. At inference, generation is sequential: sample, append, repeat. GPT, Claude, Llama, Mistral, and Gemini-Pro are all autoregressive decoder-only transformers. The sequential nature is the root cause of inference latency — speculative decoding and continuous batching exist precisely to mitigate it.

Why it matters

Autoregressive generation defines the latency profile of every chat product. Output length and tokens-per-second drive both user experience and cost. Buyers comparing two LLM APIs at the same per-token price can still see different bills, because tokenizers split the same text into different numbers of tokens and models differ in how long their answers run.

Example

When ChatGPT writes a 500-token answer, it runs 500 forward passes through the network — each pass conditioned on the prompt plus all previously generated tokens.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Transformer Architecture (Components)
  Transformer architecture is the internal design of a transformer model: a stack of identical blocks, each combining multi-head self-attention, a position-wise feed-forward network, residual connections and layer normalization, arranged as an encoder, a decoder or both. For what the transformer is and why it dominates AI, see Transformer.
• Speculative Decoding
  Speculative decoding is an inference technique where a small "draft" model proposes multiple candidate tokens that the large "target" model verifies in a single forward pass, accepting prefixes that match the target distribution.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Token / Tokenization
  A token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.

More models terms

• Masked Language Model
  A masked language model (MLM) is a model trained to predict tokens that have been hidden in the input, learning bidirectional context rather than left-to-right next-token prediction.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.