Autoregressive Model

Full definition

During training, an autoregressive language model learns to predict token t+1 given tokens 1..t, using a causal attention mask so future tokens cannot leak. At inference, generation is sequential: sample, append, repeat. GPT, Claude, Llama, Mistral, and Gemini-Pro are all autoregressive decoder-only transformers. The sequential nature is the root cause of inference latency — speculative decoding and continuous batching exist precisely to mitigate it.

Why it matters

Autoregressive generation defines the latency profile of every chat product. Output length and tokens-per-second drive both user experience and cost. Buyers comparing two LLM APIs at the same per-token price can see 3x cost differences in practice because of throughput.

Example

When ChatGPT writes a 500-token answer, it runs 500 forward passes through the network — each pass conditioned on the prompt plus all previously generated tokens.

Related terms

• Transformer Architecture
  The transformer architecture is a neural-network design built on stacked self-attention and feed-forward layers, with no recurrence or convolution, that processes sequences in parallel.
• Speculative Decoding
  Speculative decoding is an inference technique where a small "draft" model proposes multiple candidate tokens that the large "target" model verifies in a single forward pass, accepting prefixes that match the target distribution.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Token / Tokenization
  A token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.