Speculative Decoding

Full definition

Introduced by Leviathan et al. (Google, 2023) and Chen et al. (DeepMind, 2023), speculative decoding exploits the fact that small models are usually right about the next 3–5 "easy" tokens. The big model checks them in parallel; on accept, you got 3–5 tokens for the cost of one forward pass. Acceptance rates of 60–80% are typical, cutting latency 2–3x at batch size 1 with identical outputs (Leviathan et al., 2023); because the gain relies on spare compute, it shrinks as large batches saturate the GPU. Variants include Medusa (multiple decoding heads instead of a separate draft model) and EAGLE.

Why it matters

Speculative decoding is one of the few LLM optimizations that is mathematically lossless — the output distribution is identical to the target model. Any vendor claiming "2x faster GPT-4" without quality loss is almost certainly using speculative decoding under the hood.

Example

Together AI runs speculative decoding in production: its ATLAS adaptive speculator (October 2025), part of the Together Turbo optimization suite, keeps learning from live traffic on its inference service.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Autoregressive Model
  An autoregressive model generates output one token at a time, where each new token is conditioned on every previous token in the sequence, producing text by repeated next-token prediction.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.

More infrastructure terms

• Paged Attention
  Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.
• Model Parallelism
  Model parallelism is a distributed-training and inference technique that splits a single neural network across multiple devices when the model does not fit on one device, in contrast to data parallelism (which replicates the model).