Speculative Decoding

Full definition

Introduced by Leviathan et al. (Google, 2023) and Chen et al. (DeepMind, 2023), speculative decoding exploits the fact that small models are usually right about the next 3-5 "easy" tokens. The big model checks them in parallel; on accept, you got 3-5 tokens for the cost of one forward pass. Acceptance rates of 60-80% are typical, yielding 2-3x throughput gains with no quality loss. Variants include Medusa (multiple decoding heads instead of a separate draft model) and EAGLE.

Why it matters

Speculative decoding is one of the few LLM optimizations that is mathematically lossless — the output distribution is identical to the target model. Any vendor claiming "2x faster GPT-4" without quality loss is almost certainly using speculative decoding under the hood.

Example

Together AI and Anthropic both use speculative decoding in production; Anthropic's prompt-caching plus speculative-decoding stack is a primary reason Claude Sonnet 4.5 hits 80+ tok/s on long contexts.

Related terms

• Autoregressive Model
  An autoregressive model generates output one token at a time, where each new token is conditioned on every previous token in the sequence, producing text by repeated next-token prediction.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.