Model Throughput

Full definition

Throughput trades against latency: large batch sizes raise throughput but increase TPOT, while small batches do the opposite. Continuous batching, paged attention, and speculative decoding all push the throughput-latency Pareto frontier outward. Reported numbers should always specify batch composition, input/output length distribution, and hardware. A serving engine that does 4,000 tok/s on synthetic uniform requests may do 800 tok/s on production traffic.

Why it matters

Cost-per-million-tokens equals (GPU $/hour) ÷ (throughput in millions of tokens/hour). A 5x throughput improvement is a 5x cost reduction — which is why model-serving optimization is one of the highest-ROI engineering investments in AI.

Example

A vendor running Llama 3 70B on 8xH100 reports 6,000 tok/s aggregate throughput at TPOT P95 = 50ms. At $25/hr GPU rental, that is $0.29 per million output tokens — competitive with API providers.

Related terms

• Model Latency
  Model latency is the time between request and response for a model inference call. For LLMs it decomposes into time-to-first-token (TTFT, dominated by the prefill stage) and time-per-output-token (TPOT, dominated by autoregressive decoding).
• Tail Latency (P95/P99)
  Tail latency is the latency at high percentiles (P95, P99, P99.9) of the response-time distribution — the part of the experience that drives user frustration and SLA violations, even when median latency is fine.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.