Model Throughput

Full definition

Throughput trades against latency: large batch sizes raise throughput but increase TPOT, while small batches do the opposite. Continuous batching, paged attention, and speculative decoding all push the throughput-latency Pareto frontier outward. Reported numbers should always specify batch composition, input/output length distribution, and hardware. A serving engine that does 4,000 tok/s on synthetic uniform requests may do 800 tok/s on production traffic.

Why it matters

Cost-per-million-tokens equals (GPU $/hour) ÷ (throughput in millions of tokens/hour). A 5x throughput improvement is a 5x cost reduction — which is why model-serving optimization is one of the highest-ROI engineering investments in AI.

Example

A vendor running Llama 3 70B on 8xH100 reports 6,000 tok/s aggregate throughput at TPOT P95 = 50ms. At $25/hr for the 8-GPU node, that is about $1.16 per million output tokens ($25 ÷ 21.6M tokens an hour), more than some API providers charge for a 70B model, which is why sustained throughput decides whether self-hosting pays.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Model Latency
  Model latency is the time between request and response for a model inference call. For LLMs it decomposes into time-to-first-token (TTFT, dominated by the prefill stage) and time-per-output-token (TPOT, dominated by autoregressive decoding).
• Tail Latency (P95/P99)
  Tail latency is the latency at high percentiles (P95, P99, P99.9) of the response-time distribution — the part of the experience that drives user frustration and SLA violations, even when median latency is fine.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.

More deployment & operations terms

• Trace Sampling
  Trace sampling is the practice of recording detailed observability traces (spans, prompts, tool calls, token counts) for a fraction of requests, balancing observability cost against debug coverage.
• Prompt Observability
  Prompt observability is the practice of capturing, indexing, and analyzing every prompt, response, tool call, latency, token-count, and quality metric in an LLM application — for debugging, evaluation, audit, and continuous improvement.