Model Latency
Model latency is the time between request and response for a model inference call. For LLMs it decomposes into time-to-first-token (TTFT, dominated by the prefill stage) and time-per-output-token (TPOT, dominated by autoregressive decoding).
Full definition
Total latency = TTFT + (output_tokens × TPOT). TTFT scales with input length and KV-cache state; TPOT scales with model size, batch size, and tensor-parallelism config. Targets vary: voice agents need TTFT < 300ms and TPOT < 50ms (>20 tok/s); document summarization tolerates seconds. Optimization techniques: prompt caching (cuts TTFT 5-10x), speculative decoding (cuts TPOT 2-3x), quantization, smaller models for latency-critical paths.
Why it matters
Latency is the single biggest determinant of perceived quality in real-time AI products. Procurement should require TTFT and TPOT P50/P95 numbers under target load — not just "average latency" marketing claims.
Example
A customer-service agent on Claude Sonnet 4.5 with prompt caching: TTFT P50 = 250ms (cached) vs 1.8s (uncached); TPOT = 12ms. Net effect: cached responses feel instant, uncached feel slow.
Related terms
- Model ThroughputModel throughput is the number of tokens (or requests) per second a serving system produces, aggregated across concurrent users — the metric that determines GPU economics.
- Tail Latency (P95/P99)Tail latency is the latency at high percentiles (P95, P99, P99.9) of the response-time distribution — the part of the experience that drives user frustration and SLA violations, even when median latency is fine.
- Speculative DecodingSpeculative decoding is an inference technique where a small "draft" model proposes multiple candidate tokens that the large "target" model verifies in a single forward pass, accepting prefixes that match the target distribution.
- KV CacheThe KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
Source & further reading
Primary source: NVIDIA — "Mastering LLM Techniques: Inference Optimization" (2024).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/sv/glossary/model-latency/.