Tail Latency (P95/P99)

Full definition

In LLM serving, tail latency is driven by KV-cache fragmentation, head-of-line blocking, GPU scheduling stalls, and cold cache misses. Dean and Barroso's "The Tail at Scale" (CACM, 2013) established that tail latency dominates user-perceived performance in distributed systems. Mitigations: continuous batching, request hedging, priority queues, capping max sequence length, and dedicated capacity for premium tiers.

Why it matters

Buyers should never accept "average latency" in vendor SLAs. P95 and P99 are what determine whether a chat product feels reliable or flaky. A vendor with 1.2s mean / 8.0s P99 is far worse than one with 1.5s mean / 2.5s P99 for most user-facing applications.

Example

A chat product with 1.5s mean TTFT and 12s P99 sees a 3x higher user-abandonment rate than competitors at 2.0s / 3.5s — the median is irrelevant to the bouncing 1%.

Related terms

• Model Latency
  Model latency is the time between request and response for a model inference call. For LLMs it decomposes into time-to-first-token (TTFT, dominated by the prefill stage) and time-per-output-token (TPOT, dominated by autoregressive decoding).
• Model Throughput
  Model throughput is the number of tokens (or requests) per second a serving system produces, aggregated across concurrent users — the metric that determines GPU economics.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.
• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.