Trace Sampling

Full definition

Two strategies: head-based sampling decides at request start (cheap but biased — misses rare errors); tail-based sampling buffers traces and keeps based on outcome (catches errors and slow tails, but more expensive). For LLM apps, the typical configuration is 100% sampling on errors and high-latency requests, plus 1-10% baseline. Tools: OpenTelemetry, Langfuse, Arize Phoenix, Helicone, LangSmith.

Why it matters

Naive 100% trace logging on a high-volume LLM product can cost more than the inference itself, especially when prompts and outputs are long. Smart sampling cuts observability cost 10-100x while keeping the traces that actually matter for debugging.

Example

A SaaS team logs every LLM call to Langfuse and burns $14k/mo on storage; switching to tail sampling (100% errors, 10% baseline) drops cost to $2k/mo with no loss of debugging capability.

Related terms

• Prompt Observability
  Prompt observability is the practice of capturing, indexing, and analyzing every prompt, response, tool call, latency, token-count, and quality metric in an LLM application — for debugging, evaluation, audit, and continuous improvement.
• MLOps
  MLOps is the discipline of operating machine-learning and AI systems reliably in production — covering data pipelines, model training, deployment, monitoring, drift detection, governance, and incident response — analogous to DevOps for traditional software.
• Model Monitoring
  Model monitoring is the continuous measurement of a deployed model's inputs, outputs, and performance signals — including data drift, prediction drift, label drift, latency, and ground-truth accuracy — to detect degradation before it harms business outcomes.
• Tail Latency (P95/P99)
  Tail latency is the latency at high percentiles (P95, P99, P99.9) of the response-time distribution — the part of the experience that drives user frustration and SLA violations, even when median latency is fine.