A/B Testing for ML

Full definition

A/B testing closes the gap between offline metrics (AUC, RMSE) and online outcomes (revenue per user). Best practice: pre-register the primary metric and minimum detectable effect, randomize at the unit of analysis (user, session), run for at least one full business cycle, and adjust for novelty effects. Microsoft's ExP team has published canonical guidance on common online-experiment pitfalls.

Why it matters

A model that wins offline can lose online — and vice versa. Teams that ship without A/B tests routinely degrade the metrics they claim to be improving. Mature AI organizations make A/B testing the only path to production for any model that touches user-facing decisions.

Example

A marketplace ramps a new ranking model to 5%, then 50% of users. The new model improves offline NDCG by 3% but reduces sessions-per-user by 1.4% — the team rolls back and investigates the diversity collapse.

Related terms

• Canary Deployment
  A canary deployment routes a small slice of production traffic (e.g., 1-5%) to a new model version while the majority continues hitting the stable version, surfacing latency, error, and quality regressions before full rollout.
• Shadow Deployment
  A shadow deployment sends production traffic to a new model in parallel with the live model but never returns the new model's predictions to end users — allowing teams to validate latency, cost, and prediction distribution under real load with zero user risk.
• Model Evaluation
  Model evaluation is the systematic measurement of an AI model's performance, safety, and behavior across representative tasks — using fixed benchmarks, golden datasets, human ratings, and LLM-as-judge methods — both before and after deployment.
• MLOps
  MLOps is the discipline of operating machine-learning and AI systems reliably in production — covering data pipelines, model training, deployment, monitoring, drift detection, governance, and incident response — analogous to DevOps for traditional software.