Label Noise

Full definition

Northcutt et al. (Cleanlab, 2021) audited 10 popular ML benchmarks (ImageNet, CIFAR-100, MNIST, IMDB, etc.) and found 3.4% mean label-error rates — often enough to flip benchmark rankings between models. Mitigations: confident-learning to find label errors, multi-annotator labeling with adjudication, model-disagreement audits, and Cleanlab/Snorkel tools. Eval-set label noise is especially toxic — your reported accuracy ceiling may be the noise floor of your labels, not your model.

Why it matters

Many "the model is plateauing" failures are actually label-noise problems. Spending a week auditing 500 errors with Cleanlab often beats spending a quarter on architecture changes.

Example

A team's named-entity model sits at 87% F1 for 6 weeks; a Cleanlab audit reveals 9% of test labels are wrong. After re-labeling, both ceiling and observed performance jump to 94%.

Related terms

• Data Labeling
  Data labeling is the process of attaching ground-truth annotations to raw data — text, images, audio — so a supervised model can learn from it, ranging from yes/no classification to structured extraction to multi-turn preference comparisons.
• Golden Dataset
  A golden dataset is a curated, expert-validated set of inputs and reference outputs that serves as the canonical benchmark for evaluating an AI model on a specific task — the team's "ground truth" for is-this-shippable decisions.
• Model Evaluation
  Model evaluation is the systematic measurement of an AI model's performance, safety, and behavior across representative tasks — using fixed benchmarks, golden datasets, human ratings, and LLM-as-judge methods — both before and after deployment.
• Training Data
  Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.