Active Learning

Full definition

Standard query strategies: uncertainty sampling (lowest-confidence predictions), margin sampling (smallest gap between top two classes), query-by-committee (disagreement among an ensemble), and expected model change. In most published results, active learning reaches a target accuracy with fewer labels than random sampling, though the saving varies by task and some studies report no gain (Settles, 2010). Risks: bias amplification (the model keeps choosing similar uncertain examples), and calibration failure (the model is wrong-but-confident on a region it has never queried).

Why it matters

For enterprises with limited labeling budgets, active learning is the single highest-ROI process change. Combining active learning with weak supervision can cut labeling cost further.

Example

A medical-coding team uses active learning on a 200k-record corpus; after labeling 12,000 model-selected examples, F1 reaches 0.91 — random sampling required 38,000 labels for the same score.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Data Labeling
  Data labeling is the process of attaching ground-truth annotations to raw data — text, images, audio — so a supervised model can learn from it, ranging from yes/no classification to structured extraction to multi-turn preference comparisons.
• Weak Supervision
  Weak supervision is a paradigm for training models on labels generated programmatically by labeling functions, rules, regex, knowledge bases, or distant heuristics — rather than by hand — and then learning to denoise them.
• Training Data
  Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.
• Data Augmentation
  Data augmentation is the set of techniques that synthetically expand a training dataset by applying label-preserving transformations to existing examples — flips, crops, paraphrases, back-translation, mixup — improving generalization without new labels.

More training & optimization terms

• Label Noise
  Label noise is the presence of incorrect labels in a training or evaluation dataset, which degrades model accuracy, biases evaluation metrics, and is often mistaken for a model-capability ceiling.