Active Learning

Full definition

Standard query strategies: uncertainty sampling (lowest-confidence predictions), margin sampling (smallest gap between top two classes), query-by-committee (disagreement among an ensemble), and expected model change. Active learning typically reaches a target accuracy with 30-70% fewer labels than random sampling. Risks: bias amplification (the model keeps choosing similar uncertain examples), and calibration failure (the model is wrong-but-confident on a region it has never queried).

Why it matters

For enterprises with limited labeling budgets, active learning is the single highest-ROI process change. Combining active learning with weak supervision can reduce labeling cost by 5-10x on complex domains.

Example

A medical-coding team uses active learning on a 200k-record corpus; after labeling 12,000 model-selected examples, F1 reaches 0.91 — random sampling required 38,000 labels for the same score.

Related terms

• Data Labeling
  Data labeling is the process of attaching ground-truth annotations to raw data — text, images, audio — so a supervised model can learn from it, ranging from yes/no classification to structured extraction to multi-turn preference comparisons.
• Weak Supervision
  Weak supervision is a paradigm for training models on labels generated programmatically by labeling functions, rules, regex, knowledge bases, or distant heuristics — rather than by hand — and then learning to denoise them.
• Training Data
  Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.
• Data Augmentation
  Data augmentation is the set of techniques that synthetically expand a training dataset by applying label-preserving transformations to existing examples — flips, crops, paraphrases, back-translation, mixup — improving generalization without new labels.