Data Labeling

Full definition

Modalities span single-label classification, bounding boxes, segmentation, named-entity tagging, span extraction, and pairwise preference comparisons (for RLHF). Quality controls include double labeling with adjudication, golden questions, inter-annotator agreement (Cohen's κ ≥ 0.7 is a common bar), and labeler training. Vendors include Scale AI, Surge, Labelbox, Snorkel, Toloka. Costs rise steeply from simple crowd classification to expert preference comparisons.

Why it matters

Labeling is a major line item in most fine-tuning budgets. Paying more per label for expert labelers often beats buying far more cheap labels — model gains compound when the underlying labels are right. CFOs should challenge any AI budget that does not call out labeling cost explicitly.

Example

A bank pays $400k for 18,000 expert-labeled (regulatory-clause, classification) pairs; the resulting fine-tuned model passes audit at 96% vs 71% for a cheaper crowdsourced alternative.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Training Data
  Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.
• Active Learning
  Active learning is a labeling strategy in which the model selects which unlabeled examples a human should label next — typically the cases the current model is least certain about — focusing labeling effort on the most informative examples.
• Weak Supervision
  Weak supervision is a paradigm for training models on labels generated programmatically by labeling functions, rules, regex, knowledge bases, or distant heuristics — rather than by hand — and then learning to denoise them.
• Golden Dataset
  A golden dataset is a curated, expert-validated set of inputs and reference outputs that serves as the canonical benchmark for evaluating an AI model on a specific task — the team's "ground truth" for is-this-shippable decisions.

More training & optimization terms

• Data Augmentation
  Data augmentation is the set of techniques that synthetically expand a training dataset by applying label-preserving transformations to existing examples — flips, crops, paraphrases, back-translation, mixup — improving generalization without new labels.