Data Labeling

Full definition

Modalities span single-label classification, bounding boxes, segmentation, named-entity tagging, span extraction, and pairwise preference comparisons (for RLHF). Quality controls include double labeling with adjudication, golden questions, inter-annotator agreement (Cohen's κ ≥ 0.7 is a common bar), and labeler training. Vendors include Scale AI, Surge, Labelbox, Snorkel, Toloka. Costs range from $0.05 per simple classification to $50+ per complex preference comparison from PhD-level labelers.

Why it matters

Labeling is the largest line item in most fine-tuning budgets. Spending $200k on premium labelers often beats spending $2M on cheap ones — model gains compound when the underlying labels are right. CFOs should challenge any AI budget that does not call out labeling cost explicitly.

Example

A bank pays $400k for 18,000 expert-labeled (regulatory-clause, classification) pairs; the resulting fine-tuned model passes audit at 96% vs 71% for a cheaper crowdsourced alternative.

Related terms

• Training Data
  Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.
• Active Learning
  Active learning is a labeling strategy in which the model selects which unlabeled examples a human should label next — typically the cases the current model is least certain about — focusing labeling effort on the most informative examples.
• Weak Supervision
  Weak supervision is a paradigm for training models on labels generated programmatically by labeling functions, rules, regex, knowledge bases, or distant heuristics — rather than by hand — and then learning to denoise them.
• Golden Dataset
  A golden dataset is a curated, expert-validated set of inputs and reference outputs that serves as the canonical benchmark for evaluating an AI model on a specific task — the team's "ground truth" for is-this-shippable decisions.