Scalable Oversight
Scalable oversight is the alignment problem of supervising AI systems on tasks where humans cannot easily check the answer themselves — and the family of techniques being developed to keep oversight tractable as models grow more capable.
AIDOLS Research Team · Editorial standards
Full definition
Approaches include: (1) AI-assisted critique — a second AI flags errors a human can verify; (2) Debate — two AIs argue and a human judges; (3) Recursive reward modeling — break the task into checkable sub-tasks; (4) Weak-to-strong generalization (OpenAI, 2023) — fine-tuning a strong model on a weaker model's labels; the strong model beats its supervisor but, with naive fine-tuning, recovers only part of its full capability (GPT-4 supervised by a GPT-2-level model got close to GPT-3.5-level on NLP tasks with an added confidence loss); (5) Constitutional AI / RLAIF. The "sandwiching" methodology (Cotra) measures whether non-experts using AI assistance can supervise expert-level tasks.
Why it matters
As enterprises deploy AI in domains where employees cannot easily verify outputs (medical coding, legal contract review, multi-step financial analysis), scalable oversight stops being academic. It is the core question behind every "human in the loop, but the human can't actually check" workflow.
Example
A radiology AI flags potential findings; a junior radiologist with AI critique tools achieves the same review quality as a senior radiologist alone — sandwiching working in practice.
Next steps
Related terms
- Constitutional AIConstitutional AI (CAI) is a training technique, introduced by Anthropic in 2022, in which a model critiques and revises its own outputs against a written set of principles, then learns from those revisions instead of from human-labeled preferences alone.
- AI Red-TeamingAI red-teaming is the practice of systematically probing an AI system — using adversarial prompts, jailbreak techniques, and edge-case inputs — to surface harmful, unsafe, biased, or policy-violating behaviors before and after deployment.
- AI GovernanceAI governance is the framework of policies, roles, controls, and processes an organization uses to ensure its AI systems are lawful, safe, fair, accountable, and aligned with business intent — across the full lifecycle from problem framing to retirement.
- Mesa-OptimizationMesa-optimization occurs when a learned model is itself an optimizer — a "mesa-optimizer" — pursuing its own internal "mesa-objective" that may diverge from the training (base) objective the gradient descent process was optimizing.
More governance & risk terms
- Deceptive AlignmentDeceptive alignment is a hypothetical failure mode in which an AI system behaves aligned during training and evaluation — because it predicts that misbehavior will be corrected — but pursues different goals once it determines it is no longer being supervised.
- Refusal TrainingRefusal training is the post-training step that teaches a model to decline requests that violate its policies (illegal, harmful, privacy-invasive, etc.) — and, critically, to comply with the much larger set of legitimate requests that superficially resemble refused ones.
Source & further reading
Primary source: Bowman et al. — "Measuring Progress on Scalable Oversight for Large Language Models" (Anthropic) (2022).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/en/glossary/scalable-oversight/.