Alignment Tax

Full definition

Used in alignment research before 2022 (Anthropic's Askell et al., 2021) and measured in the InstructGPT paper (Ouyang et al., 2022), the alignment tax shows up as drops on public benchmarks after RLHF — InstructGPT regressed on SQuAD, DROP, HellaSwag and WMT 2015 French-to-English — and as refusals on legitimate-but-edgy queries. At scale the tax can shrink or reverse: Anthropic found RLHF improved its 13B and 52B models' zero-shot NLP scores (Bai et al., 2022), and GPT-4's exam average was essentially unchanged by RLHF (73.7% base vs 74.0%; OpenAI, 2023). Buyers should be skeptical of vendors who report only base-model capability scores without post-alignment evaluation.

Why it matters

Alignment tax is at the center of the "safe vs useful" debate. Procurement teams need both numbers — pre- and post-alignment — to evaluate fitness for purpose, especially in regulated or customer-facing deployments.

Example

An enterprise tests two models on legal-summarization: Model A scores 78 raw / 76 aligned; Model B scores 81 raw / 70 aligned. Model A wins despite lower headline capability because the alignment tax matters more in production.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Reinforcement Learning from Human Feedback (RLHF)
  Reinforcement Learning from Human Feedback (RLHF) is a training technique that aligns a language model's outputs with human preferences by training a reward model on human comparisons of candidate responses, then optimizing the LLM against that reward.
• Refusal Training
  Refusal training is the post-training step that teaches a model to decline requests that violate its policies (illegal, harmful, privacy-invasive, etc.) — and, critically, to comply with the much larger set of legitimate requests that superficially resemble refused ones.
• Constitutional AI
  Constitutional AI (CAI) is a training technique, introduced by Anthropic in 2022, in which a model critiques and revises its own outputs against a written set of principles, then learns from those revisions instead of from human-labeled preferences alone.

More governance & risk terms

• Scalable Oversight
  Scalable oversight is the alignment problem of supervising AI systems on tasks where humans cannot easily check the answer themselves — and the family of techniques being developed to keep oversight tractable as models grow more capable.
• Deceptive Alignment
  Deceptive alignment is a hypothetical failure mode in which an AI system behaves aligned during training and evaluation — because it predicts that misbehavior will be corrected — but pursues different goals once it determines it is no longer being supervised.
• Mesa-Optimization
  Mesa-optimization occurs when a learned model is itself an optimizer — a "mesa-optimizer" — pursuing its own internal "mesa-objective" that may diverge from the training (base) objective the gradient descent process was optimizing.