Alignment Tax
The alignment tax is the performance cost — measured on capability benchmarks — of making a model safer, more honest, or more aligned with human values, relative to the same model trained only for capability.
Full definition
Coined in the InstructGPT paper (Ouyang et al., 2022), the alignment tax shows up as small drops in benchmarks like MMLU or BBH after RLHF, and as refusals on legitimate-but-edgy queries. Modern training recipes have cut the tax dramatically — Claude 3.5/4 and GPT-4 sometimes show negative tax (alignment improves capability) on real-world tasks, even if synthetic benchmarks dip. Buyers should be skeptical of vendors who report only base-model capability scores without post-alignment evaluation.
Why it matters
Alignment tax is at the center of the "safe vs useful" debate. Procurement teams need both numbers — pre- and post-alignment — to evaluate fitness for purpose, especially in regulated or customer-facing deployments.
Example
An enterprise tests two models on legal-summarization: Model A scores 78 raw / 76 aligned; Model B scores 81 raw / 70 aligned. Model A wins despite lower headline capability because the alignment tax matters more in production.
Related terms
- Reinforcement Learning from Human Feedback (RLHF)Reinforcement Learning from Human Feedback (RLHF) is a training technique that aligns a language model's outputs with human preferences by training a reward model on human comparisons of candidate responses, then optimizing the LLM against that reward.
- Reinforcement Learning from Human Feedback (RLHF)Reinforcement Learning from Human Feedback (RLHF) is a three-stage post-training pipeline: supervised fine-tuning, reward-model training on human preference comparisons, and policy optimization (typically PPO) against that reward model.
- Refusal TrainingRefusal training is the post-training step that teaches a model to decline requests that violate its policies (illegal, harmful, privacy-invasive, etc.) — and, critically, to comply with the much larger set of legitimate requests that superficially resemble refused ones.
- Constitutional AIConstitutional AI (CAI) is a training technique, introduced by Anthropic in 2022, in which a model critiques and revises its own outputs against a written set of principles, then learns from those revisions instead of from human-labeled preferences alone.
Source & further reading
Primary source: Ouyang et al. — "Training language models to follow instructions with human feedback" (InstructGPT) (2022).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/nl/glossary/alignment-tax/.