Reinforcement Learning from Human Feedback (RLHF)
Reinforcement Learning from Human Feedback (RLHF) is a three-stage post-training pipeline: supervised fine-tuning, reward-model training on human preference comparisons, and policy optimization (typically PPO) against that reward model.
Full definition
Stage 1: SFT on demonstrations. Stage 2: collect pairwise preferences (response A vs B), train a reward model to predict human preference. Stage 3: optimize the SFT model with PPO using the reward model as the reward signal, with a KL penalty back to the SFT policy to prevent reward hacking. Direct Preference Optimization (DPO, Rafailov et al. 2023) skips the explicit reward model and trains directly on preferences. RLHF is what makes GPT-4 and Claude usable as products rather than research artifacts.
Why it matters
RLHF is the most expensive part of post-training (preference data costs $5-50/comparison from skilled labelers). Vendors that claim "RLHF" without specifying preference dataset size or labeler qualifications are usually doing DPO on synthetic preferences — the quality difference is large.
Example
Anthropic's Claude is trained with RLHF and Constitutional AI: the reward signal mixes human preferences with AI-generated critiques against a written constitution.
Related terms
- Reinforcement Learning from Human Feedback (RLHF)Reinforcement Learning from Human Feedback (RLHF) is a training technique that aligns a language model's outputs with human preferences by training a reward model on human comparisons of candidate responses, then optimizing the LLM against that reward.
- Supervised Fine-Tuning (SFT)Supervised Fine-Tuning (SFT) is the stage of LLM post-training where a pretrained model is fine-tuned on curated input-output pairs, typically instruction-response demonstrations written by humans.
- Constitutional AIConstitutional AI (CAI) is a training technique, introduced by Anthropic in 2022, in which a model critiques and revises its own outputs against a written set of principles, then learns from those revisions instead of from human-labeled preferences alone.
- Refusal TrainingRefusal training is the post-training step that teaches a model to decline requests that violate its policies (illegal, harmful, privacy-invasive, etc.) — and, critically, to comply with the much larger set of legitimate requests that superficially resemble refused ones.
Source & further reading
Primary source: Ouyang et al. — "Training language models to follow instructions with human feedback" (InstructGPT) (2022).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/no/glossary/reinforcement-learning-human-feedback/.