Reinforcement Learning from Human Feedback (RLHF)

Full definition

RLHF has three stages: (1) supervised fine-tuning (SFT) on demonstrations; (2) collecting pairwise human preferences (response A vs B) and training a reward model to predict them; (3) optimizing the SFT model with reinforcement learning (typically PPO) against that reward model, with a KL penalty back to the SFT policy to prevent reward hacking. Direct Preference Optimization (DPO, Rafailov et al. 2023) skips the explicit reward model and trains directly on the preferences. RLHF is the technique that turned raw GPT-3 into InstructGPT, and a GPT-3.5 model into ChatGPT — taking a model that completed text into one that follows instructions, refuses unsafe requests, and behaves like an assistant.

Why it matters

RLHF (and successors like Constitutional AI and DPO) is what makes frontier LLMs safe and useful enough to deploy in regulated environments. Without alignment training, the same base models are not enterprise-deployable. When a vendor says it uses RLHF, ask for the size and source of its preference data: DPO on synthetic preferences is a different, cheaper process.

Example

OpenAI used RLHF to turn GPT-3.5 into ChatGPT in 2022, producing the first mass-market LLM assistant.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Fine-Tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Supervised Fine-Tuning (SFT)
  Supervised Fine-Tuning (SFT) is the stage of LLM post-training where a pretrained model is fine-tuned on curated input-output pairs, typically instruction-response demonstrations written by humans.
• Constitutional AI
  Constitutional AI (CAI) is a training technique, introduced by Anthropic in 2022, in which a model critiques and revises its own outputs against a written set of principles, then learns from those revisions instead of from human-labeled preferences alone.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Pretraining
  Pretraining is the first, most compute-intensive stage of training a foundation model — typically self-supervised next-token prediction over trillions of tokens of text, code, and other modalities — that produces the base model later fine-tuned for specific tasks.

More training & optimization terms

• Transfer Learning
  Transfer learning is the practice of using a model trained on one task as the starting point for a related task, dramatically reducing the data and compute required to reach high accuracy on the new task.
• Few-Shot / Zero-Shot Learning
  Few-shot learning is the ability of a model to perform a new task given only a handful of examples at inference time; zero-shot learning is the same idea with no examples — the model relies entirely on its pretraining and the natural-language instruction.