Reinforcement Learning from Human Feedback (RLHF)

Full definition

RLHF has three stages: (1) supervised fine-tuning on demonstrations, (2) training a reward model on human-ranked pairs of outputs, and (3) RL (typically PPO or DPO) to push the model toward higher-reward outputs. RLHF is the technique that turned raw GPT-3 into ChatGPT — taking a model that completed text into one that follows instructions, refuses unsafe requests, and behaves like an assistant.

Why it matters

RLHF (and successors like Constitutional AI and DPO) is what makes frontier LLMs safe and useful enough to deploy in regulated environments. Without alignment training, the same base models are not enterprise-deployable.

Example

OpenAI used RLHF to turn GPT-3.5 into ChatGPT in 2022, producing the first LLM mass enough to reach 100 million users in two months.

Related terms

• Fine-tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Pretraining
  Pretraining is the first, most compute-intensive stage of training a foundation model — typically self-supervised next-token prediction over trillions of tokens of text, code, and other modalities — that produces the base model later fine-tuned for specific tasks.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.