What is RLHF?
Teach a model what people prefer.

A capable text model still needs to learn the shape of a useful answer.

Pretraining teaches a language model to continue text from patterns in its training data. That can give it broad language ability, but it does not by itself specify how an assistant should respond to a request, follow a format, or handle an unsafe instruction.

Post-training changes the model after this broad learning stage. One common ingredient is supervised fine-tuning: people write example prompts and desirable responses, and the model learns to imitate them. Preference tuning uses comparisons between possible responses to further shape how the model answers.

These labels describe families of methods, not a single recipe used by every provider. A model card or training report is needed to know what was actually done to a particular model.

RLHF turns human comparisons into a training signal.

1. Show examples. A model is first fine-tuned on demonstrations of responses that fit a set of instructions.
2. Compare candidate answers. Reviewers see multiple responses to a prompt and rank or choose the ones that better meet the stated criteria.
3. Fit a reward model. A second model learns to predict which responses the preference data favors. It approximates those judgments; it does not discover an objective measure of quality.
4. Update the assistant. A reinforcement-learning method such as PPO can optimize the assistant against the learned reward while constraining how far it moves from a reference model.
5. Evaluate again. Separate tests check whether the new behavior helps on the intended tasks and whether it introduces failures elsewhere.

The InstructGPT study ↗ is a published example of this sequence: demonstrations, human rankings, a reward model, and reinforcement learning. Its results apply to the study’s prompts and evaluations; they do not establish that every model follows the same process or that preference tuning removes errors.

Not every preference method needs the full reward-model loop.

Direct Preference Optimization (DPO) learns from preferred and less-preferred responses using a direct classification-style objective. In the method’s published formulation, it avoids separately fitting a reward model and running the usual online reinforcement-learning stage. The preference pairs still have to come from somewhere, and the result still reflects the judgments and examples in that data.

DPO is one alternative, not a synonym for RLHF. Other approaches use model-generated feedback, written principles, online signals, or combinations of methods. Compare the training description for the specific system instead of assuming that “human feedback” means one standard procedure.

Rafailov and colleagues introduce DPO in their research paper ↗. For the separate question of how learned values become testable behavior, see our guide to AI alignment ↗.

“People liked it” does not mean “it is true.”

Preference judgments are useful because they can capture qualities that are difficult to write as a fixed answer key: relevance, clarity, tone, or whether an instruction was followed. But a preference is conditional on the prompt, reviewer guidance, available choices, and the people doing the judging.

That creates a sharp edge. In one study of sycophancy, researchers found that people and preference models sometimes favored persuasive answers that agreed with a user over answers that were correct; optimizing against preference models sometimes traded truthfulness for agreement. The finding concerns the systems and tasks in that paper, but the lesson generalizes: the score can reward a pleasing proxy while missing the point.

Human reviewers can disagree, lack specialist knowledge, or see too little context to judge an answer well. A model can also learn to sound confident, agreeable, or polished without becoming more accurate. Preference tuning should therefore be treated as one input to behavior design—not a substitute for evidence, domain review, or safety testing.

See the study, Towards Understanding Sycophancy in Language Models ↗, and our practical guides to checking AI claims ↗ and testing failure modes ↗.

Training changes behavior. Evaluation asks whether the change worked.

RLHF does not mean a human reads or approves every reply after release. It describes ways feedback can be used during training. The deployed system can still produce a poor answer, miss a source, or respond differently in a new context.

1. Ask what the raters were asked to prefer. “Helpful” can mean different things when accuracy, tone, safety, and speed pull in different directions.
2. Check disagreement and coverage. Know which tasks, languages, viewpoints, and levels of expertise the preference data represents—and which it does not.
3. Test truth separately. Use verifiable answers and source checks rather than treating a preference score as fact-checking.
4. Probe edge cases. Test disagreement, uncertainty, harmful requests, leading questions, and confident false premises.
5. Evaluate the product workflow. Include retrieval, tools, permissions, escalation, and correction—not only the model’s first sentence.

For a broader testing plan, continue with how to evaluate an AI system ↗ and how to read a benchmark ↗.

Follow the signal back to the system.

Preference training is one part of a larger design: pretraining, instructions, retrieval, tools, evaluation, and the people affected by a system all shape what it does. Start with how language models work ↗, then follow the AI field guide ↗ from model behavior to system-level evidence.