Ask three people to rank the same two answers to a question and you will usually get a split decision. Ask three thousand, tally the preferences, and something more useful appears: a rough map of what people actually want from a reply. That map is the raw material behind reinforcement learning from human feedback, or RLHF.
RLHF sits late in the training process, after a model has already absorbed a great deal of text. That earlier stage produces fluency and breadth. It does not produce judgment about which of several plausible replies a person would rather receive. RLHF supplies that judgment by treating human comparisons as the training signal.
What RLHF Actually Does
The recipe rests on three pieces: a starting model that already follows instructions reasonably well, a reward model that scores responses, and an optimization loop that nudges the starting model toward higher-scoring behavior.
The reward model is the interesting part. It is trained on comparisons rather than absolute scores. Given a prompt and two candidate answers, a person marks which one is better. The reward model learns to assign a higher number to the preferred answer and a lower one to its rival. Absolute ratings — "rate this from one to ten" — tend to be noisy and inconsistent between annotators, while pairwise choices are easier to make and more stable to learn from.
Once the reward model is reasonably accurate, the main model is updated to earn higher scores. Left alone, that process will happily produce strange, padded or flattering text, because the reward model is an imperfect stand-in for real preferences. A constraint keeps the updated model close to the original one, so improvements arrive without the model drifting into nonsense that merely games the scorer.
Why Human Ratings Matter
Written instructions can specify facts, format and limits. They struggle to specify taste: when to be brief, when to ask a clarifying question, how much caveat is appropriate, when to say "I don't know" instead of guessing. Those calls are easier to demonstrate than to define.
Preference data captures them indirectly. When hundreds of people, all working from the same guidelines, consistently prefer one style of answer over another, that pattern becomes learnable signal rather than one person's opinion.
The limits are real
Ratings are noisy. Annotators get tired, interpret guidelines differently, and bring their own assumptions. A single label proves almost nothing; a large, carefully curated set of comparisons across varied prompts does. The quality of the guidelines usually shapes results more than the sheer number of labels collected.
The Pipeline, Step by Step
- Write the guidelines first. Define what a good answer looks like, with examples and, importantly, reasons behind each example. Annotators need to apply the same standard, not their personal taste.
- Collect prompts that reflect real use. Pull from actual queries where possible, and cover the awkward cases — ambiguous requests, sensitive topics, requests that should be declined.
- Generate several candidate responses from the current model for each prompt, so there is something meaningful to compare.
- Collect preferences. Two-way comparisons are the workhorse. Ranking four or five answers at once gives more signal per prompt but is harder to do well.
- Train the reward model on those comparisons, holding back a slice to check it generalizes to prompts it has not seen.
- Optimize the policy against the reward model, with a penalty for drifting too far from the reference model.
- Evaluate with fresh prompts and human review. If the automated reward climbs while human raters see no improvement, you are overfitting to the reward model.
Three Failure Modes Worth Knowing
Reward hacking
The model discovers whatever the reward model likes, which is not always what people like. Longer answers, tidy bullet lists and confident phrasing are common shortcuts. Detect it by comparing reward scores against human judgments on held-out prompts, and by testing pairs matched for length.
Sycophancy
If the preference data rewards agreement, the model learns to agree. Flattery scores well in the moment. Counter it by deliberately including prompts where the user is mistaken and the correct response is a polite correction, then rewarding that behavior.
Guideline drift
Over weeks of annotation, standards quietly shift. Recalibrate with shared examples, measure how often annotators agree on the same pair, and re-check a sample of older items against current practice.
How It Fits Beside Other Methods
The training sequence usually looks like this: pretraining on large text corpora, supervised fine-tuning on curated examples of good behavior, then preference tuning. RLHF with a reward model is one way to handle that last step.
There are cheaper alternatives. Direct preference optimization fits the model straight onto preference pairs without a separate reward model, which removes a moving part and can be more stable. Rejection sampling is simpler still: generate several answers, keep the best ones according to a reward model or a written rubric, then fine-tune on those. Approaches that have a model critique its own output against explicit principles can cut down the need for fresh human labels, though they inherit whatever those principles miss.
None of these fixes a weak base model or vague guidelines. Preference tuning shapes behavior; it does not add knowledge the model never had.
Getting Started Without a Large Budget
Pick one narrow task — support replies, code explanations, summaries of a specific document type — and stay there long enough to learn whether the method works at all.
- Draft ten to twenty example comparisons yourself, with one sentence explaining each choice. That is your calibration set.
- Annotate a few hundred pairs before scaling up. If annotators disagree often, the guidelines are ambiguous, not the annotators.
- Keep a frozen evaluation set that nobody trains on, and check it after every round.
- Test for length bias by pairing answers of similar length and seeing which one wins.
- Log every change — data, guidelines, reward model, settings — so a regression can be traced to its cause.
Treat the guidelines as a product, not paperwork. They decide what the reward model learns, and the reward model decides what your model becomes. Get them right at the start and the rest of the process gets much less mysterious.
Photo: Yan Krukau / Pexels
