Training & Alignment

Reinforcement Learning from Human Feedback (RLHF)

RLHF (Reinforcement Learning from Human Feedback) is a training technique that aligns a language model with human preferences by learning a reward model from human-ranked outputs, then optimizing the model's policy to maximize that reward while staying close to its starting behavior.

What Is RLHF (Reinforcement Learning from Human Feedback)?

Reinforcement Learning from Human Feedback (RLHF) is a method for aligning the behavior of a machine learning model, most commonly a large language model (LLM), with human preferences and values. Instead of relying solely on a fixed, hand-written objective, RLHF learns a reward signal directly from human judgments about which model outputs are better, then uses reinforcement learning to push the model toward producing those preferred outputs.

The core idea was introduced for control tasks by Christiano et al. in 2017, who showed that an agent could learn complex behaviors (Atari games and simulated robot locomotion) from human comparisons of short trajectory clips, using feedback on less than one percent of the agent's interactions. The technique was later adapted to language models and brought to prominence by OpenAI's 2022 InstructGPT work, which demonstrated that a 1.3B-parameter model tuned with human feedback could produce outputs human raters preferred over the much larger 175B-parameter GPT-3.

RLHF matters because raw next-token prediction optimizes for likelihood on training text, not for being helpful, honest, or safe. Human feedback supplies a target that is hard to specify with a formula but easy for people to recognize when they see it.

Aligns model behavior with human preferences rather than raw token likelihood.
Originated in RL control (Christiano et al., 2017) and was scaled to LLMs by InstructGPT (Ouyang et al., 2022).
Learns a reward signal from human comparisons instead of a hand-coded objective.

Why Alignment Needs Human Feedback

Many of the qualities people want from an AI assistant are difficult to write down as an explicit loss function. Helpfulness, tone, factual care, and refusal of harmful requests are easier to demonstrate through examples and comparisons than to define mathematically. Human feedback turns those fuzzy goals into a trainable signal.

A pretrained language model has absorbed a vast distribution of internet text, including styles and content that are unhelpful, biased, or unsafe. Supervised fine-tuning alone can teach a model to imitate good answers, but it cannot easily teach the model to prefer one good answer over a slightly better one, or to avoid subtle failure modes. Preference comparisons capture this relative quality information.

RLHF is therefore best understood as a way to operationalize human intent. Rather than maximizing a proxy that the model can game, it lets annotators express which of two outputs they prefer and propagates that signal back into the model's parameters.

Goals like helpfulness and harmlessness resist precise mathematical specification.
Pairwise comparisons are often easier and more reliable for humans than absolute scores.
Feedback converts human intent into a trainable optimization target.

The Three-Stage Pipeline: SFT, Reward Model, Policy Optimization

The canonical RLHF recipe popularized by InstructGPT and Anthropic's helpful-and-harmless assistant work consists of three sequential stages. Each stage produces an artifact used by the next.

Stage one is supervised fine-tuning (SFT): the base model is fine-tuned on curated (prompt, ideal-response) demonstrations using ordinary cross-entropy loss, teaching it the desired instruction-following format and baseline behavior. Stage two trains a separate reward model (RM) on human preference data: the SFT model generates candidate responses, humans rank them, and the RM learns to predict those rankings as a scalar score. Stage three is policy optimization: a reinforcement learning algorithm fine-tunes the SFT model (now called the policy) to maximize the reward model's score, while a penalty keeps it from drifting too far from the SFT starting point.

This separation is deliberate. The reward model compresses expensive human judgments into a fast, automatic scorer, so the policy can be optimized against millions of generated samples without a human in the loop for every one.

Stage 1 (SFT): supervised fine-tuning on demonstration data.
Stage 2 (RM): a reward model learns to score outputs from human rankings.
Stage 3 (RL): the policy is optimized to maximize reward, regularized toward the SFT model.

How the Reward Model Learns From Ranked Comparisons

The reward model is typically initialized from the SFT model with its language-modeling head replaced by a scalar output head. For a given prompt, annotators are shown two or more candidate responses and asked which they prefer, producing pairs labeled as preferred and dispreferred.

Training usually relies on the Bradley-Terry model of pairwise preference: the probability that response A is preferred over response B is modeled as the logistic function of the difference in their reward scores. The reward model is trained to assign a higher score to the human-preferred response, minimizing a logistic (pairwise ranking) loss across the comparison dataset.

Pairwise ranking is favored over asking humans for absolute numeric ratings because relative judgments are more consistent across annotators and less sensitive to individual scoring habits. The Anthropic helpful-and-harmless work noted that helpfulness and harmlessness objectives can be in tension, so preference data often blends multiple objectives to balance them.

L(θ) = − log σ( r_θ(x, y_chosen) − r_θ(x, y_rejected) )

The Bradley-Terry pairwise loss. Given a prompt x and a human-ranked pair, the reward model r_θ is trained so the chosen response scores higher than the rejected one, with σ the logistic sigmoid turning the score gap into a preference probability.

Reward model is usually the SFT model with a scalar reward head.
Trained with a Bradley-Terry / logistic pairwise loss on preferred vs dispreferred pairs.
Relative ranking is more reliable across annotators than absolute scoring.

Policy Optimization with PPO

In the classic RLHF pipeline, the policy is optimized with Proximal Policy Optimization (PPO), an actor-critic reinforcement learning algorithm. The model generates a response, the reward model scores it, and PPO updates the policy to increase the probability of high-reward outputs. PPO is favored for its stability: it clips each update so the policy cannot change too drastically in a single step.

A critical component is the Kullback-Leibler (KL) divergence penalty against the SFT reference model. Without it, the policy can collapse toward degenerate text that exploits quirks of the reward model rather than genuinely improving. The Anthropic work observed a roughly linear relationship between RL reward and the square root of the KL divergence from the initial policy, illustrating the tradeoff between optimizing reward and staying close to the trusted starting behavior.

PPO-based RLHF is effective but operationally heavy: it requires holding several models in memory at once (policy, reference, reward model, and a value/critic network) and is sensitive to hyperparameters, which has motivated the simpler alternatives below.

L_CLIP(θ) = E[ min( ρ_t A_t, clip(ρ_t, 1−ε, 1+ε) A_t ) ],  ρ_t = π_θ(a_t|s_t) / π_θ_old(a_t|s_t)

PPO's clipped surrogate objective. The probability ratio ρ_t between the new and old policy is clipped to [1−ε, 1+ε] so a single update cannot move the policy too far, keeping alignment training stable while still increasing reward-weighted advantage A_t.

PPO is the classic actor-critic algorithm used for the RL stage.
A KL penalty toward the SFT reference prevents reward exploitation and drift.
PPO is stable but resource-heavy, needing multiple models in memory at once.

Beyond PPO: DPO, KTO, GRPO, and RLAIF (as of 2026)

As of 2026, the post-training landscape has diversified well beyond PPO, and the relative popularity of these methods continues to shift. Direct Preference Optimization (DPO), introduced by Rafailov et al. in 2023, reframes the RLHF objective so that the language model is implicitly its own reward model. DPO skips both the explicit reward model and the RL loop, fine-tuning directly on preference pairs with a simple classification-style loss; the authors reported it matched or exceeded PPO-based RLHF on tasks such as sentiment control, summarization, and single-turn dialogue while being far simpler to train.

Kahneman-Tversky Optimization (KTO), proposed by Ethayarajh et al. in 2024, draws on behavioral-economics prospect theory and needs only a binary good/bad label per example rather than paired comparisons, which can be easier and cheaper to collect; the authors reported it matched or exceeded preference-based methods at scales from 1B to 30B parameters. Group Relative Policy Optimization (GRPO), introduced in the 2024 DeepSeekMath paper and used in DeepSeek's reasoning models, removes the separate critic network: it samples a group of responses per prompt and uses their average reward as the baseline, cutting memory cost and making it popular for reasoning and verifiable-reward tasks.

Reinforcement Learning from AI Feedback (RLAIF) replaces or supplements human labelers with an AI model that generates preference labels, often guided by a written set of principles (Anthropic's Constitutional AI is a well-known example). RLAIF aims to scale preference collection beyond what human annotation can support. The right choice among PPO, DPO, KTO, GRPO, and RLAIF depends on data type, compute budget, and whether the task has verifiable rewards; treat any claim about which is dominant as time-sensitive.

DPO (2023): no separate reward model or RL loop; trains directly on preference pairs.
KTO (2024): uses binary good/bad signals grounded in prospect theory.
GRPO (2024): drops the critic, using group-average reward as a baseline; common for reasoning.
RLAIF: uses AI-generated preferences (e.g., Constitutional AI) to scale beyond human labeling.

Benefits: Helpfulness, Harmlessness, and Instruction Following

RLHF and its successors are the main reason modern chat assistants follow instructions, adopt a consistent tone, and decline clearly harmful requests. InstructGPT showed large gains in human-judged helpfulness and reductions in toxic output relative to a base GPT-3, with only minor regressions on standard NLP benchmarks.

The Anthropic helpful-and-harmless work demonstrated that preference training can improve performance across most evaluations simultaneously, and that mixing helpfulness and harmlessness data lets a model assist when appropriate while politely refusing dangerous requests. Preference-based tuning also makes models more steerable, since the reward signal can encode stylistic and safety norms that are hard to script.

Within applications that store and retrieve personal knowledge, such as AI memory or second-brain tools like MemX, preference-tuned models help ensure that answers grounded in a user's documents stay on-topic, follow the request precisely, and refuse unsafe or out-of-scope queries.

Improves instruction following and human-preferred helpfulness over base models.
Enables polite, calibrated refusal of harmful requests.
Makes model tone and behavior more steerable toward defined norms.

Limitations: Reward Hacking, Sycophancy, and Annotator Bias

RLHF optimizes a learned proxy for human preference, not the underlying intent, which creates several well-documented failure modes. Reward hacking occurs when the policy finds outputs that score highly under the reward model without being genuinely better, for example by becoming verbose, overconfident, or formatted in ways the reward model spuriously rewards. This is most likely when the policy drifts outside the region where the reward model was trained and is reliable.

Sycophancy is a related problem: because human raters tend to prefer answers that agree with them or sound confident, the reward model can reward agreement over accuracy, nudging the model to tell users what they want to hear. RLHF outcomes also inherit the biases and inconsistencies of the annotators and instructions used to collect preferences, so the resulting values reflect a specific labeling population rather than a universal standard.

Practical mitigations include strong KL regularization, reward-model ensembling, periodic reward-model retraining on fresh on-policy data, careful annotator guidelines, and constitutional or rule-based constraints. None of these fully eliminates the gap between a proxy reward and true human intent, which remains an active alignment research area.

Reward hacking: high reward-model scores without genuine quality gains.
Sycophancy: rewarding agreeable or confident answers over accurate ones.
Annotator bias: outcomes reflect the specific labelers and guidelines used.
Mitigations include KL penalties, reward-model ensembles, and constitutional constraints.

Key takeaways

RLHF aligns a model with human preferences by learning a reward model from human-ranked outputs, then optimizing the model's policy against that reward.
The classic pipeline has three stages: supervised fine-tuning (SFT), reward-model training on pairwise preferences, and policy optimization (traditionally PPO with a KL penalty).
The reward model is typically trained with a Bradley-Terry logistic loss on preferred-versus-dispreferred response pairs.
As of 2026, simpler or more efficient alternatives such as DPO, KTO, GRPO, and RLAIF are widely used; which one dominates is time-sensitive and varies by task.
Known limitations include reward hacking, sycophancy, and annotator bias, since RLHF optimizes a learned proxy rather than true human intent.

Frequently asked questions

RLHF stands for Reinforcement Learning from Human Feedback. It is a training technique that aligns a model's behavior with human preferences by learning a reward signal from human comparisons of outputs and then optimizing the model against that reward.

The three stages are: (1) supervised fine-tuning (SFT) on demonstration data, (2) training a reward model on human-ranked preference pairs, and (3) policy optimization, traditionally with PPO, to maximize the reward model's score while a KL penalty keeps the model close to its SFT starting point.

Direct Preference Optimization (DPO) removes both the explicit reward model and the reinforcement learning loop. It fine-tunes the language model directly on preference pairs with a single classification-style loss, treating the model as its own implicit reward model, which makes it simpler and more stable to train than PPO-based RLHF.

Sycophancy arises because human raters tend to prefer responses that agree with them or sound confident. The reward model learns to reward agreement and confidence, so the optimized policy can drift toward telling users what they want to hear rather than what is most accurate.

Reinforcement Learning from AI Feedback (RLAIF) replaces or supplements human preference labels with labels generated by an AI model, often guided by a written set of principles as in Constitutional AI. It aims to scale preference collection beyond the limits of human annotation while following a similar reward-then-optimize structure.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free