Training & Alignment

Direct Preference Optimization (DPO)

Direct Preference Optimization (DPO) is a method that aligns a language model to human preference data by training directly on chosen-versus-rejected response pairs, replacing the separate reward model and reinforcement learning loop of classic RLHF with a single classification-style loss.

What Direct Preference Optimization is

Direct Preference Optimization (DPO) is a training method for aligning a language model with human preferences. It was introduced by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn in the 2023 paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (arXiv:2305.18290). The paper was named an Outstanding Main Track Runner-Up at NeurIPS 2023.

The setup mirrors reinforcement learning from human feedback (RLHF): a dataset of prompts is paired with two candidate responses each, one labeled chosen (preferred by a human or judge) and one labeled rejected. The goal is to fine-tune the model so it assigns higher probability to the kinds of responses people prefer. Where DPO differs is the mechanism. Classic RLHF first trains a separate reward model to score responses, then optimizes the language model against that reward with a policy-gradient algorithm such as Proximal Policy Optimization (PPO). DPO skips both stages.

DPO's central result is that the reward model in RLHF can be re-parameterized so that the optimal policy has a closed-form relationship to the reward. Inverting that relationship lets the reward be written implicitly in terms of the policy itself, which is why the paper's subtitle calls the language model "secretly a reward model." The preference-fitting objective then collapses into a simple binary classification loss over the chosen and rejected pairs, with no reward model and no on-policy sampling required during fine-tuning.

Aligns a model to preference pairs (chosen vs. rejected) instead of supervised target outputs.
Removes the explicit reward model and the reinforcement learning step used in classic RLHF.
Trains the policy directly with a classification-style loss derived from preference data.
Introduced by Rafailov et al. (2023), arXiv:2305.18290, a NeurIPS 2023 Outstanding Main Track Runner-Up.

From the RLHF objective to a closed-form policy

Classic RLHF maximizes expected reward while staying close to a reference policy, usually the supervised fine-tuned model, via a Kullback-Leibler (KL) penalty. The objective is to find a policy that earns high reward from the learned reward model without drifting too far from the reference. This KL-constrained reward maximization problem has a known optimal solution: the optimal policy is the reference policy reweighted by the exponential of the reward, divided by a normalizing constant called the partition function.

The partition function sums over all possible responses and is intractable to compute directly, which is why standard RLHF resorts to reinforcement learning rather than solving the problem in closed form. DPO's key move is algebraic. The closed-form expression can be rearranged to express the reward as a function of the optimal policy and the reference policy, plus a partition-function term that depends only on the prompt.

Because preferences are modeled with the Bradley-Terry model, which depends only on the difference between two rewards, the awkward partition-function term cancels out when comparing a chosen response against a rejected one. What remains is a reward expressed purely through log-probability ratios of the policy relative to the reference. Substituting that expression into the Bradley-Terry preference likelihood yields the DPO loss, which the policy can minimize directly with gradient descent.

max_{π} E_{x, y~π}[ r(x,y) ] - β D_KL[ π(y|x) || π_ref(y|x) ]

The KL-constrained reward-maximization objective that DPO inherits from RLHF. The parameter β controls how far the policy π may move from the reference policy π_ref.

r(x,y) = β log ( π_θ(y|x) / π_ref(y|x) ) + β log Z(x)

The implicit reward that falls out of the closed-form optimal policy. The partition-function term β log Z(x) depends only on x, so it cancels in any same-prompt pairwise comparison.

The KL-constrained RLHF objective has a closed-form optimal policy: reference times exp(reward), normalized.
Its intractable partition function is what forces classic RLHF to use reinforcement learning.
Rearranging the closed form expresses reward through policy-to-reference log-probability ratios.
The partition-function term cancels in the Bradley-Terry pairwise comparison, leaving a tractable loss.

The DPO loss

The DPO loss is a maximum-likelihood objective under the Bradley-Terry preference model. For each training example, a prompt x has a chosen (winning) response y_w and a rejected (losing) response y_l. The model computes, for both responses, the log ratio of its own probability to the reference model's probability. The difference of these two log ratios, scaled by the temperature-like coefficient β, is passed through a sigmoid and a logarithm, exactly the shape of a logistic-regression loss.

Intuitively, the loss increases the policy's probability of the chosen response and decreases its probability of the rejected response, but always measured relative to the frozen reference model. The β coefficient sets how aggressively the policy is allowed to diverge from that reference. Larger β keeps the model closer to its starting point; smaller β permits larger shifts in behavior.

The quantity β log ( π_θ(y|x) / π_ref(y|x) ) is the implicit reward that DPO never trains separately. At evaluation time, libraries report it as the reward assigned to chosen and rejected responses, and the gap between them as the reward margin. A positive margin means the model now prefers the chosen response, which is the signal DPO is built to grow.

L_DPO(π_θ; π_ref) = - E_{(x, y_w, y_l)} [ log σ( β log (π_θ(y_w|x) / π_ref(y_w|x)) - β log (π_θ(y_l|x) / π_ref(y_l|x)) ) ]

The DPO loss. σ is the logistic sigmoid, π_θ is the policy being trained, π_ref is the frozen reference model, y_w is the chosen response, y_l is the rejected response, and β > 0 controls deviation from the reference.

python

import torch
import torch.nn.functional as F

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps, ref_rejected_logps, beta=0.1):
    # Each *_logps is the summed log-probability of a full response
    # under the policy (pi_theta) or the frozen reference (pi_ref).
    pi_logratios = policy_chosen_logps - policy_rejected_logps
    ref_logratios = ref_chosen_logps - ref_rejected_logps
    logits = pi_logratios - ref_logratios          # difference of log ratios
    loss = -F.logsigmoid(beta * logits).mean()     # -log sigma(beta * logits)
    # implicit rewards, useful for monitoring the margin
    chosen_reward = beta * (policy_chosen_logps - ref_chosen_logps).detach()
    rejected_reward = beta * (policy_rejected_logps - ref_rejected_logps).detach()
    return loss, chosen_reward, rejected_reward

Minimal DPO loss from precomputed log-probabilities, matching the default sigmoid loss used by Hugging Face TRL's DPOTrainer. logsigmoid implements log σ in a numerically stable way.

Each example contributes a logistic loss over chosen and rejected log-probability ratios.
Gradients raise the relative probability of chosen responses and lower that of rejected ones.
β acts like an inverse temperature: higher β keeps the policy nearer the reference model.
The implicit reward β log(π_θ/π_ref) is read out as the chosen/rejected reward at eval time.

DPO versus classic RLHF with PPO

The practical appeal of DPO is engineering simplicity. Classic RLHF is a multi-stage pipeline: supervised fine-tuning, then training a separate reward model on preference data, then running PPO, which repeatedly samples fresh responses from the current policy, scores them with the reward model, and updates the policy. That loop has many moving parts and many hyperparameters, and it can be unstable. DPO replaces the reward-model training and the entire PPO loop with one offline classification loss over a fixed preference dataset.

Removing on-policy sampling is the largest compute saving. PPO must generate new text on every update, an expensive autoregressive operation, and must keep a reward model and often a value model in memory at once. DPO needs only the policy and a frozen reference, evaluates log-probabilities on pre-collected pairs, and never generates during training. The original paper reports that DPO matches or exceeds PPO-based RLHF on sentiment control, summarization, and single-turn dialogue while being substantially simpler to implement and train.

DPO is not free of trade-offs. Because the loss is computed relative to a frozen reference and on a static dataset, results are sensitive to the choice of reference model and to the quality and coverage of the preference data. DPO is an offline method: it cannot collect new comparisons on responses the current model would actually produce, which can leave gaps that online RLHF would fill. A documented failure mode is likelihood displacement, where the probability the model assigns to chosen responses can fall during training even though the chosen-minus-rejected margin grows, a behavior studied in follow-up work such as "Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization" (arXiv:2410.08847).

DPO replaces reward-model training plus the PPO loop with one offline classification loss.
No response generation during training, so it avoids PPO's costly on-policy sampling.
Rafailov et al. report DPO matches or beats PPO-RLHF on sentiment, summarization, and dialogue.
Trade-offs: sensitivity to the reference model and dataset, offline-only data, and likelihood displacement.

Running DPO in practice

The most common open-source implementation is the DPOTrainer in Hugging Face's TRL library, configured through a DPOConfig object. It expects a preference dataset with a prompt, a chosen completion, and a rejected completion per row, and it accepts both standard text and conversational message formats. If no explicit reference model is passed, TRL uses the initial state of the model before training begins as π_ref.

A typical run loads a base or instruction-tuned model, points the trainer at a binarized preference dataset such as UltraFeedback, sets β (a small value like 0.1 is common), and calls train. DPO pairs naturally with parameter-efficient fine-tuning: TRL integrates with the PEFT library so a LoRA adapter can be trained instead of the full model, which cuts memory enough to run alignment on a single GPU.

TRL also exposes alternative loss types under the same trainer, including IPO and several others, reflecting how quickly the preference-optimization landscape grew after DPO. Choosing β, the reference model, and the dataset remains the work that most affects outcomes, so monitoring the logged reward margin and reward accuracy during training is the standard way to confirm the model is learning the intended preference.

python

from datasets import load_dataset
from trl import DPOConfig, DPOTrainer

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

config = DPOConfig(beta=0.1, learning_rate=1e-6, output_dir="dpo-model")
trainer = DPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",  # reference model defaults to this model's initial state
    args=config,
    train_dataset=dataset,
)
trainer.train()

Aligning a small model with DPO using Hugging Face TRL. The dataset supplies prompt/chosen/rejected pairs; beta controls divergence from the reference policy.

Hugging Face TRL provides DPOTrainer and DPOConfig as the standard implementation.
Input is a preference dataset of prompt, chosen, and rejected; π_ref defaults to the pre-training snapshot.
Integrates with PEFT/LoRA so alignment can run on a single GPU.
Track the logged reward margin and reward accuracy to verify the model is learning the preference.

Where DPO sits among preference-optimization methods

DPO started a family of direct, RL-free preference-optimization objectives that vary the loss shape or the data requirements. Identity Preference Optimization (IPO) replaces DPO's logistic transform with a squared-error regression target, which the authors argue reduces overfitting to deterministic preference labels. These later methods share DPO's core idea of optimizing the policy directly on preference signals rather than through a separately trained reward model.

Kahneman-Tversky Optimization (KTO) drops the paired-comparison requirement entirely. Instead of needing a chosen and a rejected response for the same prompt, KTO works with individual responses labeled simply as desirable or undesirable, drawing on prospect theory from behavioral economics. This makes data collection easier when paired preferences are hard to gather.

Odds Ratio Preference Optimization (ORPO) goes further and removes the reference model, folding supervised fine-tuning and preference learning into a single objective and a single forward pass, so only one copy of the model is needed. These variants do not replace DPO so much as occupy different points on the same design space, and DPO remains the reference point against which they are usually measured.

IPO swaps DPO's logistic loss for a regression target to curb overfitting to hard labels.
KTO needs only per-response desirable/undesirable labels, not paired comparisons.
ORPO removes the reference model and merges fine-tuning with preference learning in one pass.
All are direct, reinforcement-learning-free methods that descend from DPO's core reparameterization.

Key takeaways

DPO aligns a language model to preference pairs by training directly on chosen-versus-rejected responses, with no separate reward model and no reinforcement learning loop.
It works by reparameterizing the RLHF reward so the optimal policy has a closed form, making the reward implicit in the policy and reducing alignment to a classification loss.
The DPO loss is a Bradley-Terry maximum-likelihood objective over β-scaled log-probability ratios of the policy relative to a frozen reference model.
Compared with RLHF plus PPO, DPO is simpler, more stable, and cheaper because it never samples new responses during training.
Its trade-offs include sensitivity to the reference model and preference dataset, its offline-only nature, and the likelihood-displacement failure mode.
DPO is the foundation for later RL-free variants such as IPO, KTO, and ORPO, and is implemented in Hugging Face TRL's DPOTrainer.

Frequently asked questions

Classic RLHF trains a separate reward model on preference data, then uses reinforcement learning such as PPO to optimize the policy against that reward, which requires sampling new responses on every update. DPO collapses both stages into a single classification-style loss computed directly on fixed chosen-versus-rejected pairs. It needs no reward model and no on-policy sampling, which makes it simpler and cheaper to run.

Not as a separate trained component. DPO's insight is that the optimal RLHF policy implies an implicit reward equal to β times the log ratio of the policy to a reference model. That implicit reward is never trained on its own; it is read out from the policy itself. This is the meaning of the paper's subtitle, that the language model is secretly a reward model.

β controls how far the trained policy is allowed to move from the frozen reference model, acting like an inverse temperature on the implicit reward. A larger β keeps the model close to its starting behavior, while a smaller β permits bigger shifts toward the preferred responses. A value around 0.1 is a common starting point in libraries like TRL.

Likelihood displacement is a documented failure mode where the probability the model assigns to chosen responses can decrease during DPO training, even though the gap between chosen and rejected responses grows. In some cases probability mass shifts toward unintended responses, which can undermine alignment. Follow-up research, including arXiv:2410.08847, studies its causes and mitigations.

DPO is a strong default when you already have a fixed preference dataset and want a simpler, more stable, lower-compute pipeline without standing up a reward model and an RL loop. PPO-based RLHF can still be preferable when you need online data collection, since it samples and scores fresh responses the current model actually produces, which an offline method like DPO cannot do.

Both are later RL-free preference-optimization methods that descend from DPO. KTO removes the need for paired comparisons, learning from individual responses labeled desirable or undesirable. ORPO removes the reference model and merges supervised fine-tuning with preference learning into a single pass. DPO remains the common reference point these variants are compared against.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free