After pre-training, language models are usually tuned to follow instructions and match human preferences. The best-known recipe, reinforcement learning from human feedback (RLHF), made ChatGPT-style assistants possible — but it is complex and finicky. In 2023, researchers at Stanford proposed Direct Preference Optimization (DPO), which achieves similar goals with a simple loss function. The paper’s subtitle summarizes its insight: “Your Language Model is Secretly a Reward Model”.

The problem: RLHF is complicated

The classic RLHF pipeline, described in OpenAI’s InstructGPT paper, has three stages:

  1. Supervised fine-tuning on example responses.
  2. Training a reward model on human preference data: people compare two responses to the same prompt and pick the better one; a separate model learns to predict these preferences as a score.
  3. Reinforcement learning (typically with PPO) to optimize the language model to produce high-reward responses, with a penalty for drifting too far from the original model.

Stage 3 is the hard part: it requires generating samples during training, running several models at once, tuning many hyperparameters, and dealing with instability.

The key insight

The authors showed mathematically that the RLHF objective — maximize reward while staying close to a reference model — has a closed-form optimal solution. Rearranging it, the reward can be expressed in terms of the policy itself: how much more likely the trained model makes a response compared with the reference model.

This means you don’t need a separate reward model. You can plug this implicit reward directly into the preference-learning objective and obtain a loss that depends only on the language model being trained and the reference model.

How DPO training works

For each training example you have a prompt, a preferred response and a rejected response. DPO:

  1. computes how likely the current model and the frozen reference model each consider both responses;
  2. increases the relative likelihood of the preferred response and decreases that of the rejected one;
  3. uses a parameter (beta) that controls how far the model may move from the reference.

It is ordinary supervised-style training with a classification-like loss — no sampling during training, no reward model, no reinforcement learning loop.

Results

In the paper’s experiments on sentiment control, summarization and single-turn dialogue, DPO matched or exceeded PPO-based RLHF in the authors’ evaluations, while being substantially simpler to implement and more stable to train. The paper was recognized at NeurIPS 2023 with an outstanding paper runner-up award.

Why it spread so fast

  • Simplicity: a few dozen lines of code on top of standard fine-tuning.
  • Lower compute: no need to run a reward model and generate samples during training.
  • Accessibility: open-source developers could align models without large RL infrastructure.

DPO and its variants quickly became common in open-weight model training pipelines and libraries.

Limitations and follow-ups

  • Offline data: DPO learns from a fixed preference dataset; online methods that generate new samples can sometimes perform better.
  • Overfitting: the model can over-optimize on preference data, and results depend on data quality.
  • Not the end of RL: for reasoning models, reinforcement learning with verifiable rewards — for example checking math answers — has become central, so DPO is one tool among several.

Many variants followed, modifying the loss or removing the need for a reference model or paired data.

Why it matters

DPO changed how practitioners think about alignment: preference learning does not necessarily require a separate reward model and complex RL. It made preference tuning a standard, accessible step in building language models.

What to watch next

  • Comparisons of offline preference methods with online reinforcement learning.
  • Preference data quality and synthetic preference generation.
  • DPO variants for multimodal and reasoning models.
  • Combination of preference tuning with verifiable-reward RL.

Key takeaways

  • DPO aligns language models to preferences without a separate reward model or RL loop.
  • It rewrites the RLHF objective so the policy itself defines an implicit reward.
  • Training uses pairs of preferred and rejected responses with a simple loss.
  • In the paper’s experiments, DPO matched or beat PPO-based RLHF while being simpler.
  • It became widely used in open-weight model training.

Frequently asked questions

Is DPO better than RLHF?

It is simpler and often competitive. Online RL methods can still perform better in some settings, especially with verifiable rewards.

What data does DPO need?

Prompts with pairs of responses where one is labelled preferred over the other, from humans or AI judges.

Who wrote the DPO paper?

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn, at Stanford University.