This change-of-variables approach avoids fitting an explicit, standalone reward model, while still optimizing under existing models of human preferences, such as the Bradley-Terry model. In essence, the policy network represents both the language model and the (implicit) reward. Deriving the DPO objective. We start with the same RL objective as prior work, under a general reward function r. Following prior work, it is straightforward to show that the optimal solution to the KL-constrained reward maximization objective takes a specific form where the optimal policy is proportional to the reference policy times the exponentiated reward, scaled by beta. The proportionality constant is a partition function. Even if we use the MLE estimate of the ground-truth reward function, it is still expensive to estimate the partition function, which makes this representation hard to utilize in practice. However, we can rearrange the equation to express the reward function in terms of its corresponding optimal policy, the reference policy, and the unknown partition function. Specifically, we first take the logarithm of both sides and with some algebra we obtain that the reward is equal to beta times the log ratio of the optimal policy to the reference policy, plus beta times the log of the partition function. We can apply this reparameterization to the ground-truth reward and corresponding optimal model. Fortunately, the Bradley-Terry model depends only on the difference of rewards between two completions. Substituting the reparameterization for the reward into the preference model, the partition function cancels, and we can express the human preference probability in terms of only the optimal policy and reference policy. Thus, the optimal RLHF policy under the Bradley-Terry model satisfies the preference model, which can be expressed as a logistic function of the difference between the log-ratios of the optimal and reference policies for the two responses, scaled by beta. While this uses the Bradley-Terry model, we can similarly derive expressions under the more general Plackett-Luce models. Now that we have the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for a parametrized policy. Analogous to the reward modeling approach, our policy objective becomes a negative log-likelihood loss based on the logistic function of the difference in scaled log-ratios of the policy probabilities. This way, we fit an implicit reward using an alternative parameterization, whose optimal policy is simply the policy itself. Moreover, since our procedure is equivalent to fitting a reparametrized Bradley-Terry model, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution. In Section 5, we further discuss theoretical properties of DPO in relation to other works. What does the DPO update do? For a mechanistic understanding of DPO, it is useful to analyze the gradient of the loss function. The gradient with respect to the parameters can be written as an expectation over the preference data. Intuitively, the gradient of the loss function increases the likelihood of the preferred completions and decreases the likelihood of dispreferred completions. Importantly, the examples are weighed by how much higher the implicit reward model rates the dispreferred completions, scaled by beta, i.e, how incorrectly the implicit reward model orders the completions, accounting for the strength of the KL constraint. Our experiments suggest the importance of this weighting, as a naive version of this method without the weighting coefficient can cause the language model to degenerate. DPO outline. The general DPO pipeline is as follows: 1) Sample completions for every prompt, label with human preferences to construct the offline dataset of preferences and 2) optimize the language model to minimize the DPO loss for the given reference policy, preference dataset, and desired beta. In practice, one would like to reuse preference datasets publicly available, rather than generating samples and gathering human preferences. Since the preference datasets are sampled using the SFT model, we initialize the reference policy as the SFT model whenever available. However, when the SFT model is not available, we initialize the reference policy by maximizing the likelihood of preferred completions. This procedure helps mitigate the distribution shift between the true reference distribution which is unavailable, and the reference policy used by DPO. Further details related to the implementation and hyperparameters can be found in the Appendix. Section 5: Theoretical Analysis of DPO