Direct Preference Optimization with an Offset (2024)

Abstract: Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.

## introduction reinforcement learning from human feedback (rlhf) is a key building block in training the most modern large language models. the algorithm aligns the language model's responses to human preferences @xcite @xcite . a typical implementation of rlhf is as follows. first, humans compare paired responses from a language model, i.e., they determine which of the responses is better, e.g., more helpful or less toxic. second, a reward model is trained to give higher rewards to the responses preferred by humans. lastly, a reinforcement learning algorithm is used to update the model's parameters to maximize the expected reward, while not diverging too much from the model's initial parameters. the aforementioned implementation of rlhf, however, is tedious for two reasons. first, the reward model is usually a model as large as the language model itself, which is expensive to train and store. second, reinforcement learning algorithms are known to be very sensitive to the choice of hyperparameters @xcite , and, thus, hard to tune. therefore, a fruitful line of work attempts to design alternative simpler implementations of rlhf @xcite @xcite . a highly successful and straightforward alternative to rlhf is direct preference optimization (dpo; @xcite . unlike the conventional implementation of rlhf, dpo does not train a reward model and sidesteps the use of any reinforcement learning algorithm. instead, it finetunes the language model's weights to maximize the likelihood of the preference data directly using the bradley-terry model (bradley and @xcite . intuitively, each dpo gradient update to the model's parameters increases the likelihood of the preferred response and decreases the likelihood of the dispreferred response. however, dpo only takes the ordering between the model's responses into account and not the extent to which one response is preferred over another. in many settings, the difference in quality between the two responses is known-either by asking humans @xcite foot_0 or through point-wise scores, e.g., toxicity scores, sentiment scores, code-based execution scores, etc. in that context, a natural question is the following: how can we make use of this information to better align language models with human preferences? to answer this question, we propose a new method, which we term dpo with an offset figure 1 : odpo takes into account the extent to which one output should be preferred over another. the model has to put more probability mass on the preferred output compared to the dispreferred output by an offset that is determined based on how much the winning output is preferred over the losing output. in our experiments, we fine-tune language models of various sizes with dpo and odpo and compare the two methods. in cases where a ground-truth reward function is given, we measure two competing metrics: (i) the average reward associated with the generations from the fine-tuned model, and (ii) the kl divergence between the language model before and after the fine-tuning. based on the results of sentiment and toxicity control tasks, we observe that odpo more often appears on the pareto frontier of reward and kl compared to dpo. we then apply odpo to a summarization task, where we use the scores given by humans on a likert scale to define an offset between the two summaries. we observe that on average and across different sampling temperatures, odpo results in a higher win rate over human-written summaries compared to dpo. ## preliminaries given a prompt @xmath0, a language model π θ is a distribution over the responses @xmath1, where σ is the alphabet. 2 as is standard, we parameterize the conditional distribution over responses given a 2 an alphabet is a finite, non-empty set. we use π sft to refer to the language model finetuned with this objective on d sft . such a fine-tuning approach can be effective for solving downstream tasks with language models. however, there is a disparity between the maximum-likelihood objective and the goal of the downstream task, which is to leverage the language model to generate high-quality responses, as judged by humans. this misalignment arises partly because the maximum-likelihood objective, eq. ( 2 ), fails to distinguish between major and minor errors. for instance, when fine-tuning a language model to perform summarization, a significant error might involve hallucination, while a minor one might involve a failure to employ the appropriate synonyms @xcite . training with the maximum-likelihood objective encourages the model to assign nonzero probability mass to all responses in d sft , even those of lower quality. to address this issue, one can optimize the expected reward of language model generations instead of using the maximum-likelihood objective. consider a reward function r(x, y), estimated from quality assessments performed by humans. the reward function assigns real values to promptresponse pairs; we will discuss in §3 how such a reward model can be learned from human feedback. our objective is to make sure that the responses generated by the language model have high quality. therefore, we can directly optimize the expected reward of the responses generated by the language model, the above objective is optimized using proximal policy optimization (ppo; @xcite or another actor-critic algorithm @xcite . ## reward modeling pointwise rewards. a key component in eq. ( 4 ) is the task-dependent reward function that assigns pointwise real-valued rewards to each output. in many tasks, learning such reward functions is straightforward. for example, in open-ended text generation, a desired attribute could be the presence of indicators of positive sentiment in the generated text, while an undesired attribute could be toxicity. in such cases, the reward model might take the form of a classifier that assesses responses based on their sentiment or toxicity. similarly, in code generation tasks, the quality of the code can be automatically evaluated, providing another straightforward example. pairwise preferences. learning a point-wise reward function for tasks like summarization or dialogue generation is more complex. judging the absolute quality of a summary can depend on several factors, e.g., coherence, faithfulness, and conciseness, which makes it hard to collect human feedback datasets for reward model training. an effec-tive strategy in such cases is to collect human preferences instead of point-wise judgments @xcite @xcite . in this setup, humans are shown two (or more) responses to a prompt x and are asked to select the response they prefer. therefore, a datapoint in a human feedback dataset is a triple (x, y w , y l ) ∼ d hf , where y w is preferred over y l . given such a dataset, one needs to learn the point-wise reward function. it is common to assume that the preference data can be modeled by a bradley-terry model (bradley and @xcite , intuitively, the reward function should assign higher reward values to the responses that are preferred by humans. the estimated reward function is then plugged into eq. ( 4 ) to enable policy estimation. it is often the case that directly optimizing eq. ( 4 ) is complex and resource-intensive. next, we will elaborate on an alternative method to using rlhf that has demonstrated comparable or even superior effectiveness to ppo. ## direct preference optimization rafailov et al. (2023) introduce a method to avoid reward model training and, thus, to directly optimize the language model. their method, termed direct preference optimization (dpo) works as follows. the critical observation is that the optimal solution that maximizes the eq. ( 4 ) is, note that z(x) is, in general, difficult to compute as it involves a sum over a countably infinite set. nevertheless, we can write the reward as a function of π ⋆ θ by rearranging the terms, under the assumption that the preference data is well-modeled by the bradley-terry model (eq. ( 5a )), we substitute the reward in eq. ( 6 ) with eq. ( 10 ), and formulate the loss function in terms of the language model parameters. therefore, we directly optimize the language model by maximizing the likelihood of the preference data in eq. ( 7b ). in this equation, r θ (x, y) = β log π θ (y|x) πsft(y|x) is called the estimated reward. note that the intractable normalization factor z(x) cancels out. simply put, eq. ( 7b ) requires the estimated reward for the preferred response to be larger than the estimated reward for the dispreferred response. a strong assumption behind the dpo's loss is that the preference data is well-modeled by a bradley-terry model. however, the bradley-terry model only tells us the probability that one response is preferred over another, and not the extent to which this preference will hold. to enhance dpo, we will introduce our modification to dpo, which generalizes dpo and introduces an offset between the responses in its loss function. ## dpo with an offset the intuition behind odpo is simple: we want the estimated reward for the preferred response to be larger than the estimated reward for the dispreferred response, by an offset that is a function of the actual reward differences assigned to the two responses. to begin our exposition of odpo, we first discuss the connection between the bradley-terry model and gumbel random variables, which we then use to construct the loss function for odpo. theorem 1. let y w and y l be two responses to a prompt x, and let r θ (x, y w ) and r θ (x, y l ) be their associated estimated rewards. finally, let r w ∼ gumbel( r θ (x, y w ), 1) and r l ∼ gumbel( r θ (x, y l ), 1) be gumbel random variables. then, we have, proof. the proof is simple and follows directly from the gumbel-max trick @xcite . see app. a for details. ## experiments in this section, we empirically compare the performance of odpo with dpo across different tasks. we refer the reader to app. c for the computational budget used for conducting the experiments. ## ablation studies in this section, we investigate and ablate certain design decisions that were made in the experiments. specifically, we investigate the effect of the offset hyperparameter α, and the scaling function f (•) on the performance of odpo. ## related work in this section, we review alternative methods to rlhf for aligning language models to human preferences. sequence-level contrastive methods. the intuition that is shared by these methods is simple: they encourage the language model π θ to assign more probability mass to the preferred response compared to the dispreferred response(s). @xcite employ a rank calibration loss that requires log π θ (y w | x) to be greater than log π θ (y l | x), by an offset δ. importantly, in their formulation δ is a hyperparameter and does not depend on the the responses y w , y l . similarly, @xcite uses a ranking loss without the offset. while intuitive, the objective that contrastive losses are optimizing for, and its connection to the rlhf objective (maximizing the expected reward) is unclear. dpo and variants. dpo proposes an alternative method to optimize the regularized expected rewards without using rl algorithms. importantly, dpo shares the objective with conventional methods for rlhf @xcite , but the optimization is done without training a separate reward model nor using rl algorithms. since the introduction of dpo, several follow-up studies attempted to improve dpo along different dimensions. to prevent dpo from overfitting to the preference dataset, azar et al. (2023) introduce identity preference optimization (ipo). ipo replaces the unbounded function of preference probabilities in dpo loss formulation with the bounded identity function. @xcite propose a method called kahneman-tversky optimisation (kto) that dispenses the need for paired preference data altogether. kto's loss function relies on unpaired examples that are labeled either as "good" or "bad". @xcite suggest another variant of dpo specifically for multi-objective alignment. our approach. odpo attempts to solve another shortcoming of dpo, which is to treat every preference pair equally and not take into account the extent to which the two responses differ from each other. odpo's loss requires the estimated reward for the preferred response to be larger than the dispreferred response by an offset that depends on the difference between the quality of the responses. ## conclusion we propose odpo, a generalization of dpo for aligning language models with human preferences. just as with dpo, odpo does not rely on a pretrained reward model and does not require an rl algorithm. however, in contrast to dpo, odpo does not treat every preference pair equally and incorporates the extent to which one response should be preferred over another in its loss function. experiments on a variety of tasks suggest that odpo is more effective than dpo in aligning language models to human preferences.