Disentangling Length from Quality in Direct Preference Optimization (2024)

Abstract: Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful and objective. A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a separate reward model or use reinforcement learning directly, so previous approaches developed to control verbosity cannot be directly applied to this setting. Our work makes several contributions. For the first time, we study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping. We then develop a principled but simple regularization strategy that prevents length exploitation, while still maintaining improvements in model quality. We demonstrate these affects across datasets on summarization and dialogue, where we achieve up to 20% improvement in win rates when controlling for length, despite the GPT4 judge’s well-known verbosity bias.

## introduction recently, large language models (llms) have seen significant improvements in capabilities, such as code-generation, mathematical reasoning, and tool use. importantly, they can now fluently interact with users and follow their instructions, leading to their widespread adoption. fine-tuning with reinforcement learning from human feedback (rlhf) @xcite *denotes equal contribution figure 1 : average win rates versus generation length (liu, 2024) on the alpaca eval benchmark @xcite . while the highest-scoring open-source models can match the overall performance of strong closed models, they lag significantly on length-corrected basis. the quality of the learned reward model is crucial for the rlhf process @xcite . however, prior works have demonstrated that reward models can be exploited @xcite due to a goodhart's law effect @xcite @xcite . under this phenomenon, the model can achieve high rewards during the rl training while generating undesirable behaviours @xcite . a particular case of the reward exploitation phenomenon is the well-known verbosity issue -models fine-tuned with rlhf generate significantly longer answers, without necessarily improving the actual quality @xcite . this has been linked to an explicit bias in the preference data towards longer responses @xcite . however, the statistical increase in verbosity of rlhf-trained models significantly outmatches the the difference of distribution lengths between the preferred and rejected answers. this effect is even observed in in strong propriety models, such as @xcite , which is now frequently used to evaluate the performance of other llms @xcite @xcite . however, even as an evaluator, gpt-4 exhibits strong preferences for length. prior work @xcite has noted that when evaluating 13b parameter models in head-to-head comparisons with the davinci-003 model, win rates and the average number of unique tokens in the model's response have correlation of 0.96. recently, direct preference optimization @xcite has emerged as an alternative to the standard rlhf pipeline. the key observation of dpo is that the reward model can directly be re-parameterized through the optimal llm policy obtained in the reinforcement learning stage. this allows us to directly train the language model through the reward learning pipeline, eliminating the need for the reinforcement learning stage. this algorithm has become widely used, since it can train completely offline, yielding better simplicity of tuning, speed, and resource efficiency, while still maintaining performance @xcite . for these reasons, it has also been widely adopted by the open-source community. at the time of this writing, 9 out of the top 10 models on the huggingface open llm leaderboard use dpo as part of their training pipeline. while the question of length exploitation has been extensively studied in the classical rlhf pipeline, it has not been explored in the dpo setting. rlhf-style reward models are explicit, making them susceptible to issues like reward overoptimization @xcite . it is unclear whether these issues transfer to dpo, where the lack of an explicit reward model means the problem of reward overoptimization is harder to define. to complicate this issue, others have argued that apparent gains in open-source model performance across automated benchmarks are driven by evaluator's verbosity bias @xcite . these statistics are demonstrated in figure 1 , as open-source models can match the overall performance of proprietary ones, but lag significantly on length-corrected basis. we make several contributions in our work: first, we show the length exploitation is quite prevalent in dpo. we demonstrate empirically (for the first time) that in this settings ood extrapolation issues emerge similarly to classical rlhf. next, we derive a simple but efficient regularization approach, showing it can effectively control verbosity and minimally impact performance even under a length-biased judge which also explains other empirical phenomena in dpo training, such as early convergence. ## preliminaries in this section, we will outline the core components of the standard rlhf pipeline ziegler et al.; @xcite and the direct preference optimization algorithm @xcite , which is central to our analysis and regularization derivations. ## building in explicit regularization in dpo prior works have explicitly considered lengthregularization in the classical rlhf pipeline (singhal et al., 2023), however these methods do not transfer directly to direct alignment algorithms, such as dpo @xcite . we derive a length-regularized version of the algorithm by adding a regularized term to eq. 3. the below considerations hold for a general regularizer, but we focus on a length term α|y|, where α is a hyperparameter and |y| denotes the token-length of the answer y. we then formulate the regularized rl problems in the following objective: with some simple algebra, we can then obtain the equivalent regularized reward re-formulation: we can then plug in eq. 8 into the reward modeling stage in eq. 2, which yields the following regularized dpo objective: this is similar to the standard dpo objective, except for an additional regularization term (α|y w | -α|y l |) in the logit of the binary classification loss. recent work @xcite , aims to develop a multi-objective optimization approach and arrives at a similar objective involving a mixture of reward models, while we focus on optimization robustness of dpo. another concurrent work (chen et al., 2024) also consider the length exploitation problem in the classical rlhf pipeline, suggesting a similar regularization in the reward modeling stage in eq. 2. this regularizer helps disentangle answer quality from length, demonstrating meaningful improvement in length-controlled performance. our derivations can be seen as the dpo implicit reward counterpart to this classical rlhf approach, explicitly linking the regularized reward modeling problem to an equivalent regularized rl setup. similar to the original dpo formulation, the regularized objective still aims to increase the likelihood along the preferred answer, while decreasing the likelihood along the dis-preferred answer, modulated by a weighting term. this term is equivalent to the original dpo formulation with the addition of the regularization margin α|y w | -α|y l |. we can interpret this as an additional per-example learning rate, which up-weighs the gradient on feedback pairs, in which the selected answer is shorter and down-weights the gradient on pairs in which the selected answer is longer, proportional to the difference in length. ## experiments in this section we will empirically investigate the verbosity exploitation issues in dpo, the effectiveness of our regularization strategy and the potential causes of these effects. we beging with a description of our evaluation tasks and models. ## related work in this section we outline relevant work on reward exploitation in rlhf and the verbosity bias. reward exploitation in rlhf: rlhf reward exploitation, also known as reward over-optimization, is a well-known issue @xcite @xcite in which during the reinforcement learning stage, the expected reward keeps improving, but the quality of the model begins to degrade after some point. these effects were confirmed analytically in controlled experiments @xcite , as well as empirically in user studies @xcite . increased model verbosity has been explicitly linked to this phenomenon @xcite . a number approaches have been proposed to mitigate this issue, such as penalizing epistemic uncertainty @xcite @xcite or using mixture reward models @xcite ), but they do not explicitly target the length issue. mitigating length biases in rlhf: a number of works have sought to explicitly address length biases in rlhf policies. @xcite suggest setting a simple discount factor, which improves naturalness of the generated language, @xcite carry out an extensive study of length correlations in classical rlhf and suggest a number of heuristic-based mitigating approaches. the closes to our approach are the works of @xcite and the concurrent work of @xcite , which propose to disentangle length-biases from quality during the reward modeling stage. our work can be seen as a dpo equivalent counter-part to these approaches. as far as we are aware, this is the first work to study the length exploitation problem for direct alignment algorithms such as dpo. ## conclusion in this work, we study the problem of length exploitation in the direct preference optimization (dpo) algorithm, extending its analysis from classical rlhf to dpo for the first time. on two standard human feedback datasets, we empirically show that dpo exhibits significant length hacking across a range of hyperparameters. we then specifically link this phenomenon to out-of-distribution bootstrapping. we derive an analytical lengthregularized version of the dpo algorithm and show empirically that we can maintain model performance, as evaluated by gpt-4 without significant increases in verbosity, boosting length-corrected win rates by up to 15-20%. given the strong length bias in public feedback datasets and the prominence of dpo in the open source community, we hypothesize that many open-source models suffer from similar length-exploitation issues, driving the observations of fig. 1 . our results are encouraging, suggesting that open-source models could match proprietary ones on automated evaluations on a length corrected basis as well.