Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence (2024)

Abstract: Direct Preference Optimization (DPO) has emerged as a prominent algorithm for the direct and robust alignment of Large Language Models (LLMs) with human preferences, offering a more straightforward alternative to the complex Reinforcement Learning from Human Feedback (RLHF). Despite its promising efficacy, DPO faces a notable drawback: “verbosity”, a common over-optimization phenomenon also observed in RLHF. While previous studies mainly attributed verbosity to biased labels within the data, we propose that the issue also stems from an inherent algorithmic length reliance in DPO. Specifically, we suggest that the discrepancy between sequence-level Kullback–Leibler (KL) divergences between chosen and rejected sequences, used in DPO, results in overestimated or underestimated rewards due to varying token lengths. Empirically, we utilize datasets with different label lengths to demonstrate the presence of biased rewards. We then introduce an effective downsampling approach, named SamPO, to eliminate potential length reliance. Our experimental evaluations, conducted across three LLMs of varying scales and a diverse array of conditional and open-ended benchmarks, highlight the efficacy of SamPO in mitigating verbosity, achieving improvements of 5% to 12% over DPO through debaised rewards. Our code can be accessed at: https://github.com/LuJunru/SamPO/.

## introduction reinforcement learning from human feedback (rlhf) is a crucial strategy for effectively align large language models (llms) with human minds @xcite @xcite , showcasing significant improvements of llm's instruct-following capability compared with the other two popular approaches: pretraining and supervised fine-tuning (sft). in fact, a series of leading llms have adopted rlhf as the final stage of their entire training pipelines @xcite @xcite . nevertheless, traditional rlhf involves several intricate multi-stage steps, typically starting with fine-tuning a reward model that captures complex human intuition @xcite , followed by optimizing llms to maximize preference scores. therefore, the quality of the reward model is crucial. however, modeling elusive human intuition is inherently difficult @xcite . on the contrary, direct preference optimization (dpo) @xcite proposed to re-parameterize the reward model, integrating preference feedback from online rewards into offline labels. in specific, dpo employs the bradley-terry model @xcite to maximize implicit rewards via pairwise offline preference labels. the implicit reward is mathematically equivalent to the discrepancy in sequencelevel @xcite divergences @xcite between chosen and rejected labels. the kl divergence for each label is calculated based on probability outputs from the finetuning policy model and a frozen reference model. dpo eliminates the need for complex prefix finetuning of an external reward model, while maintains performance comparable to rlhf @xcite . despite its effectiveness, dpo faces several notable challenges, including issues of overfitting @xcite , high computational costs @xcite , and verbosity @xcite . this paper specifically focuses on addressing the "verbosity" issue. traditional multi-stage rlhf methods argue that due to a statistical bias in length distribution, that is, where preferred labels tend to be longer than rejected preference labels @xcite , the reward model trained on such preference data inherently exhibit a length bias @xcite . therefore, subsequent finetuned policy model exploit this bias as a shortcut to achieve higher reward scores by generating longer responses @xcite , without necessarily improving quality @xcite . various regularization approaches have been proposed to mitigate this inherent bias within reward models @xcite @xcite . on the other hand, although dpo does not explicitly use a reward model, the length distribution bias inherent in the offline preference labels still contributes to the verbosity issue @xcite . analysis suggests that policy models trained using dpo tend to generate responses that are almost twice the length of the labeled data @xcite . in this paper, we propose that, in addition to the length bias in the data, dpo exhibits a hidden algorithmic dependence on response length. as illustrated in the upper portion of figure 1 , the loss function in dpo is based on the discrepancy between sequence-level kl divergence, which can also be computed and aggregated at the token-level. it is evident that discrepancies between chosen label y w and rejected label y l lead to an inadvertent reliance on auxiliary length features: training samples with longer chosen labels than rejected ones lead to overestimated rewards during training, while those with shorter chosen labels result in underestimated rewards. therefore, overestimated re-wards contribute more significantly to gradient optimization, ultimately exacerbating verbosity. we believe this algorithmic dependence on response length is a unique drawback of dpo, since the explicit rewards in rlhf typically manifest as scalar values @xcite . we propose that addressing this reliance on response length can be effectively achieved through a straightforward down-sampling method. illustrated in the middle of figure 1 , this approach involves down-sampling equal token-level probability features for computing regularized kl divergences. our contributions in this paper are threefold: ## related work optimization from human preference aims to align neural models with human minds. as a seminal work, @xcite collected human preferences on 123k pairs of summary outputs, then trained a reward model that guides the gpt-3 model @xcite to produce more coherent and human-preferred summaries. @xcite then further scaled similar pipeline with 1m diverse text instructions, and reported that outputs from the 1.3b parameter instructgpt model were preferred to outputs from the 175b gpt-3 model, according to downstream human evaluation. rlhf has become an essential part of aligning llms @xcite @xcite . however, as it follows a multi-stage training strategy, and heavily relays on the quality of reward model, rlhf's training cost and stability are widely criticized @xcite . therefore, dpo came into being, providing a stable alternative that does not rely on an explicit reward model @xcite . it has been proved that dpo can achieve the same alignment effect as rlhf @xcite . over-optimization in rl is a well-known obstacle @xcite @xcite , which refers to the phenomenon that feedback scores from the reward model are getting higher, but the updated policy model produces lower quality responses. and one particularly noticeable low-quality feature is verbosity. it is general to blame for exploitation of reward model @xcite , and thus various regularization approaches have been proposed, including uncertainty-based regularization @xcite , composite reward models @xcite , and length decorrelation @xcite . however, since the reward model is eliminated in dpo, none of the above approaches can be directly applied. herein, specific methods are introduced, @xcite introduced a pairwise length regularization term to dampen the verbosity trends, and simpo @xcite used average probability to eliminate length reliance. in this paper, we present that the verbosity issue in dpo is further related to algorithmic biased length reliance, which is never analyzed in previous literature. and this drawback can be effectively handled via down-sampling over kl divergence. ## sampo: down-sampled dpo in this section, we first give a brief introduction of dpo's optimization target ( §3.1), then dive into further analysis of its potential length reliance ( §3.2). subsequently, we present sampo, which intuitively regularizes the biased length-specific reward ( §3.3). ## experimental setup in this section, we start by introducing our datasets ( § 4.1, § 4.2), followed by the baselines ( § 4.3, § 4.4), and then provide an overview of our experimental design ( § 4.5). ## experimental results in this section, following the above designs, we first report the group experiments of length reliance ( § 5.1), then present comparison studies against strong baselines ( § 5.2). we discuss quantitative results in the main body. we leave more ablation studies and case analysis in appendix e, f, and h. ## conclusion in this paper, we identify and address the verbosity issue in dpo related to biased length reliance. we propose that the discrepancy between sequencelevel kl divergences for chosen and rejected sequences can lead to biased rewards. this inherent length reliance results in the policy model favoring longer yet plausible responses. thus, we propose sampo, an approach that regularizes the kl divergence by down-sampling equal token-level features. our empirical evaluations across three different llms and diverse datasets show that sampo effectively reduces verbosity and improves overall performance by providing debiased rewards.