DPL : Diverse Preference Learning Without A Reference Model (2025)

Abstract: In direct preference alignment in LLMs, most existing methods seek to retrieve the reward function directly from preference data. However, real-world preference data often contains diversity in preference annotations reflective of true human preferences. Existing algorithms, including KTO, do not directly utilize such nuances in the annotations which limits their applicability. In this work, we propose Diverse Preference Learning (DPL), a reference model-free method that simultaneously learns a baseline desirability in LLM responses while being robust to the diversity of preference annotations. Our experiments for instruction-following on Ultrafeedback and AlpacaEval 2.0 and for text-summarization on Reddit TL;DR suggest that DPL is consistently better at learning the diversity of preferences compared to existing methods, including those that require a reference model in memory. Apart from overall quality, we find that DPL’s completions, on average, are more honest, helpful, truthful and safe compared to existing methods.

## introduction well-known approaches to human preference learning in llms, including rlhf-based methods like proximal policy optimization (ppo; @xcite ), require costly reward models to guide alignment, limiting their utility in practical resource-constrained settings. in contrast, offline algorithms like direct preference optimization (dpo; @xcite ) provide better compute-efficiency by optimizing directly on preference data without an explicit reward model. reference-model free variants like orpo @xcite and simpo @xcite do this without a reference model in memory. at the core of direct alignment methods is the bradley-terry (bt) model @xcite , which models preferences as implicit reward differences between "good" and "bad" responses. while appealing for their simplicity, the bt model assumption that true human preferences directly correlate with model-assigned implicit reward differences is problematic: firstly, popular preference datasets like ultrafeedback @xcite contain a non-negligible amount (∼17%) of weak preference pairs that are rated "equally" (reward difference ∼0) by high-capacity automatic evaluators like gpt-4. when the preference signal is weak, common direct alignment frameworks can lead to significant policy degeneracy or underfitting, especially under finite data settings @xcite . additionally, such underfitting can effectively limit such methods from learning a "baseline desirability" of all available responses, especially for weak preference pairs where both examples are rated low. (see figure 1 for an illustration.) figure 1 : reward estimates (preference strength) from gpt-4 on the ultrafeedback dataset @xcite . diverse preference learning (dpl) learns both a "baseline desirability" as well as the "relative goodness" between responses, outperforming competitive baselines including @xcite on popular preference learning benchmarks (see in particular, reward underspecification in the bradley-terry model @xcite can hinder these frameworks from capturing the nu-anced preferences or relative "goodness" between responses-which is arguably available in widelyused datasets. this is true when the preference distribution 1 is varied or more nuanced, i.e., the reward differences fall within a spectrum as shown in figure 1 . even more sophisticated methods like @xcite ), often reduce pointwise preferences to binary labels, sacrificing the granularity needed to fully leverage rich annotations. in this work, we introduce diverse preference learning (dpl), a novel reference-model-free alignment framework that enables policies to capture the rich, often overlooked nuances of human feedback. dpl simultaneously learns baseline desirability and optimizes for the relative strength of preference labels, resulting in a more accurate and nuanced representation of true human preferences. our novel contributions are: ## related works reinforcement learning from human feedback (rlhf) aims to harmonize large language models with human preferences and values (christiano 1 while the true distribution of human preferences is unknown and likely multidimensional @xcite , we assume gpt-4's reward estimates provide a reasonable approximation, particularly when direct annotations are unavailable. @xcite @xcite . the conventional rlhf pipeline typically consists of three primary phases: supervised fine-tuning @xcite @xcite @xcite @xcite @xcite , reward model training @xcite @xcite @xcite , and policy optimization @xcite . more recently, supervised offline contrastive learning approaches like dpo @xcite , @xcite , e-dpo @xcite , @xcite , and dpo-positive @xcite have been proposed. these approaches include a reference model in memory as a baseline policy and intend to minimize compute by avoiding online sampling for reward maximization. specifically, algorithms like dpo and kto seek to retrieve optimal policies that analytically derive from the kl-constrained optimization problem @xcite ; dpo seeks to align with human preferences, and kto, with human preferences and utility. in contrast, reference-free approaches like cpo @xcite , orpo @xcite and simpo @xcite often assume the reference model is a uniform distribution, and proposes alternative bradley-terry reward formulations, which increase efficiency by dispensing with the need for an additional reference model. dubey et al. (2024 et al. ( 2024 ) simultaneously learn preferences and rewards with reward-distillation. specifically, our work can be seen extending this line of work but with a novel unlikelihood @xcite penalty on dispreferred responses, that along with our nuanced preference categorization, allows more diverse preference alignment. ## background: implicit rewards (r * ) in "supervised" preference alignment, we are given a dataset of pairwise preference data d @xmath0, y w , y l , s ij } n @xmath1, where x denotes the prompt or context, y w and y l are the preferred and dispreferred completions respectively, and s ij represents the strength of the relative preference between y w and y l . 2 the parameters of the aligned policy π θ are then estimated "directly" on d pref by formulating the preference of y w over y l using the bradley terry (bt) model as: ## diverse preference learning framework (dpl) to prevent such degeneracies, we aim to define the dpl objective such that it (a) is reference modelfree for computational efficiency, (b) uses a bt model-based loss to learn from pairwise contrastive preference data, (c) regularizes the loss function to avoid overfitting to the "good" and "bad" responses in contrastive feedback pairs, (d) incorporates unpaired "good" and "bad" responses when available, and (e) offers a flexible weighting scheme to capture the granular desirability of samples when such information is present. to achieve these objectives, we propose two variants of the dpl objective, l dpl odds and l dplnorm . these two dpl objectives represent the odds-ratiobased and the length-normalized formulations of the implicit reward signal, respectively. we can rewrite the above two formulations of the l dpl odds and l dplnorm losses in their full parametric form by replacing the l sft with log(π θ (y w | x) and replacing r * odds(x,y) and r * norm(x,y) with their parametric form from eq. 4 and eq. 5, respectively. the full parametric forms of the dpl losses are represented below: finally, the α, γ, and η weights can be seen as a combination (e.g., products) of sampleindependent regularization hyperparameters or sample-wise weights reflecting baseline desirability or relative quality. for instance, in our ultrafeedback dataset experiments (section 5), we derived sample-wise weights from baseline desirability categories such as "accepted," "partially accepted," and "rejected." in many real-world applications, more granular response quality scores may be available, allowing for more advanced sample weighting approaches @xcite . a sample-efficient way for π θ to learn such granularity is to use a "weighting" function @xcite that applies regularization dynamically to y w and y l based on the distribution of relative preference strengths s ij provided in d pref . 4 this function h(d pref ) : @xmath2, γ, η) ascertains the relative importance that π θ should place on y w and y l during alignment. as such, dpl works in two phases: first, a twolayered categorization of preference pairs is computed from d pref based on their i) baseline desirability and ii) relative strengths. secondly, dpl applies a weighting function that uses this catego-rization to dynamically learn a refined distribution of preferences. determining baseline desirability a practical way to determine the baseline desirability is to decompose s ij into a tuple (s i , s j ) such that s @xmath3: "accepted," "partially-accepted," and "rejected," signifying a decreasing order of baseline desirability. specifically, response pairs are defined as "accepted" if s @xmath4, "partially-accepted" if t @xmath5, and "rejected" if s @xmath6, where t a and t p are precomputed from d pref , akin to @xcite . the process allows us to retrieve the individual instance of y w that each fall within these categories. how "good" is y w relative to y l ? though intuitive, baseline desirability alone is not sufficient since ideally we also want π θ to learn the goodness of y w relative to y l . as such, we also retrieve the relative placement of y l in terms of y w 's complement by determining where s j falls between t a and t p . to do this, if the lower score, s j , does not satisfy the same threshold of y w , then the pair (s i , s j ) is termed "accepted/rejected" if s @xmath7. similarly, if the scores are almost equal (s i ∼ s j ), indicating no preference, we call the preference relation in this pair nondeterministic or equally preferred. ## results here we present automatic evaluation results in the llm-as-a-judge setting (section 6.1) and ablation results with a traditional rm (section 6.2). appendix i contains example model outputs. ## conclusion in this work, we present diverse preference learning (dpl), a reference-free preference alignment framework that allows policies to explicitly model the diversity of preference annotations in popular offline datasets, that current methods often ignore. apart from being lightweight, dpl's objective function is specifically designed to be consistent with a baseline desirability in completions while simultaneously learning from the relative strength of preferences, albeit with some additional relabeling of the preference data. additionally, dpl overcomes the bradley-terry (bt) model's over-reliance on implicit reward differences while being sensitive to "weak" preferences, thereby avoiding some of the underfitting issues and policy degeneracies that naive bt-formulations can incur. this leads to consistent high-performance on key benchmarks for targeted tasks, including ultrafeedback and reddit tl;dr (tested on cnn/-daily mail)-through controlled evaluations conducted with both an llm-as-a-judge as well as a traditional reward model (rm). furthermore, across various preference desiderata including honesty, helpfulness, truthfulness, and safety as well as on ood-settings in alpaca eval 2.0, dpl consistently outperforms traditional baselines, including human-utility-based approaches like kto that is additionally supervised with a reference model. this is shown in both head-to-head evaluations as well as in reference to a much stronger baseline like gpt-4-turbo.