MWPO : Enhancing LLM s Performance through Multi-Weight Preference Strength and Length Optimization (2025)

Abstract: Direct Preference Optimization (DPO) have proposed offline alternatives to Reinforcement Learning from Human Feedback (RLHF). In DPO, each preference pair, which serves as the foundation for learning, is typically constructed by first generating multiple responses to the same instruction and then annotating them to indicate the preferred choice. However, when the responses are highly similar, the weak preference signal can introduce annotation noise, which may hinder model optimization. Additionally, DPO suffers from the drawback of over-optimizing for verbose generation. A potential reason is the presence of length bias in preference datasets, which can lead to length exploitation. To address these issues, we propose a DPO-based **m**ulti-**w**eight **p**reference strength and length **o**ptimization (MWPO) method. Specifically, we propose to reweight preference pairs based on implicit reward margins and response length margins, unifying them through a geometric mixture to generate synthetic weights for optimization. This method allows preference pairs with stronger preference signals or more favorable length feature to have a more pronounced impact on model parameters. Moreover, our method does not require additional annotators. We validate our method on models of four different scales across multiple benchmarks. Our method surpasses state-of-the-art (SOTA) baselines, outperforming DPO by up to 8.7% on AlpacaEval 2 while reducing generation length by 9.4% in the Mistral setting. Our code is available at https://github.com/AIR-hl/MWPO.

## introduction reinforcement learning from human feedback (rlhf) is a key approach for effectively aligning large language models (llms) with human values and preferences @xcite @xcite . rlhf further improves the instruction-following capability and safety of supervised fine-tuned (sft) models. in traditional rlhf pipelines, the reinforcement methods such as proximal policy optimization (ppo) @xcite , have demonstrated their effectiveness in popular llm applications like chat-gpt @xcite and claude @xcite . however, due to its multi-stage online process, which involves training reward models and optimizing policy models, rlhf presents significant challenges in training. recently, offline preference optimization algorithms, represented by methods like direct preference optimization (dpo) @xcite , have emerged as highly promising approaches. dpo reparameterizes the reward function used in rlhf, eliminating the need for a reward model and directly learning the policy model from offline preference data. due to its simplicity and efficiency, dpo has been widely adopted in practice. in offline preference optimization methods, data quality directly determines alignment performance. therefore, making efficient use of existing offline data is crucial. dpo and its variants, such as simpo @xcite , treat each preference pair equally, making them vulnerable to certain noise in datasets. specifically, each preference pair in an offline dataset is typically constructed by annotating preferences for multiple responses sampled from the same prompt. @xcite and @xcite also argue that using the same prompt to construct preference pairs often results in responses of similar quality, making it challenging to distinguish between them. in this scenario, the data with narrow preference margins can lead to training signals being overshadowed by noisy preference labels, ultimately impairing the performance of the trained model. this inspired us to consider that distinct preference pairs might be @xcite and wpo @xcite are recent sota baselines. additionally, another significant challenge faced by preference optimization methods is "verbosity", where the optimized models tend to generate much longer responses without necessarily improving the generation quality. analysis shows that policy models trained with dpo generate responses that are significantly longer than the base model @xcite . this problem is related to bias in the length distribution of the training data, where annotators typically prefer more detailed responses, causing the chosen responses to be longer than the rejected responses. models may exploit this bias, leading to a statistically significant increase in response length. to address these issues, in this paper, we propose a dpo-based multi-weight preference strength and length optimization (mwpo) method. as shown in fig. 1 , unlike dpo which treats each data equally, our method measures the preference data from two aspects. first, to effectively distinguish and leverage each preference pair, we argue that during training, the policy model can act as a preference discriminator leveraging previously learned preference information to assess preference differences in new data. strongly preferred data are given more attention during training. to achieve this, we propose a reward-margin weight. second, to address the potential length bias issue, we construct an effective length-controlled weight based on the scaled length margin between the chosen and re-jected responses. finally, we can integrate both or one of the weights with the loss of dpo to obtain the final training objective. our contributions are summarized as follows: ## related work rlhf has become an essential method to align or enhance the performance of llms @xcite @xcite @xcite . rcld @xcite employs contrasting instructions to create differentiated preference data. west-of-n @xcite samples multiple responses and selects the best and the worst response through an additional reward model to construct a preference pair. however, reinforcement learning methods still suffer from data and training inefficiency due to the need for repeated sampling and policy updates during training @xcite . more offline methods, exemplified by dpo @xcite , typically optimize policy models using offline preference dataset without additional reward model. gpo @xcite unifies several offline methods, including dpo, ipo @xcite , and slic @xcite . kto @xcite learns from a binary signal of whether an output is (un)desirable. wpo @xcite simulates the on-policy learning process by adding weights to the optimization objective of dpo. none of these methods account for the potential noise introduced by similar responses. another significant challenge in reinforcement learning is over-optimization, where reward scores from the reward model increase, but the quality of model generation deteriorates, often becoming excessively verbose @xcite @xcite . @xcite mitigates the risk of dpo overoptimization through theoretical analysis. @xcite propose simpo, which avoids reliance on a reference model by using length-normalized rewards to prevent verbose responses. r-dpo proposed by @xcite , effectively controls the response length by simply adding a length regularization term to the implicit reward. in this work, we extend the dpo method by jointly optimizing the preference strength and length from different aspects. unlike existing offline methods, we reweight each preference pair according to its preference strength. additionally, we introduce a length-controlled weight to replace the additive length regularization, which effectively mitigates length exploitation and allows for flexible adjustment of the constraint strength through coefficients. we benchmark our method against a series of offline preference optimization methods, including dpo, simpo, wpo, and et al. results demonstrate that our method achieves a good performance and controls the generation length effectively. ## method in this section, we first present the theoretical background of rlhf and dpo ( §3.1). then, we propose two distinct weights: one for measuring pref-erence strength ( §3.2) and the other for addressing length bias ( §3.3). finally, we construct a mixed weight approach ( §3.4) and provide a theoretical analysis of our method ( §3.5). ## experimental setup in this section, we begin with the models and training settings ( §4.1); followed by the evluation benchmarks ( §4.2) and baselines ( §4.3). ## experimental results in this section, we first report the group comparison experiments ( §5.1) against strong baselines on a series benchmarks, our methods make a superior performance and length control. then, we study the two weights in section 5.2 and 5.3 respectively. ## conclusion in this work, we propose mwpo, an effective method that reweights preference pairs using multiple weights based on preference strength and length bias, respectively. extensive experiments demonstrate that it can effectively control verbosity and improve overall performance. furthermore, we analyze each weight to illustrate its necessity and effectiveness. we also identify an inherent flaw in dpo that may lead to length exploitation, representing a potential avenue for future breakthroughs.