WPO : Enhancing RLHF with Weighted Preference Optimization (2024)

Abstract: Reinforcement learning from human feedback (RLHF) is a promising solution to align large language models (LLMs) more closely with human values. Off-policy preference optimization, where the preference data is obtained from other models, is widely adopted due to its cost efficiency and scalability. However, off-policy preference optimization often suffers from a distributional gap between the policy used for data collection and the target policy, leading to suboptimal optimization. In this paper, we propose a novel strategy to mitigate this problem by simulating on-policy learning with off-policy preference data. Our Weighted Preference Optimization (WPO) method adapts off-policy data to resemble on-policy data more closely by reweighting preference pairs according to their probability under the current policy. This method not only addresses the distributional gap problem but also enhances the optimization process without incurring additional costs. We validate our method on instruction following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2 but also establishes a remarkable length-controlled winning rate against GPT-4-turbo of 76.7% based on Gemma-2-9b-it. We release the code and models at https://github.com/wzhouad/WPO.

## introduction large language models (llms; @xcite @xcite ) have demonstrated remarkable capabilities in generating human-like responses. however, they still face challenges in scenarios demanding high standards of reliability, safety, and ethics. to address these challenges, reinforcement learning from human feedback @xcite @xcite ) is † correspondence @xmath0.zhou@zoom.@xmath1. depending on how the outputs are generated, rlhf can be categorized into on-policy and offpolicy settings. in the on-policy setting @xcite @xcite , the policy model used to generate outputs is the same as the policy model being optimized. during this process, a policy model is first initialized from supervised finetuning (sft). then, a reward model @xcite @xcite is obtained based on human @xcite or ai @xcite feedback. finally, the policy model samples outputs during training, which are then evaluated using the reward model. the policy model is optimized to improve the expected reward using training objectives such as proximal policy optimization (ppo; @xcite and direct preference optimization (dpo; @xcite . however, onpolicy rl relies heavily on policy sampling during training and online rewards, which can incur high costs. in contrast, in the off-policy setting @xcite , the outputs are generated from different models, and the policy model is optimized based on these data instead of its sampled outputs. consequently, off-policy rl offers significant advantages in terms of cost and data efficiency and is easier to scale up. nevertheless, off-policy rl often shows worse performance than on-policy rl, due to the distributional gap between the policy used to collect data and the target policy being optimized, which leads to instability and inefficiency in training @xcite @xcite @xcite . in offpolicy preference optimization, the optimization is typically performed on preference data sampled from other models, and all the preference singles are equally treated. however, some preference data, distant from the current policy, are less informative for training, resulting in inefficient and suboptimal optimization. in this paper, we propose simulating on-policy preference optimization with off-policy preference data, combining the efficiency of off-policy rl with the performance benefits associated with onpolicy rl. our method is motivated by the following conceptual data generation process. this process begins with transforming the existing preference dataset into a preference labeling function. we can then resample a new preference dataset through bootstrapping from the existing data. this process involves uniformly sampling inputs from the preference dataset and online sampling new pairs of outputs with the current policy model. each pair is retained if it can be labeled by the labeling function; otherwise, it is rejected. we then perform dpo on the regenerated preference dataset. in practice, this bootstrapping process can be implemented with the weighted policy optimization (wpo) objective, where different preference pairs are reweighted according to the joint probability of their outputs. we further devise a weighting alignment mechanism to ensure that all on-policy generated pairs are equally weighted. in this way, wpo can effectively mitigate the distribution gap during rl without incurring additional costs. we evaluate wpo on instruction following benchmarks, including alpaca eval 2 @xcite and mt-bench @xcite . in the off-policy setting based on ultrafeedback @xcite , wpo improves the length-controlled winning rate against gpt-4-turbo on alpaca eval 2 by up to 14.9% over sft model, outperforming dpo by up to 5.6%. particularly, in the hybrid rl setting where the off-policy preference data is further enriched with on-policy outputs, wpo (figure 1 ) achieves a new sota length-controlled winning rate of 76.7% on alpaca eval 2. additionally, we find that wpo can be integrated into other loss functions for preference optimization and shows consistent improvements. furthermore, we systematically compare the model performance in different rl settings. our analysis reveals that the hybrid setting, which utilizes both on-policy and off-policy preference data, achieves the best results, and on-policy, dispreferred data is more important for preference optimization. to summarize, our contributions are three-fold: general alignment methods. the advancement of chatgpt has propelled significant advancements in the field of large language models (llms). notable models such as zephyr (tunstall et al., 2023) and gpt-4 (achiam et al., 2023) have effectively demonstrated the application of techniques like reinforcement learning from human feedback (rlhf; christiano et al. 2017; ouyang et al. 2022; glaese et al. 2022) and direct preference optimization (dpo; rafailov et al. 2023), highlighting their efficacy in achieving improved model alignment. these approaches, along with related methods such as sequence likelihood calibration @xcite and generalized preference optimization (gpo) @xcite , aim to refine the objectives of rlhf by clearly enhancing the distinction between more and less preferred outputs. additionally, the introduction of the direct nash optimization (dno) algorithm by @xcite represents a further innovation. this algorithm utilizes cross-entropy to assess the gap between actual and predicted win rates. practical applications more frequently rely on the iterative framework of dpo @xcite . yet, dpo often reveals a discrepancy between the output distributions produced by the policy and those in the preference dataset. to address this, we propose simulating on-policy reinforcement learning using off-policy data, thereby combining the benefits of on-policy rl with enhanced efficiency. on-policy reinforcement learning. self-play fine-tuning @xcite operates under an iterative framework akin to dpo, utilizing humanlabeled responses as "winners" and outputs from previous iterations as "losers" within each pairing. similarly, adversarial preference optimization @xcite incorporates contrastive losses, which obviate the need for direct feedback from annotators. this method introduces a tokenlevel loss function known as cringe loss @xcite , which differentiates the correct subsequent token from a deliberately incorrect token from the vocabulary. pairwise cringe loss @xcite utilizes this cringe loss mechanism within a continuously improving iterative training framework. moreover, the recent introduction of sami @xcite targets optimizing a lower bound on the conditional mutual information between prompts and responses through a contrastive estimation technique. in our approach, we adjust the importance of each pair in the training process by assigning greater weight to those pairs more likely to be sampled from the policy model, thus simulating on-policy reinforcement learning. ## method in this section, we provide the theoretical background of rlhf and dpo in section 3.1. we then introduce the distributional gap problem and algorithm 1: weighted preference optimization (wpo) input: dataset (d) with prompts and responses, policy lm π θ , total number of iterations t , learning rate α t , for @xmath2, y w , y l ) from d, calculate the alignment weight via eq. (2), compute l wpo via eq. ( 1 ), update policy parameters θ using gradient descent: θ ← θα t ∇θ(x, y w , y l , θ). end for propose the wpo method (algorithm 1) in section 3.2. finally, we explore how to better simulate on-policy rl through weight alignment in section 3.3. ## experiment in this section, we outline our experimental settings (section 4.1) and present the main results along with ablation studies (section 4.2). we then compare different rl settings (section 4.3). additional analysis of wpo is provided in appendix a. ## conclusion in this study, we tackled the distributional gap problem inherent in off-policy preference optimization. by introducing weighted preference optimization (wpo), we successfully simulated onpolicy preference optimization using off-policy preference data, merging the benefits of both approaches. our method not only addressed the distributional gap without incurring additional costs but also enhanced the effectiveness of preference optimization. extensive experiments demonstrate that wpo can produce better llms that are more closely aligned with human preferences.