Filtered Direct Preference Optimization (2024)

Abstract: Reinforcement learning from human feedback (RLHF) plays a crucial role in aligning language models with human preferences. While the significance of dataset quality is generally recognized, explicit investigations into its impact within the RLHF framework, to our knowledge, have been limited. This paper addresses the issue of text quality within the preference dataset by focusing on direct preference optimization (DPO), an increasingly adopted reward-model-free RLHF method. We confirm that text quality significantly influences the performance of models optimized with DPO more than those optimized with reward-model-based RLHF. Building on this new insight, we propose an extension of DPO, termed filtered direct preference optimization (fDPO). fDPO uses a trained reward model to monitor the quality of texts within the preference dataset during DPO training. Samples of lower quality are discarded based on comparisons with texts generated by the model being optimized, resulting in a more accurate dataset. Experimental results demonstrate that fDPO enhances the final model performance. Our code is available at https://github.com/CyberAgentAILab/filtered-dpo.

## introduction large language models (llms) have become pivotal in performing various language processing tasks, such as text generation, dialogue, and summarization @xcite @xcite . aligning these models with human preferences and ethical standards is paramount to ensuring they are practical, trustworthy, and socially accepted @xcite . reinforcement learning from human feedback (rlhf) is developed to tackle this challenge, aiming to enhance llm performance by leveraging human feedback @xcite @xcite @xcite . rlhf operates by taking a preference dataset and a language model (lm) as inputs to produce an lm refined by these preferences @xcite . it is broadly divided into two approaches concerning the use of a reward model (rm): rmbased rlhf, which learns an rm from the preference dataset and then uses it to optimize an lm through reinforcement learning (rl), and an rmfree approach that directly adjusts an lm based on preference data. this division mirrors the distinction between offline model-based and model-free rl @xcite . 1 each approach offers unique advantages and requires careful application based on specific goals and contexts. for instance, in scenarios with limited data, modelbased rl might be preferable due to its data efficiency, though its computational cost is generally higher than that of model-free rl @xcite . consequently, rm-based rlhf may be more effective in leveraging data than rm-free methods, despite the higher computational cost and algorithmic complexity. direct preference optimization (dpo) is a representative method of the rm-free rlhf @xcite . dpo reformulates the rl problem as a type of supervised learning problem, bypassing key challenges in rm-based rlhf, such as the need for reward modeling and balancing exploration and exploitation in rl fine-tuning. thus, dpo simplifies the learning process. however, this approach relies solely on the initially given preference dataset for training, similar to supervised learning. this reliance might make dpo more in this paper, we explore the impact of preference dataset quality on the performance of lms optimized by dpo, specifically focusing on the quality of response texts rather than labeling accuracy. we demonstrate that dpo is more affected by text quality variations within the dataset than typical rlhf methods, as shown in figure 1 (a ). notably, we observe that lower-quality data can create performance bottlenecks. in realistic applications of llm alignment, the quality of responses can be highly diverse due to several factors such as differing skill levels among experts creating responses and the need to combine manually generated responses with those automatically generated by llms to manage annotation costs. this quality variation in response quality can severely impact performance of dpo. in response to this challenge, we introduce a novel approach named filtered direct preference optimization (fdpo), which aims to harness potential data efficiency advantages of rm-based rlhf. it uses a trained rm to identify and discard samples of lower quality than those generated by an lm during fine-tuning. our experiments show that fdpo significantly enhances the effectiveness of dpo, as illustrated in figure 1 ( @xcite . for simplicity, we will henceforth refer to rmbased rlhf simply as rlhf, unless a distinction is necessary. this study's contributions are threefold: the remainder of this paper is organized as follows: section 2 reviews related work. section 3 explains the background. in section 4, we detail the proposed method, fdpo, explaining its mechanisms and the rationale behind its design. section 5 presents the experimental results, illustrating the effectiveness of fdpo and its impact on lm performance. finally, section 6 concludes the paper, and section 7 discusses limitations and directions for future work. ## related work we examine methods for aligning lms with human preferences, focusing on rlhf and its alternatives. most rlhf approaches utilize an rm @xcite @xcite . these methods fine-tune lms using rl algorithms such as reinforce @xcite , proximal policy optimization (ppo) @xcite , or their variants @xcite . however, there are notable reinforcement-learning-free approaches @xcite , and learning-free methods that leverage the rm at decoding time, with best-of-n (bon) sampling being a prominent example @xcite . a significant challenge in these methods is the estimation error of rms, which can lead lms to overfitting to a proxy reward, a phenomenon termed rm overoptimization @xcite . various strategies have been proposed to address this issue, including rm ensembles @xcite , uncertainty evaluation @xcite , and analysis of out-of-distribution @xcite . @xcite proposes using bon sampling to improve the data used for reward modeling, which is relevant to our fdpo approach focusing on dataset quality. as fdpo also leverages an rm, it can benefit from these developments. @xcite @xcite represent significant rm-free methods. some dpo variants explore different regularizations @xcite or use a divided dataset for stepwise training @xcite . other variants propose adapting dpo online @xcite , where an rm is used to evaluate newly generated training data. while both online dpo and fdpo employ rms, fdpo remains an offline approach that filters the dataset and does not use generated data for training. this distinction makes fdpo less reliant on the accuracy of rms, as it focuses solely on data refinement rather than reward optimization. @xcite , on the other hand, utilizes an rm for rejection sampling to adjust the distribution of the preference data by aligning it with the optimal lm distribution. in contrast, fdpo focuses on filtering low-quality data. other dpo variants evaluate the quality difference between chosen and rejected responses for adding an offset to the dpo objective function @xcite or incorporating curriculum learning @xcite . these approaches focus on response quality, which is relevant to our method. despite various advancements in dpo, the dependence on preference dataset quality has not been thoroughly analyzed. our study aims to explore this significant dependence and attempts to refine the dataset for better performance. additionally, our proposed fdpo method complements most of these developments. integrating fdpo with these methods is an exciting possibility for future work, potentially leading to even more effective ways to align lms with human preferences. ## background this section explains rlhf in section 3.1 and explores dpo in section 3.2. ## filtered direct preference optimization in this section, we propose an approach called filtered direct preference optimization (fdpo), which refines the dataset used in dpo. the principle of fdpo is straightforward: it aims to discard lowerquality samples compared to those generated by the lm. this strategy is intuitively derived from observing that lower-quality data can create performance bottlenecks in dpo. first, we give an implementation of fdpo in section 4.1. then, we will elaborate on the motivation of fdpo by analyzing dpo's behavior in section 4.2. ## experiments we first detail our setup regarding pretrained models in section 5.1 and datasets in section 5.2. we then evaluate the impact of data quality on dpo in section 5.3 and the effectiveness of fdpo in section 5.4 on instruction following tasks using the alpacafarm dataset @xcite , focusing on the general ability to generate appropriate responses to prompts. furthermore, we assess fdpo on the anthropic hh datasets @xcite in section 5.5, under a realistic setting where there are two types of responses: dataset responses and those generated by the sft model. this setup closely mimics real-world applications, where the system must handle both pre-existing and newly generated responses. for our baseline comparison, we use dpo and ppo-based rlfh implementations from the transformer reinforcement learning (trl) library. 2 all experiments are conducted using a single nvidia a100 accelerator. the experiments using the alpacafarm dataset took approximately 3 days, and those using the anthropic hh datasets took about 9 days of computation time. details of the experimental parameters are provided in appendix c.1. ## conclusions this study explores how the quality of a preference dataset impacts lms optimized using dpo, especially when compared with the rlhf method. we found that the quality of chosen responses significantly influences dpo performance. to address this, we proposed filtered dpo (fdpo), which uses a reward model to identify and discard lowerquality data, refining the dpo process. our experiments demonstrated that fdpo improved dpo's performance, effectively handling datasets with quality discrepancies. while the use of a reward model introduces additional computational costs and complexity, it allows for more effective leveraging of limited data. overall, this highlights the practical value of fdpo's approach, especially in scenarios where data quality is heterogeneous.