RS - DPO : A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024)

Abstract: Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. However, proximal policy optimization (PPO) based RLHF is occasionally unstable requiring significant hyperparameter finetuning, and computationally expensive to maximize the estimated reward during alignment. Recently, direct preference optimization (DPO) is proposed to address those challenges. However, DPO often relies on contrastive responses generated from human annotator and alternative LLM, instead of the policy model, limiting the effectiveness of the RLHF. In this paper, we addresses both challenges by systematically combining rejection sampling (RS) and DPO. Our proposed method, RS-DPO, initiates with the development of a supervised fine-tuned policy model (SFT). A varied set of k responses per prompt are sampled directly from the SFT model. RS-DPO identifies pairs of contrastive samples based on their reward distribution. Finally, we apply DPO with the contrastive samples to align the model to human preference. Our experiments indicate that our proposed method effectively fine-tunes LLMs with limited resource environments, leading to improved alignment with user intent. Furthermore, it outperforms existing methods, including RS, PPO, and DPO.

## introduction state-of-the-art (sota) llms such as gpt-4 (ope-nai, 2023), llama @xcite etc., are trained with several stages. with pre-training and supervised instruction tuning, llms learns to follow specific instructions to complete various tasks with zero-shot or few-shot prompts @xcite . to further improve the llms' intelligence as close as to human and ensure a more helpful and harmless model, alignment is important as the last-mile llm training procedure @xcite @xcite . reinforcement learning with human feedback (rlhf) @xcite is the most adopted approach for alignment training and it usually involves training a reward model with human preference datasets which optimizes a reward function based on the human-annotated preference. then llms are fine-tuned to learn to maximize the reward of their responses using reinforcement learning algorithms, including proximal policy optimization (ppo) @xcite , reinforce @xcite , and similar variants. while ppo is used by sota llms due to its ease of use and good performance, training with ppo has few limitations, including complexity of training multiple llms, and sampling from policy model in training loop, high gpu memory requirement with hosting multiple llms during training, and sensitivity to training data and reward models. to make rlhf training more efficient, there are methods proposed from different perspective. in order to reduce the preference data effort by human annotation, @xcite and @xcite proposed to train the llm to align to the llm's preference rating in order to save human effort. @xcite proposed a combined strategy to merge sft and reward models as well as in ppo with lora selection in order to reduce latency and memory footprint. @xcite used reward model to select ranked high-reward good samples to supervise fine-tune the models and iteratively repeating this process yield good results. to reduce the memory and save training resources, @xcite proposed the direct preference optimization (dpo) to remove the need of training reward model, and directly optimize the policy model using a simple classification to maximize the difference between likelihood of human preference pairs. this method proves equivalent performance by implicitly maximize the reward. however, it is mainly trained on human preference data to learn the alignment, instead of sampling the policy model's response for optimization. llama2 @xcite adopts several rounds of rejection sampling to select the best samples from k model-generated samples for fine-tuning before ppo in order to boost the model performance. but rejection sampling only selects the best samples instead of preference pairs, with low data usage efficiency. rso @xcite proposes to generate preference data from the target optimal policy using rejection sampling, enabling a more accurate estimation of the optimal policy. compared to rso, our proposed method (rs-dpo) directly employs a point-wise reward model for response ranking and optimization, utilizing logistic loss exclusively during policy optimization. unlike rso's approach of statistical rejection sampling and tournament ranking for response generation and selection, rs-dpo generates a fixed number of responses per prompt and relies on computing reward gaps between responses for preference data generation, resulting in reduced computational expense. additionally, while rso lacks evaluation on standard alignment benchmarks and comparison against ppo, rs-dpo demonstrates its effectiveness against other rlhf methods on such benchmarks. in this work, we propose rs-dpo method for rlhf training that combines the advantages of existing efficient methods, including offline preference data generation using rejection sampling, and using dpo in order to reduce the training gpu memory consumption. specifically, rs-dpo generates responses from the large language model directly, and leverages rejection sampling (rs) to sample synthetic preference pairs based on the reward distribution of llms responses. then, it uses the generated preference pairs for alignment with dpo. the main contributions of our proposed rlhf training method can be summarized as follows: (1) rs-dpo demonstrates stability and robustness against variations in the reward model quality, consistently outperforming existing methods like dpo, ppo and rs. (2) in contrast to the rejection sampling approach that focuses solely on the best response among k generated responses for alignment, rs-dpo selects pairs of contrastive samples based the reward distribution, thereby enhancing overall performance. (3) rs-dpo samples contrastive data directly from the sft model, distinguishing itself from dpo which often relies on responses from alternative language models or human annotations. this approach contributes to the superior performance of rs-dpo. (4) our proposed method is efficient, being less resourceintensive compared to ppo, making it practical for applications in limited resource environments. ## method the aim of this study is to utilize reinforcement learning from human feedback (rlhf) to train a policy model with the purpose of aligning a large language model to user intent. as the pipeline shown in 1, our proposed method, rs-dpo, systematically combines rs and dpo. it starts by generating a diverse set of k distinct responses for each prompt, selecting a pair of contrasting samples based on their reward distribution. subsequently, the method employs dpo to enhance the performance of the language model (llm), thereby achieving improved alignment. our proposed method consists the following steps: ## experiments details this section outlines our experiments to demonstrate the effectiveness of our proposed rs-dpo method for the alignment task. we conduct all of our experiments on the llama-2-7b llm @xcite which is one of the state-of-the-art llms at 7b parameter scale. we perform supervised fine-tuning (sft), reward modeling (rm), dpo training, and ppo training based on huggingface transformer reinforcement learning (trl) library @xcite . we utilize deep-speed zero-3 @xcite for optimizing gpu memory and training speed. all experiments are conducted on 8 a100s gpus with 40g memory per gpu. ## results and ablations this section presents our main results. we show sample model completions in appendix d. to comprehensively assess the effectiveness of our proposed method, we employ a comparative analysis of various preference data generation policies. these policies guide the selection of the superior model response, denoted as y w , and the inferior model response, denoated as y l , from a set of k generated answers. the following preference data generation policies are considered: best-vs-worst: this policy ranks the k responses according to their respective rewards and selects the response with the highest reward as y w and the response with the lowest reward as y l . best-vs-random: this policy selects the response with the highest reward as y w , while y l is chosen randomly from the remaining k -1 responses. original annotation: this policy chooses y w and y l from the original preference data annotated by humans or larger models. ## discussion and conclusion in this paper, we proposed rs-dpo method that generates responses from the large language model directly, and leverages rs to sample synthetic preference pairs, and dpo for rlhf training. extensive experiments show the effectiveness of rs-dpo compared to existing methods including rejection sampling (rs), proximal policy optimization (ppo) and direct preference optimization (dpo). additionally, rs-dpo is stable, and is not as sensitive to the quality of the reward model as other methods. our proposed method also offers a more efficient and less time-consuming solution for the alignment task as compared to ppo, minimizing resource requirements. during rlhf training, ppo conducts online sampling from the policy model and evaluates them using the loaded reward model in real-time. consequently, ppo necessitates loading three models during training: the initial sft, policy model, and reward model, demanding a significant amount of gpu memory and decelerating the training process. furthermore, the online sampling from the policy model incurs increased memory consumption as the generated sequences lengthen. in practical terms, even with 1-2 moderate gpus, training a small-scale (e.g., 7b) llm using ppo is unfeasible. in our experiments, we had 8 a-100 gpus each having 40g memory, but we resorted to 8-bit quantization of both the policy and reward model to circumvent gpu memory constraints. our proposed method conducts response sampling offline from sft and constructs a dataset of synthetic preference data to bypass the high computational cost of ppo, while remaining viable on 1-2 moderate gpus. notably, the operational cost of running dpo and rs-dpo is identical; the sole disparity lies in rs-dpo performing offline sft sampling, rendering our proposed method an on-policy reinforcement learning approach. moreover, as emphasized by prior researches @xcite , ppo represents an unstable process prone to sensitivity towards reward model quality and hyperparameters, necessitating multiple runs to converge to a satisfactory model. for instance, in