BPO : Staying Close to the Behavior LLM Creates Better Online LLM Alignment (2024)

Abstract: Direct alignment from preferences (DAP) has emerged as a promising paradigm for aligning large language models (LLMs) to human desiderata from pre-collected, offline preference datasets. While recent studies indicate that existing offline DAP methods can directly benefit from online training samples, we highlight the need to develop specific online DAP algorithms to fully harness the power of online training. Specifically, we identify that the learned LLM should adhere to the proximity of the behavior LLM, which collects the training samples. To this end, we propose online Preference Optimization in proximity to the Behavior LLM (BPO), emphasizing the importance of constructing a proper trust region for LLM alignment.We conduct extensive experiments to validate the effectiveness and applicability of our approach by integrating it with various DAP methods, resulting in significant performance improvements across a wide range of tasks when training with the same amount of preference data. Even when only introducing one additional data collection phase, our online BPO improves its offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on Anthropic Helpfulness in terms of win rate against human reference text.

## introduction reinforcement learning from human feedback (rlhf) @xcite has achieved tremendous success in aligning the powerful pretrained large language models (llms) with human preference @xcite @xcite , revolutionizing human society. however, traditional rlhf methods @xcite are computationally expensive due to their two-stage training pipeline that consists of a reward modeling phase and can suffer from rl training instability @xcite . to address these issues, recent advances in direct alignment from preferences (dap) methods provide solutions to avoid learning a reward model (rm) and stabilize the training process. prominent examples include direct preference optimization (dpo) @xcite and its variants @xcite @xcite , which directly optimize lms using a static, pre-collected set of preference data, streamlining the alignment procedures. however, recent studies @xcite @xcite @xcite identified that aligning an llm with offline preference datasets prevents the llm from getting feedback for its own generations. these studies emphasize the importance of incorporating online data generated by intermediate models during training. while empirical evidence shows that offline dap methods can directly benefit from online preference data, we argue that making algorithm-level adjustments is essential to fully harness the power of online training. specifically, we identify that existing online dap methods @xcite @xcite @xcite do not adjust the trust region designed in offline dap methods @xcite . these methods still construct their trust region @xcite by penalizing the kl divergence between the learned llm and a fixed reference model π ref , even when training samples are dynamically generated by intermediate models. drawing inspiration from existing rl literature @xcite @xcite , we propose online preference optimization in proximity to the behavior llm (bpo), emphasizing that a better trust region should be instead constructed around the behavior llm π β that collects the training samples. in other words, we should set π β as π ref when performing online dap. we provide an overview of our training pipeline in fig. 2 . however, setting a dynamic π ref during online dap can lead to instability. to mitigate this issue, we propose optimizing an ensemble of lora @xcite weights and merging them during infer-ence. we verify the effectiveness of our method by building on top of various dap methods, including dpo @xcite , @xcite and slic @xcite . empirically, we show that our bpo significantly improves over their online and offline dap counterparts on tl;dr @xcite , anthropic helpfulness and harmlessness @xcite , demonstrating the generalizability of our methods. on the other hand, conducting iterative preference annotations at each training step can be practically infeasible when hiring human annotators. given the same annotation budget, we anticipate that a successful online dap method to perform well at a low annotation frequency. in other words, we aim to minimize the number of preference annotation phases throughout the training. to this end, we evaluate our method with different annotation frequencies while keeping the total amount of preference data constant. we demonstrate that even with just one additional preference annotation phase compared to offline dpo, our bpo significantly improves over its offline dpo counterpart from 72.0% to 80.2% on tl;dr and from 82.2% to 89.1% on anthropic helpfulness in terms of win rate against human reference text. furthermore, we conduct an ablation study to verify our performance improvement comes from our better trust region constructed around π β instead of π β 's higher quality compared to π ref . our results show that even when using a high-quality llm as π ref for online dap baselines, our ap-proach still outperforms it. our contributions are three-fold: reinforcement learning from human feedback methods @xcite @xcite @xcite first learn a rm from a set of preference data based on the bradley-terry modelization @xcite and then leverage an rl algorithm, e.g., ppo @xcite , to optimize a pretrained or supervised finetuned (sft) llm towards the learned rm. however, the two-stage learning process is computationally demanding and rl training can suffer from training instability @xcite . to address these issues, recent advancements, including dpo @xcite and dpo variants @xcite @xcite @xcite , enable direct alignment from a fixed, offline set of preference data. these methods leverage the dual form of the original rlhf objectives, successfully eliminating the reward modeling phase and stabilizing the training. empirically, these methods achieve impressive performance on standard evaluation benchmarks @xcite @xcite . concurrently, tr-dpo @xcite also explores setting a dynamic π ref for dap methods. however, tr-dpo only considers offline dap settings and thus never considers setting π @xmath0. instead, tr-dpo explores setting π ref as the moving average of their π θ or periodically updates π ref with π θ . therefore, tr-dpo is substantially different from our methods. online dap methods. recent studies @xcite @xcite @xcite @xcite have recognized the importance of on-policy training data. specifically, these methods collect human preference on the responses generated from intermediate models and use them for training. although these online methods improve their offline counterparts, they still constrain the kl divergence between the learning llm and a static reference model. in this paper, we provide extensive empirical results demonstrating that the reference llm should be set dynamically as the behavior llm that collects the training data during online preference learning. ## improving online dap by constructing a better trust region as discussed in sec. 3, offline dap methods never update the preference dataset d with new samples after the initial data collection stage. consequently, the learning llm π θ will gradually deviate from the training data distribution as the training continues. to mitigate the distribution shift, several @xcite @xcite have advocated for online dap training and provided empirical evidence to demonstrate the benefit. however, existing online dap methods still constrain kl divergence between π θ and a fixed reference model π ref even when annotating new preference data online. in this paper, we propose bpo, arguing that we should construct a better trust region by constraining the kl divergence between the learning π θ and the behavior llm π β . given a triplet of (x, y w , y l ), π β that generates (y w , y l ) given x and a dap loss function l dap , our loss function of l bpo is defined as below: algorithm 1 provides the pseudo-codes. we use f to denote the preference annotation frequency. we can simulate different dap settings by varying the value of f . for example, when we use a preference simulator (ai feedback), we can set @xmath1, meaning that we collect new preference data with π θt at every training step for training, corresponding to the on-policy dap setting. when using human annotation, we need to lower the value of f . when setting @xmath2, we reduce to the offline dap settings, where d is collected by π sft and π @xmath3. in summary, we observe that setting a dynamic π @xmath4. to overcome this challenge, we optimize an ensemble lora @xcite weights of the llm to stabilize the training. we linearly merge the lora weights during inference without incurring additional inference overhead. ## experiments we conduct experiments to address the research questions below: dataset & evaluation metric we performed our experiments using reddit tl;dr @xcite , anthropic helpfulness and harmlessness @xcite dataset. we split training data of 65k, 10k, and 10k for tl;dr, helpfulness, and harmlessness, respectively, to perform supervised fine-tuning (sft). all sft data is selected based on preferred responses. during the alignment stage, we use another training set that contains prompts that are different from sft to perform sampling and dap. we have 10k prompts for each of the three tasks. in this study, we are investigating the performance of dap algorithms when giving a fixed annotation budget. in practice, the annotation is performed by human raters. to ensure the reproducibility and scalability of our study, we use an open-sourced model, rm-deberta foot_0 as our preference simulator. the pairwise preference data is labeled by our preference simulator. rm-deberta has been trained on various preference pair datasets, including webgpt comparisons @xcite , open summarization @xcite and anthropic hh-rlhf @xcite , covering all tasks that we studied in this paper. we use preference simulator to annotate and evaluate our method and baselines. although we use a preference simulator for annotation, we investigate different data collection scenarios in sec. 5.2 to extend our approach to realistic online setting. baseline models in this study, we consider three baseline dap methods: dpo, slic, and ipo. by building on top of these methods, we learn corresponding bpo (dpo), bpo (slic), and bpo (ipo) and compare performance against their online and offline dap counterparts. implementation we use the development set to select the best-performed sft checkpoint. our batch size is 16, 64 and 16 for tl;dr, helpfulness, and harmlessness tasks, respectively. we train 625, 150, and 625 steps for each corresponding task. we set learning rate to be 5e-5 for both sft and preference learning. we fixed the regularization coefficient β = 0.1 for all preference learning methods. we leverage gemma-2b @xcite as our base lm. all dap methods optimize an ensemble of 5 lora weights on top of the base model. ## conclusion in this work, we propose bpo, an algorithm tailored to online dap training by constraining the divergence between learned llm and the behavior llm. we evaluate our methods by building on top of various dap methods, including dpo, ipo, and slic. we compare the performance of our bpo against its offline and online dap counterparts on tl;dr, anthropic helpfulness, and harmlessness, demonstrating significant performance improvement. we stabilize its training by optimizing an ensemble of lora weights. moreover, we show that our bpo can be applicable to different preference annotation frequencies f with a fixed amount of total training preference data. even by setting @xmath5, our bpo (dpo) substantially improves over offline dpo and at least matches on-policy dpo, demonstrating remarkable applicability.