Enhancing Alignment using Curriculum Learning & Ranked Preferences (2024)

Abstract: Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data (one chosen and rejected response per prompt) to align LLMs to human preferences. In practice, multiple responses could exist for a given prompt with varying quality relative to each other. We propose to utilize these responses to create multiple preference pairs for a given prompt. Our work focuses on aligning LLMs by systematically curating multiple preference pairs and presenting them in a meaningful manner facilitating curriculum learning to enhance the prominent DPO technique. We order multiple preference pairs from easy to hard, according to various criteria thus emulating curriculum learning. Our method, which is referred to as Curri-DPO consistently shows increased performance gains on MTbench, Vicuna bench, WizardLM, highlighting its effectiveness over standard DPO setting that utilizes single preference pair. More specifically, Curri-DPO achieves a score of 7.43 on MTbench with Zephyr-7B, outperforming majority of existing LLMs with similar parameter size. Curri-DPO also achieves the highest win rates on Vicuna, WizardLM, and UltraFeedback test sets (90.7%, 87.1%, and 87.9% respectively) in our experiments, with notable gains of up to 7.5% when compared to standard DPO. We release the preference pairs used in alignment at: https://huggingface.co/datasets/ServiceNow-AI/Curriculum_DPO_preferences.

## introduction aligning llms with carefully curated human feedback has shown to be critical in steering their response behavior @xcite @xcite . to align llms towards good responses, preference optimization methods such as reinforcement learning from human feedback (rlhf) (christiano et al., 2017; * equal contribution @xcite and its rl-free closedform counterpart -direct preference optimization (dpo) @xcite -are an active area of research. dpo is a proven technique that circumvents the complex rlhf pipeline by directly using preferences to finetune llms using a supervised learning loss. while dpo has shown impressive performances @xcite , it is limited to a single pair of responses per prompt (one chosen and one rejected). however, several high-quality responses could exist for a single prompt @xcite , thus resulting in multiple preference pairs per prompt for preference optimization. several ongoing and concurrent alignment methods have utilized multiple preference responses. for example, @xcite proposed lipo where the policy is directly optimized on a listwise ranked preferences. parallel to these, our approach is still primarily focused on pairwise preference optimization but with multiple preference pairs that are sequentially ranked during training. we hypothesize that the use of multiple preference pairs per prompt in the dpo framework could act as a form of data augmentation. while it may be tempting to simply collate these pairs and perform dpo training, we show that systematically introducing them to the preference optimization model is important to achieve better result. in addition, the relative quality ratings of multiple responses can be utilized for incremental preference optimization. to this end, we propose curri-dpo foot_0 , that draws inspiration from curriculum learning to organize multiple preference pairs systematically across dpo iterations thus resulting in substantial improvements. curriculum learning is a training paradigm that arranges data samples in a purposeful order with the aim of improving model performance @xcite . it has been shown to benefit the learn-ing process for both humans and machines @xcite . given a preference pair, if the chosen and rejected responses are further apart (based on a determined criteria, e.g. reward or their quality score), it would be easier for the preference model to learn distinguishing signals between chosen and rejected within the dpo framework @xcite . however, if the chosen and rejected responses have near similar quality, it would be harder for the model to learn contrastive signals. inspired by this, we order the multiple preference pairs from easy (chosen and rejected responses are farthest apart) to hard (chosen and rejected responses are closest) during dpo training (shown in fig. 1 ), resulting in improved performance. our proposed curriculum learning based dpo method -curri-dpo, significantly outperforms the standard single preference pair dpo on several benchmarks, including mt bench, wizard-lm, openassistant, and ultrafeedback test sets. although, we focus on dpo with multiple preference pairs in a curriculum learning setup, our approach can be easily extended to other preference optimization methods such as sequence likelihood calibration (slic) @xcite . the key contributions of our work are: • we introduce curri-dpo that incorporates curriculum learning with multiple preference pairs into the dpo framework. curri-dpo demonstrates strong improvements over sft and standard single preference pair based dpo with an mtbench of 7.43 (with a 7b llm and much lesser) training data) and achieves strong gains on wizardlm (upto 7.5% gains), ultrafeedback test set (up to 5.1% gains) and vicuna bench. ## approach our work is focused on developing curriculum learning based method for utilizing multiple pairs of preference data, with varying degrees of data quality, in the dpo framework. the main steps in our approach are to sample and arrange these multiple preference pairs for curriculum learning. we explain methodologies for each step below: ## results the key observations from our experiments are: ## analysis exhaustive combinations of preference pairs -we present our experiments on 3 sets of preference pairs where the best rated response (r i 1 ) was selected as chosen. one could easily create more combinations of preference pairs. for example, with 4 responses for each prompt, there are 4c2 5 detailed improvements in different categories of mtbench are shown in fig. 2 in appendix @xmath0. we train curri-dpo iteratively with 4c2 preference pairs after arranging them based on their rating score difference. as shown in safety evaluation -we evaluate our curri-dpo and baseline dpo trained with single preference pair on the llm jail break & safety dataset @xcite . the dataset contains various prompts that are specifically targeted to disrupt alignment and elicit harmful responses from llms. we observed distinctive benefits of curri-dpo on safer response generation over baseline dpo model. we show two examples in technique jail prosocial-toxic break dialogue comment (win rate) (accuracy) (accuracy) sft -47.1 55.1 dpo (r1,r4) 59.4 52.9 54.1 curri-dpo 69.0 65.5 55.3 ## discussion although, our work focuses on aligning llms by curating preference pairs and presenting them iteratively to enable curriculum learning to enhance reward-model-free techniques, our approach can also be extended to fine-tune reward models which can subsequently be used with ppo @xcite , rrhf @xcite , reinforce @xcite for rlhf alignment. we would also like to emphasize that curri-dpo is orthogonal to many concurrent extensions of dpo. curriculum training can be applied to dpo variants such as ipo @xcite and kto @xcite , as well as listwise preference optimization methods like lipo @xcite and slic @xcite . it can also integrate into each step of sdpo @xcite . additionally, creating multiple preference pairs could complement iterative dpo methods like spin @xcite for further improvements. however, explaining and incorporating all these approaches, each with its own curriculum setup, is beyond the scope of this paper. ## conclusion in this work, we presented curri-dpo that utilizes multiple pairwise preference data to further improve upon existing prominent dpo method. we showed that curriculum learning based iterative dpo training can achieve strong improvements over the vanilla dpo trained that only utilizes single preference pairs, thus highlighting unrealized potential of dpo method for preference optimization for future works. furthermore, our strong results demonstrates that -curri-dpo -is highly effective and establishes motivations for future works on preference optimization to strongly consider curriculum learning and iterative training.