LION s: An Empirically Optimized Approach to Align Language Models (2024)

Abstract: Alignment is a crucial step to enhance the instruction-following and conversational abilities of language models. Despite many recent works proposing new algorithms, datasets, and training pipelines, there is a lack of comprehensive studies measuring the impact of various design choices throughout the whole training process. We first conduct a rigorous analysis over a three-stage training pipeline consisting of supervised fine-tuning, offline preference learning, and online preference learning. We have found that using techniques like sequence packing, loss masking in SFT, increasing the preference dataset size in DPO, and online DPO training can significantly improve the performance of language models. We then train from Gemma-2b-base and LLama-3-8b-base, and find that our best models exceed the performance of the official instruct models tuned with closed-source data and algorithms. Our code and models can be found at https://github.com/Columbia-NLP-Lab/LionAlignment.

## introduction large language models (llms), pre-trained on datasets of trillion-scale tokens, have shown remarkable performance across a wide range of natural language processing tasks @xcite @xcite . however, these pre-trained models often struggle to follow human instructions and generate responses that are unsafe or inappropriate @xcite . recent research has increasingly focused on aligning llms: this includes many new algorithms based on reinforcement learning @xcite @xcite , new datasets to facilitate preference learning @xcite , and new training pipelines to improve the overall alignment * denotes equal contribution. in this work, we replicate modern alignment pipelines @xcite and analyze sources in the training process that could affect performance. in the three-stage training of supervised fine-tuning (sft), offline preference learning, and online preference learning, we find that: 1) sequence packing and loss masking significantly enhance sft, 2) scaling offline preference datasets improves overall performance, and 3) online learning greatly benefits chat benchmarks. then, we aggregate our findings and fine-tune from gemma-2b-base @xcite and llama-3-8b-base (ai@meta, 2024) using publicly available datasets and open-source algorithms. we evaluate our models on popular benchmarks such as arena-hard @xcite , alpacaeval-2 @xcite , mt-bench @xcite , and openllm @xcite . in all benchmarks, our models exceed the performance of the officially instruct models, which rely on closed-source datasets and algorithms. we believe our easily reproducible study offers useful insights for alignment research, and our fine-tuned models are valuable for downstream applications. our contributions are: ## preliminaries traditional reinforcement learning for human feedback (rlhf) methods typically starts with supervised fine-tuning, then trains a reward model r mimicking human preferences, and finally optimizes a language model π θ to maximize the reward @xcite @xcite . however, this process can be complex and difficult to tune. many recent works @xcite have proposed alternative algorithms. in this work, we examine a three-stage rlhf pipeline: 1) supervised fine-tuning; 2) offline preference learning with dpo @xcite ; and 3) online preference learning with dpo. below, we review each training stage. supervised fine-tuning stage supervised finetuning (sft) maximizes the log likelihood of the ground truth response y given a user's query x: efficiently optimizing the above objective is a challenging problem. we explore various strategies for sft, which are detailed in section 3.2. ## alignment procedure analysis this section explores and quantifies which choices are important to align language models. we introduce our experimental setups in section 3.1, and analyze different configurations in each stage of the training process in section 3.2, section 3.3, and section 3.4, respectively. for a controlled study, we fix the model architecture to gemma-2b @xcite . ## the lion series in the previous section, we empirically analyzed the best strategies to perform supervised finetuning, offline preference learning, and online preference learning. we aggregate these findings to train a series of models, the lion series, and evaluate them on numerous llm benchmarks. ## related work many recent studies extensively explored alignment methods to improve llms' ability to follow human instructions. early approaches train llms with supervised fine-tuning (sft) using high-quality human-written demonstrations @xcite @xcite . this method enjoys various properties such as fast convergence and scaling laws in training data/model sizes @xcite . however, sft is vulnerable to exposure bias, and can generate outputs that do not align well with human intent. to this end, reinforcement learning from human feedback @xcite @xcite was proposed. these work typically employs an online rl algorithm (such as ppo, @xcite ) to optimize llms towards a reward model mimicking human preferences. although effective, ppo can be complex to implement and often suffers from high reward variance, making it challenging to maintain stable performance @xcite . to address the limitations of ppo, many recent work focused on offline preference learning algorithms, such as direct preference optimization @xcite , kto @xcite , cto @xcite , and more @xcite . these algorithms are much easier to use, and spurred many recent studies to explore: 1) collecting high-quality offline preference data @xcite @xcite , and 2) designing better training pipelines such as iterative dpo and online @xcite @xcite . although these contributions demonstrate improvements, the training processes, datasets, and hyper-parameters often remain heterogeneous, and it is difficult to understand the source of improvements. our goal is to provide a comprehensive analysis of the alignment pipeline starting from sft to online preference learning, and to serve as a reference point for better understanding in the alignment process. ## conclusion we present a detailed analysis of modern alignment pipelines, and present a step-by-step recipe to finetune models using only publicly available datasets and open-sourced algorithms. we find that model performance substantially improves by 1) using packing and loss making in sft, 2) scaling offline preference datasets and training steps in dpo, and 3) training with online preference data to improve chat performance. we then aggregate our findings and train the lion series, and show that they outperform the officially released instruct models as well as models of larger sizes on benchmarks such as arena-hard-auto, alpacaeval-2, mt-bench, and openllm. these results illustrate the importance of many previously overlooked design choices, and serve as a reference point for future work in alignment research.