Hybrid Alignment Training for Large Language Models (2024)

Abstract: Alignment training is crucial for enabling large language models (LLMs) to cater to human intentions and preferences. It is typically performed based on two stages with different objectives: instruction-following alignment and human-preference alignment. However, aligning LLMs with these objectives in sequence suffers from an inherent problem: the objectives may conflict, and the LLMs cannot guarantee to simultaneously align with the instructions and human preferences well. To response to these, in this work, we propose a Hybrid Alignment Training (Hbat) approach, based on alternating alignment and modified elastic weight consolidation methods. The basic idea is to alternate between different objectives during alignment training, so that better collaboration can be achieved between the two alignment tasks. We experiment with Hbat on summarization and dialogue tasks. Experimental results show that the proposed Hbat can significantly outperform all baselines. Notably, Hbat yields consistent performance gains over the traditional two-stage alignment training when using both proximal policy optimization and direct preference optimization.

## introduction alignment training is a key technique to ensure that the behaviors of large language models (llms) are consistent with human intentions and preferences @xcite . it typically involves two stages: 1) using human-labeled data to train pre-trained llms via a supervised training method, which enables llms to understand human intentions and follow the instructions (call it instruction-following alignment), and 2) employing approaches like proximal policy optimization (ppo) @xcite and direct preference optimization (dpo) @xcite to learn preferences from human feedbacks (call it human-preference alignment). this paradigm has achieved promising results on several downstream tasks, such as dialogue @xcite @xcite , summarization @xcite , and machine translation @xcite . however, this two-stage alignment training has its inherited limitation: the optimization objectives are different for each stage, which can make an optimization conflict @xcite . such limitation could result in an inferiorly aligned llm in real-world scenarios. this phenomenon is also described in @xcite 's work, which is referred to as alignment tax. to mitigate this limitation, in this work, we propose a hybrid alignment training (hbat) approach, which offers a refinement of the collaboration among instruction-following alignment and human-preference alignment by using the following two methods. for one, inspired by interactive methods in multi-objective optimization @xcite , we propose an alternating alignment method, where the humanpreference alignment acts as a decision maker and continuously interacts with the instructionfollowing alignment to achieve a preferred alignment. specifically, we divide the instructionfollowing and human-preference training set into equal portions of mutually exclusive subsets, respectively. then, we rearrange these subsets in alternating orders during alignment training. furthermore, we introduce a modified elastic weight consolidation (ewc) @xcite to alternating alignment. ewc is a method to dynamically impose an appropriate constraint on each parameter when training a model with a new optimization objective, thereby easing an optimization conflict with the previous objective. we experiment with the proposed hbat on summarization and dialogue tasks based on llama2-7b and llama2-13b models @xcite . experimental results show that hbat can significantly surpass all baselines. notably, based on the llama2-13b model, hbat can yield a +2.26 rouge-l points improvement for the summarization task, compared to the traditional rlhf. additionally, our hbat significantly outperforms the sft over 21.01 gpt-4 win rate points on the dialogue task based on the llama2-13b model. furthermore, hbat is orthogonal to other optimized alignment approaches. for instance, when armed with esrl @xcite , our hbat gains an additional improvement of 2.59 gpt-4 win rate points on the summarization task. multi-objective optimization. multi-objective optimization problem involves optimizing multiple optimization objectives simultaneously @xcite . however, there does not typically exist a feasible solution that minimizes all loss functions. therefore, researchers always explored a pareto optimal solution that cannot be improved in any of the objectives without impairing at least one of the other objectives. recent works on this exploration could be classified into three groups. the first group focused on pareto dominance-based method. this method maintains the individual elements of the solution vectors as independent during optimization @xcite . the second group tended to design an quality indicator, such as hypervolume @xcite and r2 @xcite , to act as a proxy objective instead of optimization objectives. the third group that has attracted attention commonly aimed to solve multi-objective optimization problems through an interactive method. a typical interactive method requires a decision maker to offer preference information, which allows to search for the most preferred pareto optimal solution after each optimization @xcite @xcite . although the alignment training is not a standard multi-objective optimization problem, its goal remains consistent, i.e., seeking an aligned llm that simultaneously aligns instructions and human preferences well. ## background despite the extensive knowledge endowed from pre-training, llms are difficult to produce content that humans want. this is because that pretrained llms lack understanding of input instructions and human preferences. to address this, we often perform alignment training on them, first for instruction-following alignment and then for human-preference alignment. ## method in this work, we aim to solve an optimization conflict limitation during alignment training. we propose the hbat to achieve this. the overview of hbat is depicted in figure 1 . as shown in the figure, we propose the alternating alignment and modified ewc in hbat to achieve our goal. in the following subsections, we will describe them. ## experimental setup we evaluated hbat on summarization and dialogue tasks based on the commonly used llama2-7b and llama2-13b models. ## conclusion in this paper, we focus on solving the optimization conflict of alignment training in llms. we have proposed a hybrid alignment training (hbat) via the alternating alignment and modified elastic weight consolidation methods. our extensive experiments show that our hbat can significantly outperform all baselines.