Phased Instruction Fine-Tuning for Large Language Models (2024)

Abstract: Instruction Fine-Tuning, a method enhancing pre-trained language models’ capabilities from mere next-word prediction to complex instruction following, often employs a one-off training approach on diverse instruction dataset. However, this method may not effectively enhance models’ adherence to instructions due to the simultaneous handling of varying instruction complexities. To address this, we propose a novel phased instruction fine-tuning (Phased IFT) method, grounded in the hypothesis of progressive alignment, which posits that the transition of a pre-trained language model from simple next-word prediction to sophisticated instruction following is a gradual learning process. Specifically, we obtain the score of difficulty for each instruction via GPT-4, stratify the instruction data into subsets of increasing difficulty, and sequentially uptrain on these subsets using the standard supervised loss. Through extensive experiments on the pre-trained models Llama-2 7B/13B, and Mistral-7B using the 52K Alpaca instruction data, we demonstrate that Phased IFT significantly surpasses traditional one-off instruction fine-tuning (One-off IFT) method in win rate, empirically validating the progressive alignment hypothesis. Our findings suggest that Phased IFT offers a simple yet effective pathway for elevating the instruction-following capabilities of pre-trained language models.

## introduction instruction fine-tuning (ift) @xcite , involving training on instruction dataset using standard supervised finetuning method, aligns pre-trained language models to users's intent and has been proven as an effective alignment method to enhance their ability to follow instructions. large language models (llms) are pre-trained on raw text data using maximum likelihood estimation, equipping them with the basic ability to predict the next word @xcite . however, a gap exists between this ability and following user intentions @xcite . to bridge this gap and enable models to complete human end tasks, various instruction fine-tuning strategies have been proposed, including sft @xcite , lima @xcite , alpaca @xcite , alpagasus @xcite , cot @xcite , superfiltering @xcite and self-instruct @xcite . among these, sft and lima employ human-written instruction data for fine-tuning, while strategies like self-instruct, alpaca, alpagasus utilize chatgpt as a teacher to automatically generate extensive instruction datasets. their training approach, a one-off ift on the whole instruction data without differentiating the difficulty levels of instructions, lacks efficiency in enhancing the instruction-following capability of pre-trained models. an instruction sample comprises a triplet of an instruction, an optional input, and an output @xcite . the instruction describes the task, the input serves as an additional context to the in-struction, and the output is the answer following the instruction. diversity @xcite @xcite is a crucial aspect of a high-quality instruction dataset, indicating that such datasets are typically extensive, encompassing a wide range of tasks and examples representing different levels of instruction-following difficulty. for instance, tasks like mathematical problem-solving, code writing, entity extraction, and copy generation each present varying levels of difficulty. even within the same category of instructions, the difficulty can vary with the length of the input (refer to however, previous instruction fine-tuning methods have treated large-scale instruction dataset uniformly, feeding them to the pre-trained model for one-time alignment training without differentiating the role of instructions of varying difficulties in the fine-tuning process. this approach overlooks the nuanced differences in instruction complexities and potentially limits the efficiency of model training. to address the aforementioned issue, this paper introduces an instruction difficulty scoring mechanism. specifically, the difficulty of an instruction is defined by the complexity of the instruction itself, the input, and the challenges involved in generating the output. we employ the strongest available models (such as @xcite ) as a teacher to score the difficulty of each instruction and input, as well as the challenge in generating the output, on a scale from 1 to 5, where higher scores indicate greater difficulty. further, we plot probability density curve of the instruction difficulty as a heuristic guide @xcite , coupled with expert judgment to select thresholds for difficulty scores. this approach enabled us to segment the instruction dataset into multi-stages sub-datasets, forming a sequence with increasing difficulty levels. building upon this, we further propose the progressive alignment hypothesis: aligning a pre-trained model's existing ability to predict the next word with the capability to generate content following human intent is a gradual learning process, progressively attaining alignment with human intent. based on the hypothesis, we develop a phased instruction fine-tuning (phased ift) method, diverging from the traditional one-off ift approach. phased ift represents an effective sequential up-training framework, which entails progressively training on sub-instruction datasets of incremental difficulty. initially, training commences on an easy sub-dataset with the standard supervised loss. following training completion, the model checkpoint is saved and subsequently utilized to extend supervised training to a marginally more complex sub-dataset. this iterative process persists until the most challenging sub-dataset is addressed. we conducted extensive experiments using two leading open-source pre-trained models, llama2 7/13/70b @xcite , llama3 8/70b, and mistral 7b @xcite , on two widelyused instruction datasets, alpaca and its refined version alpacaclean. to evaluate the training effectiveness, we utilized 6 benchmarks, with gpt-4 serving as the judge to measure the performance improvements of phased ift over one-off ift in terms of win rate. specially, figure 1 illustrates the process wherein phased ift's win rate progressively surpasses that of one-off ift on the three-stage difficulty-increasing sub-datasets. it is evident that the three llama2 models and the two llama3 models exhibit a consistent trend of increasing win rates with the progression of uptraining on multi-stage sub-datasets, indicating the effectiveness of the progressive alignment hypothesis. furthermore, we conducted three sets of experiments and an ablation study to confirm the effectiveness of phased ift. the main contributions of this paper include: open-source instruction dataset the alpaca dataset @xcite aims to improve instruction-following ability of llms, consisting of 52k instructions, generated using ope-nai's text-davincic-003. it has improved the selfinstruction framework @xcite , leading to a greater diversity in the alpaca dataset compared to the self-instruction dataset. alpacacleaned (yahma, 2023) is a quality-enhanced version of the original alpaca dataset, which addresses hallucinations, empty outputs, and incorrect answers, thereby improving the overall quality of the dataset. in addition, lima @xcite has also released a carefully curated dataset, including 1000 instructions, which encompasses 750 popular queries and responses from community forums such as stack exchange and wikihow, as well as 250 meticulously crafted entries. ## method figure 2 presents the overall pipeline of our proposed phased instruction fine-tuning (phased ift), encompassing three key components: instruction difficulty calculating (idc), instruction dataset segmenting (ids), and uptraining. where, idc prompts gpt-4 to allocate a difficulty rating to each instruction. following this, ids converts the instructions's score into a cumulative probability distribution, which facilitates the division of the dataset into subsets categorized by difficulty level, and segments the whole instruction data into several stages. finally, across this stages, uptraining is performed, beginning with those classified as less challenging. detailed elaborations on the specifics of each component are provided in the subsequent sections of this chapter. instruction difficulty calculating (idc) following alpagasus's prompt @xcite , we also design a gpt-4-based scoring prompt to assess the difficulty of each triplet of (instruction, input, output) within the 52k alpaca dataset, using a scale of 1 to 5, with higher scores indicating greater difficulty. this prompt is detailed in figure 3 , where each triplet is evaluated in two dimensions. firstly, we consider the intrinsic difficulty of the instruction itself, reflecting the complexity inherent to the task described. secondly, we evaluate the challenge involved in transforming the given instruction to the expected output, thereby estimating the difficulty from instruction to output. this dual-dimensional scoring approach allows for a nuanced understanding of task complexity within the dataset. the difficulty score histogram of the 52k alpaca dataset is illustrated as shown in the left of figure 2 . another interesting finding is that a majority of instructions in the 52k alpaca are categorized as low difficulty, with approximately 63.5% scoring below 1.5. in contrast, within the alpaca-clean dataset, instructions with a difficulty score lower than 1.5 account for 52.3%; in the lima dataset, the proportion of instructions with a difficulty score below 1.5 is merely 9.9%, while those scoring above 4.5 account for a significant 58.3%. details of their comparison can be found in appendix a.1and a.2. this highlights the feasibility of employing gpt-4 to assess the difficulty level of instructions. ## experiments this paper aims to validate the progressive alignment hypothesis by comparing the efficacy of two instruction training methodologies: one-off instruction fine-tuning (one-off ift) and our proposed phased instruction fine-tuning (phased ift), through three experiments and ablation studies. experiment 1 demonstrates the superior performance of phased ift over one-off ift across two instruction datasets, establishing its effectiveness. experiment 2 evaluates the effectiveness of stratifying instruction datasets by difficulty versus arbitrary segmentation, aiming to attribute phased ift's success to difficulty-stratified instruction data categorization rather than to the phased training approach. experiment 3 conducts a full permutation analysis of phased ift to demonstrate that its success is specifically due to uptraining in a sequence of increasing difficulty, with alternative sequencing failing to produce comparable outcomes. ## conclusion the paper introduces the progressive alignment hypothesis, and then proposes phased ift approach. utilizing gpt-4 to score the difficulty of instructions, the instruction dataset is categorized into different stages according to the difficulty. diverging from prior methods that apply a one-off ift on the whole instruction data, our approach entails staged, supervised uptraining on each stage in ascending order of difficulty. through extensive experiments, we have validated the progressive alignment hypothesis and the efficacy of phased ift, demonstrating its utility in developing diverse instruction dataset and enhancing fine-tuning llms.