Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning (2024)

Abstract: Instruction tuning aims to align large language models (LLMs) with open-domain instructions and human-preferred responses. While several studies have explored autonomous approaches to distilling and annotating instructions from powerful proprietary LLMs, such as ChatGPT, they often neglect the impact of the distributions and characteristics of tasks, together with the varying difficulty of instructions in training sets. This oversight can lead to imbalanced knowledge capabilities and poor generalization powers of student LLMs. To address these challenges, we introduce Task-Aware Curriculum Planning for Instruction Refinement (TAPIR), a multi-round distillation framework that utilizes an oracle LLM to select instructions that are difficult for a student LLM to follow. To balance the student’s capabilities, task distributions in training sets are adjusted with responses automatically refined according to their corresponding tasks. In addition, by incorporating curriculum planning, our approach systematically escalates the difficulty levels of tasks, progressively enhancing the student LLM’s capabilities. We rigorously evaluate TAPIR using several widely recognized benchmarks (such as AlpacaEval 2.0, MT-Bench, etc.) and multiple student LLMs. Empirical results demonstrate that student LLMs, trained with our method and less training data, outperform larger instruction-tuned models and strong distillation baselines.

## introduction large language models (llms) have demonstrated impressive abilities in generalizing to previously unseen tasks @xcite @xcite @xcite . instruction tuning has emerged as a key technique for aligning pre-trained llms with user preferences, achieved by supervised fine-tuning (sft) on datasets annotated with instructional prompts @xcite @xcite . distinct from conventional task-specific fine-tuning, it leverages the broad knowledge that llms accumulate during pre-training, often involving a wide range of tasks. with the availability of apis for powerful proprietary llms, such as chatgpt, various approaches have been proposed to distill these black-box llms into smaller counterparts. these methods involve automatic generation of instructional prompts and their corresponding outputs @xcite @xcite . empirical studies have illustrated that enhancing the diversity and complexity of instruction tuning datasets can improve the model performance @xcite . quality outweighs quantity; thus fine-tuning over a carefully calibrated, smaller dataset may outperform instructtuned models trained on larger datasets @xcite @xcite . despite the advances, the optimal complexity of instructional data for models with varying capacities and parameters remains an open question. prior efforts have sought to maximize data diversity through the utilization of sentence embeddings @xcite . yet, this approach has not fully resolved the issue of imbalanced model capabilities, as the maximum diversity of sentence embeddings does not necessarily lead to the optimal task ratio. we observe that models fine-tuned with these methods sometimes struggle with more complex and challenging tasks, such as logical reasoning. @xcite also point out that each ability of llms has its own growth pace. to address the above challenges, we propose task-aware curriculum planning for instruction refinement (tapir), foot_0 a novel llm distillation framework that fosters balanced task capacities and incorporates dynamic adjustment of task difficulty through curriculum learning principles. tapir harnesses the strengths of an oracle llm (typically a proprietary model) to identify and expand instructions that pose challenges to a student llm, assessed by a judge llm. the essence of tapir lies in its strategic approach to instruction filtering, task re-balancing and response refinement, ensuring that the range of tasks and their corresponding instructional data is comprehensive and representative. by systematically adjusting task difficulty, tapir further enables a progressive and structured learning path in multiple rounds, akin to a curriculum, that encourages student llms to gradually achieve easy-to-hard generalizations. it addresses the critical issue of instructional imbalance that has plagued previous attempts at autonomous distillation @xcite @xcite . in the experiments, we obtain multiple student llms of varied sizes distilled with the tapir framework. the results show that trained llms surpass larger instruction-tuned models and strong distillation baselines on widely used benchmarks such as alpacaeval 2.0 @xcite and @xcite , as shown in figure 1 . we need to further emphasize that tapir is a versatile training pipeline that may continue to benefit from stronger teacher llms and more taskspecific synthesis techniques in future research. in summary, we make the following contributions: in this section, we summarize the related work in the three aspects: instruction tuning, knowledge distillation using llms and llm as a judge. ## conclusion the tapir framework introduces a strategic approach to distill large powerful llms with instruction tuning by addressing task distribution and instruction hardness. the framework's effective curriculum planning technique has been shown to enhance the performance of student llms, enabling them to outperform larger models with fewer training data, especially in complex tasks. the empirical validation provided by benchmarks such as al-pacaeval 2.0 suggests that incorporating balanced task distributions and calibrated difficulty is crucial for advancing the capabilities of llms.