Automatic Instruction Evolving for Large Language Models (2024)

Abstract: Fine-tuning large pre-trained language models with Evol-Instruct has achieved encouraging results across a wide range of tasks. However, designing effective evolving methods for instruction evolution requires substantial human expertise. This paper proposes Auto Evol-Instruct, an end-to-end framework that evolves instruction datasets using large language models without any human effort. The framework automatically analyzes and summarizes suitable evolutionary strategies for the given instruction data and iteratively improves the evolving method based on issues exposed during the instruction evolution process. Our extensive experiments demonstrate that the best method optimized by Auto Evol-Instruct outperforms human-designed methods on various benchmarks, including MT-Bench, AlpacaEval, GSM8K, and HumanEval.

## introduction fine-tuning large language models (llms) to follow detailed instructions is vital to unlocking their power @xcite . high-quality datasets, such as sharegpt @xcite , openassistant @xcite , lima @xcite , have greatly improved the performance of instruction-tuning, promoting the prosperity of llm alignment. however, annotating instruction following datasets with such quality is hard to scale, and its quality upper limit is also uncontrollable. researchers @xcite @xcite are actively exploring ways to break through the quality upper-bound of existing datasets. evol-instruct @xcite takes the high-quality data as a starting point, and further iteratively refines it using llms, improving its complexity and diversity. it has demonstrated superior performance across a broad range of public benchmarks that evaluate diverse capabilities, including instruction following @xcite , code generation @xcite , and mathematical reasoning @xcite . while evol-instruct exhibits outstanding performance, its heavy reliance on heuristic efforts presents notable challenges. whenever it is used for a completely new task, the methods for execution evolution need to be redesigned. such a process requires a high level of expertise and considerable costs, hindering its adaptation to a wider spectrum of capabilities. to address these challenges, it needs to automate the evol-instruct process, which will encounter the following difficulties: (1) design evolving methods automatically that make the instructions more complex for a given task (2) to keep the instruction evolution process working properly, the evolving method needs to avoid evolution failure. in this paper, we propose auto evol-instruct, an effective approach to utilizing llms in designing methods for executing instruction evolution. auto evol-instruct automatically designs evolving methods that make given instruction data more complex, enabling almost cost-free adaptation to different tasks by only changing the input data of the framework. firstly, to transition from manually-designed evolving rules to automated ones, we begin with a universal initial evolving method. our initial evolving method is different from the method of evol instruct, which requires human experts to specify the rules of evolution. instead, it can autonomously analyze the input instruction and brainstorm evolution rules suitable for given data. due to the diversity and complexity of varied instruction datasets, a fixed evolving method can not guarantee the stability and effectiveness of all data evolution. therefore, we leverage llm as the optimizer to optimize the initial evolving method iteratively to ensure the lowest failure rate for a given instruction dataset. we refer to the model used for evolution as the evol llm, and the model used for optimization as the to f m t and e 1 t to e m t respectively. the optimizer llm carefully analyzes the potential issues and failures exposed in instruction evolution performed by evol llm, generating feedback for subsequent optimization. (2) evolving method optimization: the optimizer llm optimizes the evolving method by addressing these identified issues in feedback. these stages alternate and repeat to progressively develop an effective evolving method using only a subset of the instruction data. once the optimal evolving method is identified, it directs the evol llm to convert the entire instruction dataset into more diverse and complex forms, thus facilitating improved instruction tuning. our experiments show that the evolving methods designed by auto evol-instruct outperform the evol-instruct methods @xcite designed by human experts in instruction tuning across various capabilities, including instruction following, mathematical reasoning, and code generation. using only 10k evolved sharegpt for fine-tuning mixtral-8x7b @xcite , we achieve 8.09 on mt-bench @xcite and 91.4 on alpacaeval @xcite , surpassing gpt-3.5-turbo and wizardlm-70b, and comparable with claude-2.0. using only 7k evolved gsm8k training data for fine-tuning mixtral-8x7b, we achieve 82.49 on gsm8k, surpassing gpt-3.5-turbo, wizardmath-70b and metamath-70b @xcite . using 20k evolved code alpaca to fine-tune deepseek-coder-base-33b @xcite , we achieve 77.4 on humaneval, surpassing gpt-3.5-turbo and wizardcoder-34b. ## auto evol-instruct unlike evol-instruct, auto evol-instruct is a fully automated framework that improves the complexity and quality of instruction data without any human intervention. its key advancements include: (1) automatically designing evolving methods for instruction evolution, facilitating adaptation to a wide range of tasks and enhancing model capabilities across a broader spectrum; (2) developing evolving methods that surpass those crafted by human experts, while minimizing failures and ensuring successful execution of instruction evolution. figure 1 illustrates the process of automating the design of evolving methods in the auto evol-instruct framework (section 3.1-3.3). we also detail specific examples of how the evolving method changes at each step in the ## experiment in this section, we conduct a detailed study on the effects of auto evol-instruct. we begin with an overview of the experimental setup, then test our method's effectiveness in instruction following, math reasoning, and code generation. we also present auto evol-instruct's performance results on the open llm leaderboard in the appendix d. ## related work instruction tuning emerges as a pivotal strategy for unlocking the potential of llms @xcite . by curating highquality datasets, we can more efficiently align these models with desired direction @xcite . the challenge of scaling high-quality instruction data remains a central research interest. some researchers prioritize human annotation for creating instruction data, such as sharegpt @xcite and openassistant @xcite . other researchers explore more efficient ways to break through the quality upper-bound of existing datasets @xcite @xcite . @xcite introduces evol-instruct, a methodology that iteratively refines instruction-following data to produce datasets that are both more complex and diverse. @xcite develop evolving methods tailored to the nuances of code data based on evol-instruct. distinct from these methodologies, our approach introduces a fully automated framework for developing evolving methods. this innovation is not only scalable but also versatile, extending its utility across a broad spectrum of capabilities. llms like gpt-4 and palm are capable of optimizing their output through internal or external feedback mechanisms @xcite @xcite . we use this capabilities to address identified issues in the evolving method and adapt to the characteristics of the instruction data. ## conclusion this paper introduces auto evol-instruct, an innovative approach that successfully automates the evolution of instruction datasets for llms, eliminating the need for human intervention. our method centers on the automatic analysis and summarization of appropriate evolutionary strategies for the given instruction data. it iteratively refines evolving methods by addressing the issues identified during the instruction evolution process. the experiments conducted have shown that methods optimized by auto evol-instruct, significantly surpass those crafted by humans across various benchmarks, including mt-bench, alpacaeval, gsm8k and humaneval.