LLM s-as-Instructors: Learning from Errors Toward Automating Model Improvement (2024)

Abstract: This paper introduces the innovative “LLMs-as-Instructors” framework, which leverages the advanced Large Language Models (LLMs) to autonomously enhance the training of smaller target models. Inspired by the theory of “Learning from Errors”, this framework employs an instructor LLM to meticulously analyze the specific errors within a target model, facilitating targeted and efficient training cycles. Within this framework, we implement two strategies: “Learning from Error,” which focuses solely on incorrect responses to tailor training data, and “Learning from Error by Contrast,” which uses contrastive learning to analyze both correct and incorrect responses for a deeper understanding of errors. Our empirical studies, conducted with several open-source models, demonstrate significant improvements across multiple benchmarks, including mathematical reasoning, coding abilities, and factual knowledge. Notably, the refined Llama-3-8b-Instruction has outperformed ChatGPT, illustrating the effectiveness of our approach. By leveraging the strengths of both strategies, we have attained a more balanced performance improvement on both in-domain and out-of-domain benchmarks.

## introduction llms have achieved great success in various tasks, while it is still time-consuming and labor-intensive for further training, e.g., adapting to specific domains, following instructions, or aligning with human preference. to ensure continuous improvement during training, developers usually keep analyzing the model's responses and modifying/supplementing the corpus accordingly. some advanced ## llms-as-instructors framework the framework depicted in figure 2 illustrates our llms-as-instructors approach, which is characterized by undergoing a series of n iterations across four integral stages: 1) data selection (section 2.2) is the initial step where we meticulously select target data samples designed to evaluate the capabilities we intend to enhance in subsequent iterations. 2) result collection (section 2.2) follows, where we scrutinize the performance of the target model and meticulously gather its responses for in-depth subsequent analysis. 3) instructor analysis and data supply (section 2.3) is the phase where the instructor, employing a predefined analytical strategy, dissects the errors made by the target model. this analysis is pivotal in formulating strategies for improvement and crafting tailored training corpora to facilitate the target model's learning process. 4) target model training and evaluation (section 2.4) concludes the cycle, where the target model undergoes fine-tuning with the supplemented training corpus. it is then evaluated to measure the extent of improvement using a dedicated evaluation dataset. upon receiving a target llm m 0 target , we engage a stronger model as the "instructor" (m instructor ), utilizing responses from the target model to analyze the errors, and then generating customized training materials to guide the model to learn from the errors for performance increase. the process of this method is outlined in algorithm 1. ## discussion given the proven effectiveness of our llms-as-instructors method, we further delve deeper to ascertain the various contributing factors to its success. specifically, we have examined the impact of the training set size (section 4.1), and the num- the results concerning the llama-3-8b-instruction model are detailed in appendix b. ## generated training sample analysis results indicate that the model achieves substantial improvements on both in-domain and out-ofdomain datasets, thereby affirming the generalizability of our method. in this section, we conduct a quantitative analysis of the characteristics of the training data generated to further demonstrate that these performance enhancements are not merely due to similarities between the training and evaluation data. for each generated training sample, we calculate the closest match within the evaluation dataset d eval using rouge-l @xcite score. the results, presented in ## related work instructor model guided model improvement: enhancing a target model is a widely adopted methodology that leverages the capabilities of llms to create innovative training datasets @xcite like: wizardlm @xcite , orcamath @xcite , tülu @xcite and etc. @xcite @xcite @xcite . concurrent with this, another study @xcite employs an "instructor" model to facilitate learning from the target model's errors. however, this method trains individual models for each dataset, thereby overlooking the potential advantages of integrating correct samples into the training. self-improving llms: recent research has seen multiple efforts to explore self-improvement in language models. some studies, such as @xcite , focus on training the target model with incorrect samples, while @xcite utilize the base model to generate context for performance improvement. other works @xcite @xcite @xcite @xcite ) employ the target model itself to generate training samples. while these studies predominantly concentrate on autonomous iterations without external influences, our approach diverges by incorporating feedback mechanisms. these mechanisms not only capitalize on the target models' errors but also introduce direct intervention through strategic analysis. such an integration mainly aims to refine the model's learning process, thereby fostering a more sophisticated and dynamic model evolution. ## conclusion in this work, we introduced the innovative "llmsas-instructors" framework, leveraging advanced large language models (llms) to guide smaller target models in learning from their errors, inspired by human learning theory. our experiments demonstrated significant improvements for mistral-7b-intruct and llama-3-8b-intruction across seven benchmarks, underscoring the generalizability and efficacy of our approach. notably, the improved llama-3-8b-intruction model surpassed chatgpt's performance, highlighting the effectiveness of our method. our two strategies each have distinct characteristics, allowing them to be used in combination depending on the specific data scenario. in the future, we aim to focus on exploring more model analysis strategies, thereby enhancing the adaptability and effectiveness of our llmsas-instructors framework and integrating explicit and fine-grained error analysis into the training stage, potentially through methods like "meaningful learning" @xcite .