Can LLM s Learn from Previous Mistakes? Investigating LLM s’ Errors to Boost for Reasoning (2024)

Abstract: Large language models (LLMs) have demonstrated striking reasoning capability. Recent works have shown the benefits to LLMs from fine-tuning golden-standard Chain-of-Thought (CoT) rationales or using them as correct examples in few-shot prompting. While humans can indeed imitate correct examples, learning from our mistakes is another vital aspect of human cognition. Hence, a question naturally arises: can LLMs learn and benefit from their mistakes, especially for their reasoning? This study investigates this problem from both the prompting and model-tuning perspectives. We begin by introducing CoTErrorSet, a new benchmark with 609,432 questions, each designed with both correct and error references, and demonstrating the types and reasons for making such mistakes. To explore the effectiveness of those mistakes, we design two methods: (1) Self-rethinking prompting guides LLMs to rethink whether they have made similar previous mistakes; and (2) Mistake tuning involves finetuning models in both correct and incorrect reasoning domains, rather than only tuning models to learn ground truth in traditional methodology. We conduct a series of experiments to prove LLMs can obtain benefits from mistakes in both directions. Our two methods offer potentially cost-effective strategies by leveraging errors to enhance reasoning capabilities, which costs significantly less than creating meticulously hand-crafted golden references. We ultimately make a thorough analysis of the reasons behind LLMs’ errors, which provides directions that future research needs to overcome. CoTErrorSet will be published soon on https://github.com/YookiTong/Learn-from-Mistakes-CotErrorSet .

## introduction large language models (llms) @xcite @xcite ; tou- * †corresponding author. figure 1 : the overview pipeline of our work includes (1). mistake collection and analysis (section 3). (2) two novel methods to instruct llms to learn from mistakes(section 4 and section 5). @xcite have demonstrated strong capabilities across various tasks and applications @xcite . to further unleash the reasoning abilities of llms and align their thinking process with humans, many recent studies explored chain-of-thought (cot)-based prompting @xcite @xcite @xcite to instruct llms to solve the given problem with human-like logic. besides logical step-by-step thinking, another critical learning pattern of us humans is to rethink and learn from our previous mistakes so that avoid repeating the same mistakes in the future @xcite . however, few studies have focused on systematically understanding what kinds of intermediate errors occur in making cot procedures and whether llms can learn from those mistakes. to address these issues, we aim to explore the potential of llms to effectively utilize their previous mistakes to boost reasoning. to enhance the scalability and efficiency of analyzing and learning from the mistakes of llms, we began by collecting a vast dataset of llms' reasoning outputs and built coterrorset, which consists of 609,432 questions sourced from 1060 tasks across diverse domains. each query in this set is meticulously structured, featuring both a manually curated correct reference and the incorrect rationales collected from palm2 @xcite )'s responses. furthermore, we prompt the llms with the correct reference and the incorrect responses in order to make it reflect why making such mistakes. the introspective responses are also collected and subsequently utilized in our work. we employ this data for cluster analysis to identify specific details of the errors. with our coterrorset, we introduce two innovative paradigms, namely mistake tuning and self-rethinking, aimed at efficiently augmenting llms by leveraging their historical errors during both tuning and inference stages. diverging from the conventional approach of only relying on correct rationales in traditional supervised fine-tuning, our mistake tuning strategy incorporates combinations of both correct references and incorrect rationales. to facilitate the learning process for llms, we introduce the prefixes [correct ra-tionale] and [incorrect rationale] before the corresponding rationales. intuitively, this prompt tuning facilitates llms to distinguish between correct and incorrect rationales while avoiding corruption from the incorrect ones with the two separated prefixes. for self-rethinking, inspired by contrastive in-context learning @xcite , we expose llms to both correct and incorrect rationales in demonstration samples. after obtaining the initial answer output by the llm, we iteratively prompt it to rethink and rectify the result based on the historical mistakes. to manage computational resources and prevent potential loops, we implement a threshold, limiting the number of times the model can engage in self-rethinking and corrections. figure 1 gives an overview pipeline of our work. to substantiate the efficacy of our proposed methodologies and to delve into the learning capabilities of llms from their mistakes, we undertake experiments encompassing diverse reasoning tasks and llms of varying sizes. the application of our methods consistently yields performance enhancements across a spectrum of tasks, underscoring the effectiveness and broad applicability of our approaches in leveraging llms' mistakes during both the tuning and inference stages. additionally, we conduct thorough analyses of the error types exhibited by llms, offering comprehensive insights and guidance on mitigating the most prevalent errors in these models. in general, our contributions are as follows: ## related work human-like reasoning with llms. cot @xcite demonstrate the great potential of equipping llms with human-like reasoning capability. following them, various logical and structural reasoning strategies @xcite @xcite @xcite are proposed to align llms' thinking processes with humans. these enhanced reasoning approaches have been adopted in different tasks and areas, including commonsense reasoning @xcite , logical reasoning @xcite and mathematical reasoning @xcite and achieved promising performance. in this work, we aim to investigate whether llms can benefit from rethinking and learning from previous mistakes, which is one of the most important learning patterns of humans. ## our methodology: self-rethinking self-rethinking offers an innovative approach to encourage llms to consider if they are repeating past errors. this method starts with an initial cot reasoning. following this, the model uses the provided reasoning outputs and a random selection of examples from coterrorset. this step is designed to assess if the model's most recent response includes similar inaccuracies. if errors are detected, it will formulate a new rationale and undergo the evaluation process again. this cycle continues until the model deems its latest answer to be correct or it reaches a set limit of evaluation rounds. the main goal is to empower the llm to learn from its errors introspectively and minimize the recurrence of such mistakes. one example is shown in the core of self-rethinking lies in the backwardchecking stage. in this phase, the llm reviews its reasoning chain, but with a specific focus on the error types it previously identified. this explicit demonstration of errors, coupled with the question, golden reference, and incorrect rationales, is instrumental in enabling the llm to recognize specific types of mistakes it tends to make. this targeted review helps the llm to not just correct the random errors but to consciously avoid repeating the same types of mistakes it has made in the past. the process includes a loop for error correction and confirmation. if the llm finds that it has repeated any of the previously identified mistakes, it revisits the reasoning process to correct them. otherwise, the last response is adopted as the final result. moreover, the iterative checking process should have a crucial repeating boundary, denoted as k iterations. if the llm's error-checking and correction cycles surpass this predefined threshold and errors still persist, the process concludes under the assumption that the issue at hand or the error detection might exceed the llm's current capabilities. this constraint prevents the llm from being caught in an endless loop of self-rethinking, ensuring the efficiency and practicality of the reasoning process. ## our methodology: mistake tuning in order to fully investigate the other potential utilization of our principles, we introduce mistake tuning, which demonstrates our motivation is a broad and pioneering framework not only in the field of in-context learning. this approach is designed to finetune llms on the combinations of both correct rationales and incorrect mistakes. by simply appending prefixes [correct rationale] and [incorrect rationale] before corresponding rationales, mistake tuning can further improve llms' abilities to distinguish between correct and incorrect rationale. mistake tuning is built upon the foundational motivations and conclusions of self-rethinking, where llms can learn from the implicit reasons and types of mistakes they made in cot reasoning. this process can be formulated as: where q, s and r represent the given question, special prefix and corresponding rationale respectively. ⊕ represents the operation of concatenation. mistake tuning presents a cost-effective, straightforward, and efficient alternative. previous work has proven pretraining with some controlled signals based on human feedback can result in llms' better ability to generate more satisfactory contents @xcite . hence, incorporating fixed prefixes in finetuning llms in the field of reasoning can also help models differentiate information from golden references and mistakes. our results also demonstrate its effectiveness for promoting llms' reasoning abilities without additional costs similar to annotating golden reasoning references. ## experiments in this section, we conducted a series of experiments to compare the proposed self-rethinking methods with the existing approach on both arithmetic and commonsense reasoning benchmarks. ## unveiling llm's reasoning errors in this section, we delve into the detailed types and underlying reasons that lead to mistakes in llms's inference process. we sample mistake examples from gsm8k and logiqa to conduct an in-depth analysis of both arithmetic and commonsense reasoning. we put some examples in appendix b. for commonsense reasoning, we find errors like the misinterpretation of facts or concepts usually arise due to the model's limitations in understanding and applying context accurately. this reveals current llms may still fall short of consistently recalling precise factual knowledge within a given context. consequently, this underscores the imperative to advance toward the development of retrieval-augmented generation(rag) systems @xcite , as they hold the promise of yielding more faithful and contextually aligned results. additionally, errors stemming from logical fallacies or incorrect inferences reveal llms' reliance on pattern recognition over logical reasoning, sometimes leading them to make logically inconsistent or unsupported connections by the given facts. as shown in figure 5 , the most errors made by llms in arithmetic reasoning are about calculation. this can be attributed to the different nature of llms compared to other tools like calculators. to address this issue, @xcite 's suggestion using program-of-thought (pot) is a promising approach to instruct llms to generate a segment of code to solve the given problem, resulting in more accurate calculation results. furthermore, it's important to note that logical error is also a type of error that llms always suffer from. compared with calculation errors and numeric errors, the causes of logical errors are more complicated and nuanced. for instance, errors like misinterpreting given data or misapplying arithmetic operations reveal a lack of depth in understanding mathematical relationships. this can result from the model's limitations in comprehending the nuances of mathematical concepts or its inability to correctly infer the needed function from the context of the question. in the future, more fine-grained analysis and methods are needed to address such complex logical errors in arithmetic reasoning. ## conclusions and future work in this work, we explore whether llms can learn from their mistakes. in order to investigate llms' abilities to differentiate and learn from mistakes, we introduce coterrorset, a novel benchmark collecting both correct and incorrect cot rationales across various domains and designed with demonstrations for making errors. we propose two possible solutions to expose the effects of mistakes from different perspectives: self-rethinking and mistake tuning. both of them have achieved consistent and significant improvements, which demonstrates the potential benefits of learning from reasoning errors. in the last, we conduct a comprehensive and detailed analysis of llms' common mistakes in both arithmetic and commonsense reasoning. the findings will provide a clear direction for future improvements. for future work, we envision proposing corresponding algorithms or loss functions to learn implicit information from mistakes. the primary intent of this work is to provide a new paradigm so there are still a lot of improvements that can be down following this work. for example, incorporating contrastive learning to differentiate correct references and errors is intuitive to make more improvements. also, some memorization and retrieval-augmented skills can help models benefit from mistakes similar to each question.