LLM s cannot find reasoning errors, but can correct them given the error location (2024)

Abstract: While self-correction has shown promise in improving LLM outputs in terms of style and quality (e.g. Chen et al., 2023b; Madaan et al.,2023), recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in worse performances overall (Huang et al., 2023). In this paper, we show that poor self-correction performance stems from LLMs’ inability tofind logical mistakes, rather than their ability to correct a known mistake. Firstly, we benchmark several state-of-the-art LLMs ontheir mistake-finding ability and demonstrate that they generally struggle with the task, even in highly objective, unambiguous cases. Secondly, we test the correction abilities of LLMs – separately from mistake finding – using a backtracking setup that feeds ground truth mistake location information to the model. We show that this boosts downstream task performance across our 5 reasoning tasks, indicating that LLMs’ correction abilities are robust. Finally, we show that it is possible to obtain mistake location information without ground truth labels or in-domain training data. We train a small classifier with out-of-domain data, which exhibits stronger mistake-finding performance than prompting a large model. We release our dataset of LLM-generated logical mistakes, BIG-Bench Mistake, to enable further research into locating LLM reasoning mistakes.

## introduction large language models (llms) have dominated the field of nlp in recent years, achieving stateof-the-art performance in a large variety of applications. in particular, llms have demonstrated the ability to solve tasks with zero-or few-shot prompting, giving rise to prompting methods such as chain-of-thought (cot) @xcite , self-consistency (sc) @xcite , react @xcite , etc. recent literature on few-or zero-shot prompting has focused on the concept of self-correction, i.e. having an llm correct its own outputs @xcite @xcite @xcite . @xcite for a review of the literature.) however, @xcite note that while self-correction may prove effective for improving model outputs in terms of style and quality, when it comes to reasoning tasks, llms struggle to identify and fix errors without external feedback: for example, reflexion @xcite and rci @xcite both use ground truth correctness as a signal to halt the self-correction loop. initially observed by @xcite on a math dataset, @xcite further demonstrate this shortcoming of self-correction in 2 additional datasets. while previous work typically present selfcorrection as a single process, we divide it into mistake finding and output correction to better understand each component individually. mistake finding is a fundamental reasoning skill that has been studied and utilised extensively in philosophy, psychology, and mathematics, spawning concepts such as critical thinking, and logical and mathematical fallacies. one might expect that the ability to find mistakes should also be an important requirement for llms. however, our results show that state-of-the-art llms currently cannot find mistakes reliably. output correction involves partially or completely changing previously generated outputs. with self-correction, this is typically done with outputs generated by the same model (see @xcite ). despite llms' inability to find mistakes, our results show that they can correct outputs, if given information about the mistake location. while llms struggle with mistake-finding in few-shot conditions, we can obtain more reliable mistake location information using a small, trained classifier. our contributions for this paper are as follows: 1. with chain-of-thought prompting, any task can be turned into a mistake-finding task. we collect and release foot_0 to the research community big-bench mistake, a dataset of cot-style traces foot_1 generated using palm 2 @xcite , and annotated according to where the first logical mistake is. to our knowledge, big-bench mistake is the first dataset of its kind that goes beyond problems in mathematics. ## big-bench mistake big-bench mistake contains 2186 sets of cotstyle traces. each trace is generated with palm 2 unicorn @xcite , and annotated with the location of the first logical error. sort the following words alphabetically: list: hypochlorite ponderosa phone credulity thought 1: i should start by looking at the first letter of the words in the list. the first letter: "hypochlorite": "h" (8). "ponderosa": "p" (16). "phone": "p" (16). "credulity": "c" (3). ## can llms find reasoning mistakes in cot traces? all models are given the same 3-shot prompts 5 . we use three different prompting methods: as with direct step-level prompting, the final answer is inferred from where the first "no" output occurs (subsequent steps are ignored). ## conclusion in this paper, we investigate llms' ability to find mistakes and correct outputs. we find that llms generally struggle to find mistakes, but, when given mistake location information, are able to correct outputs to boost performance. we therefore hypothesise that mistake finding is an important bottleneck preventing self-corrections strategies from performing well on reasoning tasks. we show initial evidence that a dedicated classifier for mistake finding can overcome this bottleneck. we train a small, baseline classifier on outof-domain data and demonstrate improvement over few-shot prompting results. while the classifier does not reach the threshold required for effective backtracking, our results show that it is possible to improve on mistake-finding accuracy using standard machine learning techniques. we leave the development of more sophisticated methods to future work, and release our dataset big-bench mistake to encourage this direction of research.