Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction (2024)

Abstract: The rapid advancement of Large Language Models (LLMs) in the realm of mathematical reasoning necessitates comprehensive evaluations to gauge progress and inspire future directions. Existing assessments predominantly focus on problem-solving from the examinee perspective, overlooking a dual perspective of examiner regarding error identification and correction.From the examiner perspective, we define four evaluation tasks for error identification and correction along with a new dataset with annotated error types and steps. We also design diverse prompts to thoroughly evaluate eleven representative LLMs. Our principal findings indicate that GPT-4 outperforms all models, while open-source model LLaMA-2-7B demonstrates comparable abilities to closed-source models GPT-3.5 and Gemini Pro.Notably, calculation error proves the most challenging error type. Moreover, prompting LLMs with the error types can improve the average correction accuracy by 47.9%. These results reveal potential directions for developing the mathematical reasoning abilities of LLMs.Our code and dataset is available on https://github.com/LittleCirc1e/EIC.

## introduction large language models @xcite @xcite have been successfully applied to mathematical reasoning, particularly in the math word problems (mwp) @xcite . llms cultivate a nuanced understanding of number-intensive context and multi-step reasoning. cutting-edge models such as @xcite have demonstrated impressive performance in addressing mathematical problems. for example, it has achieved an accuracy of 97% on the gsm8k dataset @xcite . with the rapid advancement of llms, evaluating their effectiveness and reliability becomes increasingly crucial. existing evaluations are mainly from the examinee perspective, which directly assess the problemsolving capability of llms regarding the correctness of answers @xcite @xcite and the consistency of intermediate reasoning steps @xcite @xcite . however, current research rarely delve into a dual perspective of examiner, i.e., the ability to identify and correct errors (figure 1 ), which is equally crucial as problemsolving and worthwhile exploring. on one hand, the performance of traditional evaluation tasks is almost approaching saturation, calling an urgent need for new perspectives of evaluation. on the other hand, accurate error recognition and correction can facilitate the development of problem-solving capability. aiming to construct fine-grained evaluation on error recognition and correction, we define four distinct tasks. these tasks are as follows: 1) error-presence identification (ep): identifying whether any error exists in the entire solution. 2) error-step identification (es): identifying the first wrong step within the solution, which is the root cause of error. 3) error-type identification (et): identifying the error type present in the first wrong step, such as calculation error. 4) error correction (ec): rectifying the wrong steps and obtaining the final corrected answer. to our knowledge, we are the first to comprehensively define the four evaluation tasks for error identification and correction regarding mathematical reasoning. moving one step further, we consider constructing the evaluation dataset for these four tasks. given a question, the dataset should include ground-truth answers, solutions with errors, step numbers of wrong steps, and types of errors. to construct this dataset, we need to define the types of error first. by collating examples from existing studies and practical instances, we distill nine common error types. subsequently, harnessing the exceptional text generation capability of @xcite , we transform initially correct solutions of @xcite and mathqa @xcite into solutions featuring singlestep and single-type errors. through this approach, we establish a dataset comprising 1800 instances to evaluate the ability to recognize and rectify errors. based on the evaluation dataset, we test closedsource models, including gpt-3.5 @xcite , @xcite , @xcite , gemini pro @xcite , and their open-source counterparts such as llama-2-7b, llama-2-13b @xcite , metamath-7b, metamath-13b @xcite , mistral-7b @xcite , llemma-7b @xcite and lema-7b @xcite . we devise diverse prompts of each task to evaluate the robustness of these llms. through extensive experiments, we derive five key findings: 1) across all four tasks, gpt-4 exhibits outstanding performance compared to other models with glm-4 closely following. gpt-3.5, gemini pro, and llama-2-7b show varying strengths and weaknesses. 2) while gpt-4 and glm-4 demonstrate overall competence across four tasks, their ability to identify and rectify calculation error lags behind other error types. this suggests a need for further enhancement of the calculation capability of llms. 3) in the task of et, many error types are easily recognized as the type of calculation error, with the type of missing step proving to be the most challenging to identify. 4) in the task of ec and es, by providing the error types, the average accuracy can be improved by 47.9% and 45.9%, respectively. 5) open-source models are highly influenced by prompts, while closed-source models demonstrate a comparatively robust performance. our contributions can be summarized as follows: 1) we define four tasks for evaluating the mathematical reasoning ability of llms regarding error identification and correction. to our knowledge, our work represents the first comprehensive assessment of the fine-grained capability of llms in recognizing and rectifying errors. 2) we define nine common error types and provide a dataset based on these error types. the dataset is intended to facilitate a more nuanced examination of the llms' performance in handling different error scenarios. ## task formulation as the proverb goes, errors are the stepping stones to wisdom. if a model is adept at mathematical reasoning, it should excel at identifying and correcting errors. hence, we gauge the mathematical reasoning abilities of llms by assessing their proficiency in recognizing and rectifying errors. to comprehensively accomplish the evaluation, as shown in figure 2 , we define four tasks at a fine-grained level of error identification and correction. figure 2 : illustration of dataset construction and the four evaluation tasks. for dataset construction, we use gpt-4 to convert ground-truth solutions into wrong solutions containing specific error types. the four evaluation tasks comprehensively access llms' error identification and correction abilities from diverse perspectives. here, c is selected from the pre-defined error types, such as calculation error. we define error types in the prompts and design zero-shot and few-shot prompts, where the few-shot prompt provides an example for each error type. figure 29 and 30 showcase the zero-shot prompts for open-source and closed-source models. considering that the order of error types might affect the accuracy of identifying error types, we design prompts that reverses the original order of error types and randomly shuffles them. we compute acc 1 and the accuracy in identifying error type acc 3 = 1 n n @xmath0, where ĉ is the error type of the first wrong step identified by llms. for task 2 and task 4, we propose to leverage the error type information in the prompts to hint llms for error step identification and error correction. accordingly, we design the zero-shot and few-shot prompts with error type information as shown in figure 27 , 28, 33 and 34. ## dataset construction a significant challenge in achieving the four evaluation tasks is lacking compatible datasets with fine-grained error annotation. therefore, we opt to construct a dataset that meets the requirements of ## experiment we conduct extensive experiments to address the following research questions: -rq1: how do different llms perform on the four tasks on error identification and correction? -rq2: how difficult are identifying and correcting different error types? -rq3: how robust are llms to different prompts w.r.t. the four evaluation tasks? experiment setup. we select typical commercial closed-source llms, gpt-3.5, gpt-4, glm-4, gemini pro, along with the general-purpose opensource llama-2 series, and the state-of-the-art mathematical metamath series in their 7b and 13b versions for evaluation. besides, we also evaluate other three cutting-edge mathematical models: mistral, llemma and lema in their 7b versions. 2 to minimize randomness, we set the temperature to 0. for ease of statistical analysis, we prompt closedsource llms to output in json format. however, open-source models do not consistently adhere to the format requirement, so we use a relaxed format for their prompts. ## conclusion we systematically delineated four evaluation tasks aimed at identifying and rectifying errors, marking the first comprehensive attempt in this domain. to facilitate the evaluation process, we curated a dataset categorized by error types. furthermore, we conducted thorough experiments across various closed-source and open-source models, yielding significant insights that bolster the mathematical reasoning capabilities of llms. in future research, we will explore more avenues such as rectifying single-step and single-type errors, single-step multitype errors on various llms, and increasing the continuity of our correction prompts which can rectify errors based on incorrect preceding steps.