Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework (2025)

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains. Math Word Problems (MWPs) serve as a crucial benchmark for evaluating LLMs’ reasoning abilities. While most research primarily focuses on improving accuracy, it often neglects understanding and addressing the underlying patterns of errors. Current error classification methods rely on static and predefined categories, which limit their ability to capture the full spectrum of error patterns in mathematical reasoning. To enable systematic error analysis, we collect error samples from 15 different LLMs of varying sizes across four distinct MWP datasets using multiple sampling strategies. Based on this extensive collection, we introduce MWPES-300K, a comprehensive dataset containing 304,865 error samples that cover diverse error patterns and reasoning paths. To reduce human bias and enable fine-grained analysis of error patterns, we propose a novel framework for automated dynamic error classification in mathematical reasoning. Experimental results demonstrate that dataset characteristics significantly shape error patterns, which evolve from basic to complex manifestations as model capabilities increase. With deeper insights into error patterns, we propose Error-Aware Prompting (EAP) that incorporates common error patterns as explicit guidance, leading to significant improvements in mathematical reasoning performance.

## introduction large language models (llms) have showcased remarkable capabilities across various domains, yet mathematical reasoning remains a notable challenge @xcite @xcite @xcite . mathematical word problems (mwps) serve as a critical benchmark for evaluating llms' mathematical reasoning abilities, as they demand both natural language understanding and mathematical computation skills. recent advancements, such as chain-of-thought (cot) prompting @xcite and specialized mathematical training @xcite , have shown promising progress in this domain. however, these improvements primarily focus on accuracy metrics while overlooking the critical aspect of understanding and addressing the underlying patterns of errors. the analysis of llm errors in mathematical reasoning has emerged as a vital research direction. @xcite introduces prm800k, a dataset featuring manually annotated solution steps to enhance model performance through pretraining. building on this foundation, @xcite develops mr-gsm8k, which incorporates potentially erroneous reasoning paths and requires models to both evaluate solution correctness and pinpoint error locations. this approach aligns with @xcite 's work, which requires models to not only identify but also correct errors. however, current approaches to error analysis in mathematical reasoning face several fundamental deficiencies. traditional error classification methods rely heavily on static, predefined categories that are based on human observation and empirical judgment. this approach proves inadequate for capturing the diverse and evolving nature of llm errors, particularly when scaling to large datasets. the challenge is further complicated by the observation that different llms often generate distinct types of errors when solving the same problem, making static classification frameworks increasingly obsolete. for instance, gpt-3.5 and llama-3.1-70b may produce entirely different types of errors when solving the same mwp, necessitating a more flexible classification approach. moreover, as illustrated in figure 1 , predefined error taxonomies often employ broad, ambiguous categories such as "hallucination" or "calculation error" that lack the granularity needed for meaningful analysis. this oversimplification obscures the root causes of errors and hampers the development of targeted improvements. additionally, current methodologies typically focus solely on identifying the first error in a solution path, overlooking the potential cascade of subsequent errors that could provide valuable insights into model behavior. to facilitate systematic error analysis, we introduce mwpes-300k (math word problem error solutions), a comprehensive dataset containing 304,865 erroneous mwp solutions. unlike existing error analysis datasets that focus on single dataset or synthetic errors, mwpes-300k captures real error patterns by collecting model outputs across four diverse mwp datasets. furthermore, we propose a novel framework for automated dynamic error classification in mathematical reasoning. our approach adaptively evolves error categories based on observed model outputs, enabling fine-grained analysis of error patterns while reducing human bias and intervention. moreover, we introduce an innova-tive error-aware prompting (eap) mechanism that explicitly guides models to avoid potential errors during problem-solving, substantially enhancing mathematical reasoning performance. our main contributions are as follows: ## error classification framework to address the limitations of static error categorization, we develop the first automated framework for adaptive mathematical error classification. our framework dynamically adapts to diverse error patterns while effectively maintaining classification consistency across different llms and problem types. the architecture of this framework is illustrated in figure 3 . question: heather's razors come 4 to a pack and cost @xmath02.00 coupon. how much will each individual razor cost, in cents, after the discount if she buys 2 packs of razors? correct answer: 25 cents. ## error-aware prompting the error-aware prompting algorithm aims to enhance the llm's ability by incorporating relevant errors of the question. this algorithm involves two main steps: knowledge point labeling and relevant error summary retrieval. 9: if f(v k , v e ) ≥ θ then 10: add e to r. 11: end if 12: end for 13: an error-aware prompt p is constructed by appending the relevant summaries r to the original question q. this prompt is then fed into the llm to generate a response. the detail of the algorithm is shown in algorithm 1. ## conclusion we introduce a dynamically adaptive framework for error classification in mwp solving. it automatically updates error categories based on observed outputs, rather than using static categories. we propose the mwpes-300k dataset, with 304,865 error samples across 15 llms and 4 mwp datasets, enabling a thorough error analysis. specifically, dataset complexity positively correlates with error diversity. as llms enhance their core mathematical skills, the errors they make tend to become more complex, reflecting challenges in advanced reasoning. increased model parameter size correlates with reduced overall errors and more balanced distributions. we propose error-aware prompting (eap), which leverages categorized errors to significantly improve both in-distribution and out-of-distribution performance on math and tal-scq5k datasets. we suggest that a focus on error analysis can serve as a good way to develop more robust and reliable llms for mathematical reasoning tasks.