Evaluating Robustness of LLM s to Numerical Variations in Mathematical Reasoning (2025)

Abstract: Evaluating an LLM’s robustness against numerical perturbation is a good way to know if the LLM actually performs reasoning or just replicates patterns learned. We propose a novel method to augment math word problems (MWPs), producing numerical variations at a large scale utilizing templates. We also propose an automated error classification framework for scalable error analysis, distinguishing calculation errors from reasoning errors. Our experiments using the methods show LLMs are weak against numerical variations, suggesting they are not fully capable of generating valid reasoning steps, often failing in arithmetic operations.

## introduction recent llms @xcite @xcite have reported high accuracy rates on mathematical reasoning benchmarks, including gsm8k and math @xcite . however, a natural concern is that the models just follow surface patterns observed in their pretraining data rather than performing mathematical reasoning @xcite @xcite . perturbing superficial elements like names of individuals or specific numbers does not change how the problem should be solved. if an llm can perform reasoning in solving a math question, it should give correct answers with similar reasoning steps for both the original and its perturbed one. recent studies @xcite @xcite evaluated models' robustness against the perturbations based on this hypothesis. these studies have the following limitations: a) the size of the introduced variations was limited, b) they did not discuss ranges of numerical values such as digit sizes, and c) they did not distin-guish reasoning errors and computational errors and could not explain the source of errors. to address the limitations, we propose a scalable method to augment a math word problem (mwp) dataset by changing numerical values based on template questions. to analyze the impact of digit sizes on models' mathematical reasoning, we generate two distinct subsets by controlling the range of the replaced values, one with questions containing a small number of digits and the other with questions containing a large number of digits. we construct a new dataset, gsm-alt, generating 2,000 variants for each original question from gsm8k. moreover, we propose a novel framework for automated error analysis to distinguish two sources of errors: logical reasoning and numerical calculation. ## related work despite strong performance on math benchmarks, researchers are questioning whether current benchmarks can adequately evaluate the reasoning abilities of language models. @xcite expanded questions by adding non-essential contents, showing that models' performance decreases when the number of tokens in a problem increases. planbench @xcite is a benchmark to evaluate planning and reasoning capabilities. their findings suggest that even the state-of-the-art models still struggle with this. @xcite functionalized the math questions to create a dynamic dataset, providing a robust evaluation metric against potential data leakage to models' pretraining. @xcite demonstrated that the models' high accuracy depends on a specific token bias, and the models' reasoning capability depends on recognizing certain superficial patterns. @xcite and @xcite constructed reversal versions of the original questions and showed that the current models performed poorly on the reversal ones. ## experimental settings we use gsm8k as the base dataset for our experiment. gsm8k consists of mwps for primary and we randomly sample 250 questions from the gsm8k training set to create 250 question templates manually. given the templates, we generate 1,000 hard variants and 1,000 easy variants for each template. as a result, our new dataset gsm-alt consists of the hard and easy variant set, each containing 250,000 variants. we use accuracy as a primary evaluation metric. for the original instances from the base dataset (original gsm8k), we use a standard accuracy. for generated variants from our dataset, we first calculate the accuracy for each template variant set containing 1,000 variants, and then we average them over all the 250 templates. the target models to be evaluated include generic models (llama-3-8b-instruct, llama-3.1-8b-instruct, llama-3.1-70b-instruct, mistral-7b-instruct-v0.3) and math models that were finetuned on mathematical contents (deepseekmath-7b-rl, wizardmath-7b-v1.1). regarding the generation settings, we use greedy search to maximize the reproducibility and stability of results. to minimize the influence of few-shot examples while ensuring that the model can perform mathematical reasoning, we adopt the zeroshot cot prompting for solution generation and extract the final answer in the same way as @xcite for generic models. as for math models, we adopt the specifically designed prompts, which are recommended on their web pages. the prompts used in the experiment can be found in appendix e.2. ## results ## error analysis on solutions to identify the source of errors, we classify errors into two types: calculation errors and reasoning errors. if an incorrect solution only contains failures in calculations, we call it a calculation error. if an incorrect solution contains incorrect reasoning steps, we label it a reasoning error regardless of its incorrect calculations. as gsm-alt will be larger than its original dataset, manually checking each generated solution is not practical, and thus, we propose a novel framework that automatically classifies errors into calculation or reasoning errors. ## conclusion we proposed a novel method to augment mwp datasets, which produces a dataset for evaluating llms' robustness against numerical variations at a large scale. using our templates, anyone can easily generate thousands of variants from one original question in the gsm8k, which was not possible with any preceding proposals. we also proposed an automated error classification framework for detailed error analysis, distinguishing calculation errors from reasoning errors. using the methods, we empirically showed that the six llms we tested were weak against numerical variations, especially when the numerical values were large. this finding is consistent with previous studies @xcite @xcite , but we confirm it with more variants. our error analysis uniquely identified that calculation errors contributed to a substantial proportion of incorrect solutions, suggesting llms' incapability of arithmetic operations is the main source of limited capabilities in math word problems. moreover, we found that llms still fail in their reasoning steps, especially when they encounter variants with larger numerical values. given our findings, it is still hard to say that current llms are robust against numerical variations.