Evaluating LLM s’ Mathematical and Coding Competency through Ontology-guided Interventions (2025)

Abstract: Recent advancements in Large Language Models (LLMs) have showcased striking results on existing logical reasoning benchmarks, with some models even surpassing human performance. However, the true depth of their competencies and robustness in reasoning tasks remains an open question. To this end, in this paper, we focus on two popular reasoning tasks: arithmetic reasoning and code generation. Particularly, we introduce (i) a general ontology of perturbations for math and coding questions, (ii) a semi-automatic method to apply these perturbations, and (iii) two datasets, GSMore and HumanEval-Core, respectively, of perturbed math and coding problems to probe LLM capabilities in numeric reasoning and coding tasks.Through comprehensive evaluations of both closed-source and open-source LLMs, we show a significant performance drop across all the models against the perturbed questions, suggesting that the current LLMs lack robust problem solving skills and structured reasoning abilities in many areas, as defined by our ontology.

## introduction logical reasoning, which includes mathematics and programming, in a structured and well-defined domain, becomes increasingly complex with the increasing presence of diverse situations, events, and contexts represented in natural language. however, the process of solving a mathematical expression should remain consistent and a logical reasoner should be robust to certain changes in the context. consider, problems i) i have two apples and sam has three more. how many apples are there?, and ii) i see two wombats. and my friend sees another three. how many wombats are there?. variations i and ii should yield the * now at google deepmind. research question: how robust are the capabilities of large language models (llms) in reasoning and understanding the problem-solving process? our approach: introducing small modifications to the selected seed math problems from gsm8k and humaneval, where leading llms, such as o1-preview, have shown excellent performance. our goal is to examine whether these models can effectively handle slight variations to the seed questions, which they previously solved successfully. an example is presented in figure 2 . if the models struggle with these modified problems, it would suggest that there is still room for improvement in llms' core reasoning abilities. evaluating llms on these modified questions could provide fine-grained insights into the robustness of the reasoning abilities in the context of math and coding. specifically, our evaluation benchmark is based on our novel ontology of perturbation operations that lists various changes across a diverse set of factors. we apply these perturbations to existing arithmetic and coding problems. these perturbations allow us to assess whether the model truly comprehends underlying concepts and structures or regurgitates. for instance, while a model may correctly answer a math question in a dataset like gsm8k, it may struggle with a simple perturbation to the question, such as replacing numerical values @xcite in math questions with variables, which challenges the model to establish relationships among the variables, revealing its deeper understanding (or lack thereof). by introducing these ontological perturbations, (1) we gain insights into the models' reasoning abilities and (2) uncover strategies for future data augmentation that can then be utilized to enhance llms through weakly supervised fine-tuning methodologies. our perturbations extend beyond mere numerical alterations to seed questions; they also challenge with the underlying concepts and principles required to solve a problem. for instance, we introduce a category critical thinking, which includes scenarios where multiple correct answers may be possible, depending on the constraints on various variables, as demonstrated in the example below: original question: a merchant must choose between two purchasing plans: jewelry valued at @xmath08,000. his financial advisor predicts a **2.5% increase** in the jewelry market and a **1.2% increase** in the electronic gadgets market over the next month. if the merchant aims to maximize profit by the end of this month, what would be his profit? perturbed question: a merchant must choose between two purchasing plans: jewelry valued at @xmath18,000. his financial advisor predicts that the jewelry market will rise by **x%** and the electronic gadgets market by **1.2%** over the next month. if the merchant aims to maximize profit by the end of this month, what would his profit be? such instances are unique to our datasets making the evaluation more rigorous. we sample five questions from gsm8k, subjecting them to 44 types of perturbations, followed by extensive manual filtering and validation, resulting in 216 perturbed questions in gsmore. as our ontology is generic, we also apply these perturbations to randomly sampled five code-generation questions from humaneval producing 219 questions in humaneval-core. this work makes the following contributions: 1. we propose a manually-curated, novel, extensive, and extensible ontology of perturbation operations for math and coding problems. 2. we provide a strategy to semi-automatically apply such perturbations first through gpt-4, followed by manual filtering. we refer to this dataset as gsmore-mathematics-oriented robustness evaluation, consisting of 216 examples starting from just five questions in gsm8k. 3. we show that the ontology and the semiautomatic pipeline can be easily extended to domains like code-generation, by creating the dataset humaneval-core -codingoriented robustness evaluation, having 219 samples from five seed questions in humaneval. 4. we further evaluate the math and code-based reasoning abilities of the latest llms, including gpt-4o and o1, using our datasets, revealing their vulnerability to these perturbations. the integration of our ontology with the newly introduced dataset, referred to as gsmore and humaneval-core, paves the way for a fresh perspective in assessing the math and code reasoning capabilities of llms. this evaluation approach transcends the often unidimensional leaderboard performance metrics by delving into llm performance on finely defined classes of problems. in the nli context, @xcite utilized human workers to directly perturb a hypothesis, keeping the premise constant; and in a post-hoc way, identifies the categories (or factors) which such revisions pertain to. previous works @xcite @xcite discusses ways of perturbation, by identifying a set of factors that is specifically designed to increase the complexity of a seed questions in limited ways. the categories are broad and do not exploit the logical nature of the underlying domain (along with the linguistic dimensions of the instruction). this is where, we believed, an ontological approach may help, where broader categories can help us generalize, while fine-grained sub-categories exploit the domain-specific characteristics. let's take mathematics for example. the solution to a reasoning problem can depend on the number and complexity of operations, variables, functions, and possible existing theorems (external knowledge). similarly, code generation problems can depend on the data structures, variables, functions, and libraries it needs access to. on top of this well-defined set of factors existing in structured reasoning problems, the list of factors expands as the problem is expressed in natural language. entities and relations expressed in the text need to be mapped to variables and constants (in both). physical actions (giving and taking apples) may need to be mapped to mathematical operations (or code). it is clear that the set of logical and linguistic factors co-exist in these reasoning problems, detailed in appendix i.1. therefore we come up with an extensible ontology, capturing the above nuances. we believe it will capture and categorize the factors where llms fail over multiple domains. as others have shown, the same process can be enabled to perform data augmentations. ## curation of gsmore and humaneval-core our objective is to assess the resilience of llms to perturbations of math and coding questions along various dimensions. thus, as seed datasets, we use gsm8k (cobbe et al., 2021b)-a collection of mathematical problems demanding rigorous arithmetic and logical reasoning-and hu-maneval (chen et al., 2021a) for coding. five questions foot_1 from gsm8k are perturbed using our ontological framework (see appendix i) to generate gsmore. on the other hand, we sampled five coding problems from humaneval dataset (chen et al., 2021a) that were perturbed using the ontology explained in appendix i. these perturbations are aimed at modifying the problems in terms of complexity and representation to assess the robustness of the llms to these ontological categories of perturbations. fig. 1 shows examples of three perturbed questions and answers from gsmore and humaneval-core. examples and definitions of all the remaining perturbations are present in appendix i. we use a three-staged combination of automatic generation from gpt-4 (openai, 2023) with human verification and annotation to create gsmore and humaneval-core: (i) perturbed question generation ( §3.1), (ii) filtering and validation of generated questions ( §3.2), and (iii) annotating final answers ( §3.3). ## related works a variety of datasets are developed to assess reasoning capabilities across multiple domains. in causal reasoning, the key datasets are by @xcite . despite the advancements in large language models @xcite , recent studies @xcite contend that these models more closely resemble stochastic parrots @xcite than true systematic reasoners, exhibiting significant limitations particularly in scenarios not covered by their training data @xcite . therefore, recent works have focused on the robustness of reasoning under various perturbations that alter reasoning questions. different domainspecific methods have been proposed for generating test cases for reasoning tasks @xcite , as summarized in beyond perturbation-based question generation, educational measurement research has long sought assessment instruments that can reliably discriminate learners according to specific mathematical skills. for example, homjan et al. ( 2022 ) develop a diagnostic test targeting grade-4 addition and subtraction, while @xcite design and standardise a geometry achievement test for grade-9 students. these studies emphasise psychometric qualities such as item difficulty, discrimination index, and overall test reliability rather than the micro-level composition of individual questions. the resulting insights into difficulty calibration, diversity of item types, and longitudinal tracking of learner progress can inform how we evaluate the evolving capabilities of large language models (llms) on mathematically grounded tasks. ## conclusion our study evaluated the robustness of several prominent large language models (llms) in handling mathematical and coding problems. by employing an ontology for random perturbations on questions from the gsm8k and humaneval datasets, we crafted two specialized datasets, gsmore and humaneval-core, containing 216 and 219 questions, respectively. these datasets target a broad variations of mathematical and coding problem-solving and analytical skills, resulting in notable performance drops in llms upon evaluation. the introduction of gsmore and humaneval-core provides a new framework for assessing llms' abilities in mathematics and coding, while also revealing their vulnerabilities in consistent reasoning across different formats. this research highlights the complex challenges that llms face, stressing the importance of continued exploration into their strengths and weaknesses in logical reasoning tasks.