UTM ath: A Benchmark for Math Evaluation with Unit Test (2025)

Abstract: The evaluation of mathematical reasoning capabilities constitutes a critical pathway toward achieving Artificial General Intelligence (AGI). Prevailing benchmarks including MATH and AIME mainly feature single-instantiation problems with fixed numbers, permitting pattern matching instead of principled deductive reasoning and leaving generalization on isomorphic problem variants untested. To address these limitations, we propose the UTMath Benchmark, employing rigorous unit testing methodology that simultaneously quantifies solution accuracy and solution space generality. It comprises 1,053 problems spanning 9 mathematical domains, each accompanied by an average of 68 varied test cases. With answer possibilities per problem on average, UTMath sets new standards for robust reasoning while preventing memorization. UTMath is highly challenging, with the best-performing model, o1-mini, solving only 32.57% of the problems, followed by o1-preview at 27.16%, and GPT-4o at 26.93%. We further propose Reasoning-to-Code Thoughts (RCoT), a prompting strategy that decouples symbolic reasoning from code synthesis. RCoT guides LLMs to first derive formal reasoning structures before generating executable code, producing generalizable solutions rather than situation-specific answers. To help the community push mathematical reasoning further, we release UTMath-Train (70k samples), a companion training set generated under the same protocol. Our benchmark can be accessed via the following link: [UTMath](https://utmathhomepage.github.io/)

## introduction the pursuit of agi necessitates strong mathematical reasoning capabilities, making the evaluation of such abilities a crucial area of research @xcite . recent advancements in llms have demonstrated remarkable proficiency in solving complex mathematical problems, achieving amazing performance on various datasets of math question ground truth: 2 let x be the measure of … there are @xmath0 integers strictly between 2.5 and 5. the same question with {@xmath1 ## 2500 however, gpt-4o is wrong with another n … figure 1 : comparison of utmath with other benchmarks. on the left, gpt-4o successfully solved the original problem, but failed to generalize when the input was modified by merely changing a single numeric value. on the right, utmath is shown, where each problem includes multiple test cases, and a solution is deemed correct only if all are passed by the code generated by the model. we also propose a new prompting method rcot in which the llm first reasons through the problem and then generates code. word problems (mwps), such as gsm8k @xcite , math @xcite , the-oremqa @xcite . however, conventional benchmarks present several intrinsic limitations that hinder the precise and comprehensive assessment of these models' mathematical reasoning capabilities @xcite . first, these benchmarks predominantly assess models using narrowly defined problem formats with fixed numerical instantiations, thereby constraining their ability to evaluate generalization across structurally analogous but variationally distinct scenarios, as illustrated in fig. 1 . second, the evaluation protocols often depend on rule-based matching or the llm-as-a-judge paradigm ( @xcite ) both of which are vulnerable to inconsistencies due to the stochastic nature of llm outputs. for example, in datasets such as gsm8k, theoremqa, and math, a correct solution must be extracted in a form that exactly matches the reference answer, which fails to accommodate semantically equivalent but syntactically divergent responses. while recent work has made great progress in developing new benchmarks, many of these approaches still fall short of addressing the fundamental limitations of earlier datasets. for instance, benchmarks like gsm-hard @xcite , gsm-ic @xcite , gsm-plus @xcite , metamath @xcite build upon gsm8k or math by introducing perturbations-including value substitution, input reversal, and distractor insertion. while these augmentations provide incremental improvements, they are often constrained by limited coverage and incur substantial human and computational costs. against this backdrop, our work aims to fill these critical gaps by constructing a principled and robust benchmark capable of rigorously evaluating the mathematical reasoning abilities of llms. inspired by evaluation paradigms in software engineering, we adopt a unit testing-based framework to assess the soundness of llms' reasoning processes. in this framework, a solution that passes all unit tests is considered to reflect reliable and consistent reasoning. to this end, we introduce utmath, a novel benchmark derived from the on-line encyclopedia of integer sequences (oeis) (oeis foundation inc., 2024). as shown in 1, the benchmark consists of 1,053 cutting-edge problems spanning 9 mathematical domains, such as number theory and geometry. each problem is accompanied by more than 68 test cases, each consisting of concrete input-output pairs that enable precise evaluation of generalization and correctness. utmath employs a unit-test-driven framework to evaluate mathematical reasoning through gen-eralizable code solutions that must pass multiple test cases per problem class. unlike benchmarks focused on numerical answers, this design explicitly requires executable implementations, testing both conceptual understanding and code-generation rigor-an advantage aligning with real-world problem-solving where precision and adaptability are critical. when testing program-of-thoughts (pot) @xcite , we observed that models' coding limitations directly hindered performance. to address this issue, we decouple reasoning and coding, and propose reasoningto-code of thoughts (rcot), which requires the llm to first perform mathematical reasoning in the initial turn, and then generate code based on that reasoning in the subsequent turn. our experiments demonstrate that rcot encourages more thorough reasoning before code generation, leading to more efficient and targeted solutions. utmath is highly challenging, we conducted a comprehensive study with 11 llms. some of our key findings are summarized as follows: (1) with the best-performing model, o1-mini, solving only 32.57% of the problems, followed by o1-preview at 27.16%, and gpt-4o at 26.93%, these results demonstrate the difficulty of utmath. (2) modern llms perform poorly in graph theory, group theory, geometry and topology (fig. 5 ). (3) with rcot, all evaluated llms generated more efficient solutions, with most models achieving higher scores (fig. 3 ). (4) rcot can improve the pass@k performance of llms ( § 5.4). (5) both reasoning and coding capabilities substantially influence overall performance. by leveraging the modular design of rcot, we can disentangle their individual contributions ( § 5.5). more interesting findings can be found in § 5. we hope our findings contribute to a deeper understanding of current reasoning ability of llms and the further development of models. ## reasoning-to-code thoughts compared to methods that simply check whether the outputs generated by llms are identical, the code-based evaluation approach enables more accurate assessment by using multiple test cases. initially, we adopted the program of thought (pot) method, where the llm was required to perform reasoning and code implementation in a single step. however, we observed that the model's code generation capability also influenced its performance on utmath. to address this, we propose the reasoning-to-code of thoughts (rcot) framework, which decouples reasoning and coding into two separate rounds of interaction. in the first round, the model is tasked solely with mathematical reasoning, without generating any code. in the second round, the model generates code based on the reasoning process from the first round. we can either use the same model for both rounds to observe its overall performance on utmath, or isolate the reasoning capability by fixing the second-round model to a dedicated coding model. the latter setting enables a more accurate evaluation of the model's reasoning ability. moreover, we find that by separating reasoning from code generation, rcot allows the llm to generate a step-by-step, detailed reasoning chain that includes relevant theorems, formulas, and mathematical properties. such deeper reasoning higer is better lower is better leads to the development of more efficient algorithms with lower time complexity. we present qualitative cases in appendix d. ## conclusion in this work, we investigate how to more accurately and effectively evaluate the mathematical reasoning capabilities of llms. we propose a cutting-edge benchmark, utmath, which comprises 1,053 problems spanning nine mathematical domains, with an average of 68 test cases per problem. this benchmark presents challenges: o1-mini, the best-performing model, successfully solves only 32.57% of the problems, followed by o1-preview at 27.16%, and gpt-4o at 26.93%. additionally, we introduce rcot (reasoning-to-code of thought). compared to pot, rcot improves pass rates and significantly enhances algorithmic efficiency of most models. overall, this research contributes to a deeper understanding of the current capabilities of llms in mathematical reasoning and lays the groundwork for the development of more advanced models in the future.