H igh MATH : Evaluating Math Reasoning of Large Language Models in Breadth and Depth (2025)

Abstract: With the rapid development of large language models (LLMs) in math reasoning, the accuracy of models on existing math benchmarks has gradually approached 90% or even higher. More challenging math benchmarks are hence urgently in need to satisfy the increasing evaluation demands. To bridge this gap, we propose HighMATH. Problems in HighMATH are collected according to 3 criteria: problem complexity, knowledge domain diversity and fine-grained annotations. We collect 5,293 problems from Chinese senior high school mathematics exams published in 2024, covering 8 subjects and 7 levels of difficulty, with each problem involving an average of more than 2.4 knowledge points. We conduct a thorough evaluation of latest LLMs on the curated HighMATH, including o1-like models. Evaluation results demonstrate that the accuracy of advanced LLMs on HighMATH is significantly lower than that on previous math reasoning benchmarks. This gap even exceeds 30%. Our results also suggest that properly trained smaller LLMs may have great potential in math reasoning. Our data is available at https://github.com/tjunlp-lab/HighMATH.

## introduction llms have achieved significant progress in math reasoning @xcite . when the challenging math @xcite benchmark was initially proposed for evaluation, the accuracy of llms did not reach 20%. however, in just three years, llms are capable of achieving over 60% accuracy @xcite . recently, with the emergence and application of techniques such as automatic process supervision @xcite , test-time scaling @xcite , and reinforcement learning @xcite , many models even achieve or exceed 90% accuracy on math (deepseek-ai * corresponding author (dfk3ureohp (dfk6xemhfw (dfk'liilfxow\/hyho &rpsdulvrq'lphqvlrqv @xmath07+ +ljk0$7+ we count the number of knowledge points in each problem, subject, and difficulty level in the problems sampled from highmath and math. it can be seen that high-math contains a higher average number of knowledge points than math. @xcite . this rapid development of llms in math reasoning indicates that existing math benchmarks are no longer sufficient to evaluate and differentiate latest llms. particularly, commonly used benchmarks, such as gsm8k @xcite , math @xcite , cmath @xcite , and gaokao @xcite , all suffer from these limitations. gsm8k, cmath and finemath @xcite , which originate from elementary school math, present problems that are simple and do not allow for complex reasoning, making it difficult to deeply evaluate advanced llms. math, sourced from american high school math competitions, is quite challenging but typically presents problems with limited knowledge points, thus lacking the ability to evaluate llms in integrating multiple mathematical subjects. gaokao, c-eval @xcite , m3ke @xcite , and other comprehensive benchmarks @xcite are designed for multidisciplinary evaluation, where math is only one of these subjects, featuring fewer problems and coarser granularity. to mitigate these challenges, we propose new chinese math benchmark highmath. our dataset offers a comprehensive and challenging assessment of llms in math reasoning by meticulously integrating a wide range of mathematical concepts, and knowledge into math problems with multiple reasoning steps. it contains 5,293 math problems, with each problem containing an average of more than 2.4 knowledge points, as shown in figure 1 . it covers all major areas of high school mathematics, including 8 domains, i.e., function derivatives, counting principles, trigonometric functions and triangle solutions, plane analytic geometry, sequences, solid geometry, statistics and probability, and others (logic, sets, inequalities, complex numbers reasoning). in addition to subject annotation, we categorize problems into difficulty levels from 1 to 7. we also collect problems with multiple sub-questions into a subset called highmath-hard. the average number of characters of problems and their solutions in highmath-hard are 249 and 1,201 respectively, posing great challenges to llms in math reasoning. we evaluate 14 llms (including both open-and closed-source models) on highmath. compared to math, all models exhibit a significant drop in accuracy on highmath, with even the latest reasoning language model, o1-mini, achieving only 52.53% accuracy. the o1-like model, deepseek-r1-distill-qwen-32b, achieves only 31.22%. we also evaluate llms under the majority voting, pass@n settings, and with the assistance of a python executor. however, these can only improve reasoning performance to a certain extent, indicating the difficulty of our benchmark. our contributions are summarized as follows. transformed datasets. currently, mathematical reasoning evaluations often use static benchmarks, which are prone to data contamination. to prevent or bypass this issue, efforts have been dedicated into (1) new datasets in different formats, (2) dynamic benchmarks and (3) synthetic datasets. mathvista collects multimodal math problems @xcite . work @xcite attempt to develop dynamic test sets. however, unlike elementary mathematics, more challenging math problems are difficult to implement in dynamic evaluations. other efforts @xcite @xcite @xcite modify the original information in the questions of the gsm8k or other datasets, such as variable names or numeric values, using templates to generate variants of the original problems for evaluation. among these methods, adding information unrelated to the questions can cause model performance to drop by as much as 65%. our dataset shows that even with pure text, without data synthesis, llms still face challenges when performing mathematical reasoning. ## experiments we conducted extensive experiments on high-math to evaluate 20 latest llms, including both open-and closed-source models. among them, o1like models were also evaluated. we divided all the problems from highmath into two categories for evaluation. the first category consists of problems that contain only one question, totaling 4,100 problems, and the second includes problems with multiple questions, totaling 1,193 problems (labeled "highmath-hard" in ## main result the main result of our work is that we found models that originally achieved high accuracy on math all experienced significant drops in accuracy on our benchmark. for example, qwen2.5-math-1.5b/7b/72b-instruct achieve 79.7%, 85.3%, and 87.8% respectively on the math benchmark using tir @xcite , while qwen2.5-math-1.5b/7b/72b-instruct only achieve 42.11%, 46.41%, and 50.62% respectively on our benchmark using tir. highmath can provide good discrimination among llms. doubao-1-5-thinking-pro-250415 performs best in handling single-question problems, with an accuracy rate of 71.02%, which greatly surpasses other models. to ensure efficiency, we set a maximum generation length of 2048 tokens for all models except for closedsource llms. we discovered that r1's overly long reasoning process prevented it from completing the reasoning within the limit, thereby affecting its accuracy. after re-evaluating the model using its official recommended settings, the accuracy of deepseek-r1-distill-qwen-7b improved to 53.32%. it can be seen that qwen-2.5-instruct, qwen-2.5-math, and qwen-2.5-math-instruct, which are based on the same 7b base model, show significant differences. comparing qwen-2.5-instruct and qwen-2.5-math, it is evident that even though qwen-2.5-math has been trained with a substantial new mathematical corpus, its performance is greatly influenced by whether the model has undergone instruction-following tuning. qwen-2.5-math-instruct, which combines the advantages of the two previous models, achieves the best results, indicating that both factors are important in model training. for qwen-math-instruct models, as model size increases, performance on highmath does not improve a lot. comparing the 7b model with the 72b model, there is only an 8% improvement in single-question problems' evaluation. the accuracy of all models drastically decreases in the evaluation of multi-question problems. the best-performing model, qwen-2.5-math-instruct, has an accuracy in multi-question problems that is less than half of its accuracy in single-question problems. models that perform poorly on singlequestion problems tend to perform even worse on multi-question problems. among the o1-like models, the accuracy of qwq-32b-preview on multi-question problems is surprisingly 5.87% lower than that of qwen-2.5-math-instruct 7b. besides skywork-o1-open-llama-3.1-8b, other o1like models also fail to meet the expected results, which may indicate that a more complex reasoning process does not necessarily improve accuracy. the majority vote and pass@8 evaluation settings significantly improve the accuracy of each model in reasoning on single-question problems, although the degree of improvement varies. the most noticeable increases occur in models that originally have lower accuracies. for example, the pass@8 accuracy of the qwen 1.5b model even surpasses that of many 7b models. these results may indicate that these smaller llms already possess strong mathematical reasoning abilities, but the responses they generate in a single pass are very unstable, leading to poor accuracy performance. using a python executor can effectively mitigate this issue. however, it is worth noting that there is an upper limit to the accuracy achievable with majority vote, pass@8, or using a python executor on highmath. the highest accuracy is achieved by skywork-o1-open-llama-3.1 under the pass@8 evaluation setting, reaching only 61.49%. ## analysis of problem subjects and difficulty levels we conducted an analysis of performance across problem subjects and difficulty levels between o1-mini, qwq-32b-preview, qwen-2.5-math-instruct-72b, skywork-o1-open-llama-3.1-8b, and the meta-llama-3-8b-instruct model. all comparisons are conducted under the evaluation results of single-question problems. ## challenges of llms in associating multiple math concepts from the evaluation results presented above, it is evident that although the mathematical knowledge tested in both the math and highmath exams originates from the high school level, there is a sig-dataset math math500 qwen-2.5-math-instruct-7b 76.6 75 as demonstrated in figure 1 , highmath contains a higher average number of knowledge points compared to math. we conducted further analysis to examine whether llms face challenges in associating multiple math concepts. we used 70 samples from math and 108 samples from highmath, all of which have core knowledge points annotated as mentioned in section 3.3. as seen in figure 3 , the range of the number of knowledge points annotated in both math and highmath varies between 1 and 4. in math, the majority of problems contain only one or two knowledge points, whereas in highmath, the majority of problems contain two or three knowledge points. in math, the accuracy of reasoning shows little variation across problems with one to three knowledge points. however, in highmath, the accuracy of reasoning varies more significantly across problems with different numbers of knowledge points, showing a clear downward trend as the number of knowledge points increases. this suggests that llms may indeed face difficulties in associating multiple math concepts, which also highlights the evaluative advantage of highmath. ## ablation study to demonstrate the effectiveness of our dataset, we conducted identical tests on the qwen-2.5-math model using both math and math-500. as shown in to test whether language affects mathematical reasoning, we extracted 1,000 questions proportionally from the eight subjects of highmath and translated them into english using gpt-4-0613 for testing the qwen-2.5-math-instruct-7b. experimental results, shown in ## conclusion we propose a new mathematical reasoning dataset, highmath, for llm evaluations. highmath features a comprehensive inclusion of different subjects of mathematical concepts, with multiple knowledge points integrated into each problem. additionally, highmath includes fine-grained annotations, divided into 8 categories and 7 difficulty levels, totaling 5,293 problems. based on high-math, we have conducted a comprehensive evaluation of the most recent popular llms. the evaluation results confirm the effectiveness of our dataset, with the best model achieving an accuracy of 62%.