LLM s Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLM s (2025)

Abstract: Large language models (LLMs) demonstrate impressive capabilities in mathematical reasoning. However, despite these achievements, current evaluations are mostly limited to specific mathematical topics, and it remains unclear whether LLMs are genuinely engaging in reasoning. To address these gaps, we present the Mathematical Topics Tree (MaTT) benchmark, a challenging and structured benchmark that offers 1,958 questions across a wide array of mathematical subjects, each paired with a detailed hierarchical chain of topics. Upon assessing different LLMs using the MaTT benchmark, we find that GPT-4 achieved a mere 54% accuracy in a multiple-choice scenario. Interestingly, even when employing Chain-of-Thought prompting, we observe mostly no notable improvement. Moreover, LLMs accuracy dramatically reduced by up to 24.2 percentage point when the questions were presented without providing choices. Further detailed analysis of the LLMs’ performance across a range of topics showed significant discrepancy even for closely related subtopics within the same general mathematical area. In an effort to pinpoint the reasons behind LLMs performances, we conducted a manual evaluation of the completeness and correctness of the explanations generated by GPT-4 when choices were available. Surprisingly, we find that in only 53.3% of the instances where the model provided a correct answer, the accompanying explanations were deemed complete and accurate, i.e., the model engaged in genuine reasoning.

## introduction large language models (llms) have increasingly demonstrated remarkable capabilities as mathematical reasoners, underscoring their potential in complex problem-solving domains @xcite @xcite . recent studies have shown that llms, when applied to mathematical problems, can exhibit a high degree of reasoning ability, often aligning with or even surpassing human-level performance in certain contexts. this proficiency in mathematical reasoning is further enhanced by innovative techniques such as chain-of-thought @xcite , tree-of-thought @xcite , and self-verification @xcite , emphasizing on the importance of the procedural steps in solving a mathematical problems. despite these advancements, several critical gaps persist in our understanding of llms' mathematical reasoning capabilities. firstly, it remains unclear which specific areas of mathematics llms excel or falter in, as comprehensive evaluations across diverse mathematical domains are lacking. secondly, distinguishing between instances where llms rely on memorization versus genuine reasoning is challenging, raising questions about the depth of their understanding. thirdly, the influence of multiple-choice formats on llm behavior is not well understood, suggesting that models' performance might be affected by the structure of the questions posed. these gaps underscore the necessity for a more robust benchmark that facilitates a holistic evaluation of llms, enabling us to dissect their strengths, weaknesses, and the nuances of their problem-solving strategies. in this paper, we developed the mathematical topics tree (matt) benchmark by initially leveraging wikipedia's "lists of mathematics topics" 2 to identify key areas in mathematics, resulting in twelve major topics that span both pure and applied mathematics. this was followed by extracting important reference books for each topic from wikipedia to build a detailed topical tree. we then further refine the benchmark by using the books' figure 1 : overview of mathematical topics tree (matt) benchmark, a challenging and structured benchmark that presents questions spanning a diverse range of mathematical subjects, each associated with a detailed hierarchical structure of topics. after developing matt, we evaluate the mathematical reasoning capabilities of various llms, including commercial models openai o1-mini, @xcite and chatgpt @xcite (turbo versions), alongside the open-source llms, llama3.1-70b @xcite and @xcite . notably, gpt-4 achieved only 54% accuracy in a multiple-choice format. furthermore, the use of chain-of-thought prompting mostly did not enhance llms' performance, underscoring the benchmark's complexity and suggesting that mere step-by-step reasoning might be insufficient. also, when questions were presented without multiple-choice options, llms accuracy dropped by up to 24.2 points. our analysis also revealed notable discrepancies in performance across topics, showing inconsistent abilities even within related subtopics of the same mathematical domain. to understand the underlying causes of the llms' inadequate performance and their inconsistent results across various topics, we did a detailed evaluation of the explanations provided by gpt-4. surprisingly, we observe that only in 53.3% of cases where the models answered correctly, the ex-planations were also complete, i.e., gpt-4 engaged in genuine reasoning. these cases were typically associated with simpler or more well-known questions that required only a few straightforward steps to resolve. for more complex questions demanding either more number of steps, complicated calculations, or creative/intelligent problem-solving, gpt-4 often failed, resorting to tactics like choice engineering, unsupported theorems, circular reasoning, or memorization instead of true reasoning. ## matt: mathematical topics tree benchmark in recent years, llms have shown remarkable abilities in mathematical reasoning. yet, their prowess is not fully understood due to the narrow focus of current benchmarks, which typically concentrate on specific mathematical areas. this limitation hinders our understanding of the depth and breadth of llms' reasoning capabilities. there's a pressing need for more comprehensive mathematical benchmarks that cover a wider array of topics and offer deeper insights into the models' reasoning processes. such benchmarks would not only challenge the models across a broader mathematical spectrum but also can help with better understanding the nuances of how and where these models apply reasoning. to address this gap, we create the mathematical topics tree (matt) benchmark. we start by harnessing the "lists of mathematics topics" available on wikipedia as a foundational resource. this exploration was crucial for identifying the spectrum of mathematical knowledge we aimed to en-compass. extracting the list of mathematics topics from wikipedia, we identified twelve principal topics that comprehensively encapsulate the breadth of pure and applied mathematics. then, for each topic, we extracted one or few key reference books listed on their respective wikipedia pages. the topics and their corresponding resources are as follows: for pure math we consider algebra @xcite @xcite , calculus and analysis @xcite , number theory @xcite , combinatorics @xcite , geometry and topology @xcite @xcite , and logic @xcite . in applied math we have game theory @xcite , probability @xcite , operations research (hillier and lieberman, 2015), differential equations ( boyce @xcite , and information theory and signal processing @xcite . next, we used the tables of contents from selected reference books to structure the matt topical tree, mapping the hierarchy of topics and subtopics to create a comprehensive graph of mathematical domains. the final step in the creation of matt involved a detailed extraction of questions from the sections of the reference books, gathering them under the leaf nodes within our topic tree. for each question identified, we then crafted multiplechoice options to facilitate an objective assessment framework. to generate the options, we selected choices that closely resembled the actual answer, such as those with similar numerical values, those attain by omitting a step from the proof, or those presenting alternative combinations. for instance, if the correct answer was "a & b", we included "a or b" as one of the possible choices. we provide an illustration of matt in figure 1 . to create the matt benchmark, the process involved the following steps: (1) the authors of the paper manually extracted questions and answers from reference books. (2) another author, independent from the one who extracted the question, reviewed and rechecked the correctness of the question and its answer, revising it if necessary to ensure the quality of the benchmark. during the review process, we observe that around 95% of the questions showed agreement between annotators, requiring no further revisions. this exhaustive process collectively took the authors more than 300 hours to complete, underscoring the dedication involved in the matt dataset creation. the statistical overview of the matt benchmark is detailed in moreover, we strategically curated a diverse mix of problem types-ranging from computational exercises to proof-based questions and applied scenarios-across all mathematical topics. by balancing the cognitive demands required to solve these problems, we achieved a uniform difficulty level. this methodological diversity ensured that each topic was evaluated on similar grounds, preventing any particular area from being inherently easier or more challenging due to the nature of its questions. finally, while extracting questions, we exclude questions that are overly popular/simplistic or had their responses provided in the books immediately following them to mitigate the risk of data contamination. ## experimental details we assessed the performance of commercial llms-openai o1-mini, gpt-4 (openai, 2023) and chatgpt @xcite (turbo versions)-alongside the open-source llms llama3.1-70b @xcite and mistral @xcite (mistral-7b-instruct-v0.2), using the matt benchmark. in our evaluation, we structured the prompts to request that llms first generate an explanation and then the final answer. in the multiple-choice setting, we specifically directed the models to select one of the provided options (a, b, c, or d) as their final answer. additionally, for zero-shot chain-of-thought prompting, we appended "let's think step by step" to the ## experiments in this section, we begin with an analysis of llms' mathematical reasoning capabilities using the matt benchmark. subsequently, we examine the variation in model performance across different sub-topics. we then assess the effect of choice availability by presenting matt questions to llms without multiple-choice options. lastly, we concentrate on gpt-4's explanations, manually annotating the level of reasoning in each explanation and exploring the strategies employed by gpt-4 to arrive at correct answers. ## related work as llms grow stronger, they exhibit significant mathematical reasoning capabilities on existing benchmarks. however, the scope of current evaluation settings is restricted in terms of the breadth of mathematical areas covered and fails to conclusively determine whether these models genuinely engage in reasoning or rely on alternate strategies. mathematical benchmarks previous research primarily concentrated on developing benchmarks for math word problems-mathematical problems in the form of written description-which typically require only a few steps to solve, often involving basic arithmetic or elementary algebra @xcite @xcite . additionally, the work in @xcite introduced a comprehensive mathematical reasoning benchmark that encompasses 23 varied tasks across four dimensions: mathematical abilities, language format, language diversity, and external knowledge. furthermore, @xcite presented a multi-modal benchmark with a focus on geometry. the most relevant to our study are the math @xcite and theoremqa @xcite benchmarks. despite providing mathematical questions on various topics, they have a much narrower scope compared to our benchmark and did not provide a detailed topical breakdown for each question. additionally, a recent effort @xcite has begun to generate large-scale synthetic mathematical benchmarks for instruction tuning of llms. llms and math in recent years, llms have shown notable achievements in mathematical reasoning @xcite . these accomplishments are supported by methods aimed at enhancing llms' performance, predominantly through decomposed reasoning. such strategies, inspired by human problem-solving processes, include providing step-by-step guidance @xcite @xcite , employing verification mechanisms to enhance model consistency and accuracy @xcite , and incorporating complex strategies @xcite . ## conclusion in this paper, we provide a comprehensive evaluation on mathematical reasoning of llms. we create the mathematical topics tree (matt) benchmark, a systematically organized set of questions covering a wide range of mathematical subjects linked to a detailed topic hierarchy. exploring llms accuracy on matt, we observe their struggle with a wide range of mathematical topics, particularly when deprived of multiple-choice options. we also observe the discrepancy in llms' performance across various topics and the lack of substantial improvement with chain-of-thought prompting. to investigate the gaps in models performances, we manually analysis their explanations in answering the questions. we find that in only 53.3% of the instances where gpt-4 provided a correct answer, the accompanying explanations were deemed complete. further, we observe that models perform better on simpler problems but rely on alternative strategies for complex ones. this indicates a fundamental gap in llms' ability to engage in deep, creative, and complex mathematical thinking. we will release code and data for matt.