M ath B ench: Evaluating the Theory and Application Proficiency of LLM s with a Hierarchical Mathematics Benchmark (2024)

Abstract: Recent advancements in large language models (LLMs) have showcased significant improvements in mathematics. However, traditional math benchmarks like GSM8k offer a unidimensional perspective, which fall short in providing a holistic assessment of the LLMs’ math capabilities. To address this gap, we introduce MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large language models. MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. The benchmark progresses through five distinct stages, from basic arithmetic to college mathematics, and is structured to evaluate models at various depths of knowledge. Each stage includes theoretical questions and application problems, allowing us to measure a model’s mathematical proficiency and its ability to apply concepts in practical scenarios. MathBench aims to enhance the evaluation of LLMs’ mathematical abilities, providing a nuanced view of their knowledge understanding levels and problem solving skills in a bilingual context.

## introduction mathematical reasoning and problem-solving represent pivotal facets of human intelligence and have captivated the interest of artificial intelligence (ai) research for decades. the capability of machines to grasp, interpret, and address mathematical challenges not only serves as a benchmark for their cognitive prowess but also fulfills a critical role in their deployment across various sectors. the advent of modern large language models (llms) such as openai's chatgpt and gpt-4 @xcite has marked a significant milestone, showcasing an unparalleled ability to gener- † corresponding authors. ate text that mirrors human-like discourse and to unravel intricate mathematical conundrums @xcite . despite these advancements, the evaluation of llms' mathematical capabilities remains hampered by some inherent limitations of existing benchmarks @xcite , @xcite , etc.). these resources predominantly offer a singular perspective on problem-solving abilities and lack comprehensive difficulty grading. math @xcite attempted to classify high-school math competition problems into varying levels of complexity based on annotators' subjective evaluations, offering an incomplete picture of mathematical proficiency. such datasets, while valuable, fall short in encapsulating the full spectrum of mathematical knowledge and overlook the importance of fundamental theory understanding, which is essential for tackling application problems @xcite . those limitations make it difficult to conduct a comprehensive evaluation of llms' math capability (both theory and application) across different levels and disciplines and under a multilingual context. figure 2 : framework of mathbench, we first categorize the mathematical content into four main educational stages and one basic arithmetic stage. then, we extend from these to fill in two more fine-grained levels of knowledge points, forming the final mathbench framework. mathbench, a novel and comprehensive multilingual benchmark meticulously created to evaluate the mathematical capabilities of llms across a diverse range of difficulties, from basic arithmetic to challenging college-level mathematics. math-bench sets itself apart with a unique five-stage taxonomy, mapped to the educational trajectory from primary school through to college. this mechanism is designed to assess llms' mathematical understanding in breadth and depth. the benchmark incorporates carefully curated questions that cover basic theory knowledge and practical applications. this dual focus enables mathbench to probe and interpret the models' capabilities from a foundational standpoint. additionally, mathbench supports bilingual evaluation in both chinese and english, which facilitates a more nuanced and comprehensive assessment of llms' math capabilities, offering a realistic reflection of the global landscape of mathematical knowledge. in this paper, we detail the methodology behind the creation of mathbench, including the hierarchical knowledge system that underpins the dataset, the data collection process, and the criteria for question selection. we hope that mathbench can serve as a valuable resource for researchers and developers seeking to advance the mathematical abilities of llms and to understand the limitations of existing models in solving diverse and complex mathematical problems. mathbench features the following contributions: ## methodology mathbench features a well-crafted difficulty hierarchy and an emphasis on evaluating the theoretical knowledge understanding of llms. sec. 2.1 presents the tiered levels and the corresponding knowledge foundations, explaining the ability taxonomy and design rationale. sec. 2.2 details the collection process and statistics of mathbench. ## related work solving math word problems through automated methods has been a long-standing concern for re-8 searchers. this section summarizes seminal studies and delineates key evaluation datasets proposed for assessing mathematical problem-solving approaches, tracing the field's evolution from its origins to the present day. preliminary mathematical datasets previous works proposed datasets such as alg514 @xcite , singleeq @xcite , and draw-1k (upadhyay and chang, 2017) are primarily concentrated on elementary linear algebraic problems. similarly, datasets like addsub @xcite and singleop @xcite multiarith @xcite are exclusively dedicated to fundamental arithmetic operations: addition, subtraction, multiplication, and division. these datasets are very limited both in the form and content of their assessments, focusing solely on a specific small part of basic mathematics. benchmarks tailored to specific educational tiers some benchmarks are designed based on educational levels. math23k @xcite collects a corpus of real math word problems for elementary school students. while asdiv @xcite expands the textual patterns to encompass most problem types found in elementary mathematics. gsm8k (cobbe et al., 2021) presents a high-quality collection of elementary mathematical word problems that, on average, require multiple steps to solve and provide solutions in natural language annotations. these datasets mostly focus on elementary mathematics and seldom examine college-level knowledge. enriching the diversity of mathematical problem types within benchmarks mathqa @xcite seeks to categorize problems from aqua @xcite into different mathematical domains based on the frequency of mathematical terminology used. mathematics dataset @xcite expands the subject of mathematics and this dataset covers a broader spectrum of mathematics, including arithmetic, algebra, probability, and calculus. math @xcite features a higher level of complexity, comprising problems ranging from arithmetic to calculus, and aims at testing models' capabilities in understanding and solving complex mathematical challenges. while these efforts have enhanced the diversity of the data in certain aspects, they are lacking in diversity in other aspects such as question formulation @xcite . integrating mathematical problems with domain knowledge numglue @xcite not only assesses the ability of models to solve mathematical problems given direct computational expressions, but it also designs multiple tasks to comprehensively evaluate the models' abilities to use other reasoning skills, such as common sense and reading comprehension. lila @xcite is developed through the extension of 20 datasets that cover a broad range of mathematical topics. this dataset exhibits varying degrees of linguistic complexity and features diverse question formats as well as background knowledge requirements. these works inspire us to design more diversified testing scenarios. ## conclusion in summary, mathbench adopts structured approaches to categorize questions by stage and knowledge level. it aims to provide a comprehensive evaluation of llms' mathematical proficiency. by covering a wide range of subject areas and topics across educational stages, mathbench offers a unique resource for researchers and educators interested in advancing the field of mathematical learning and assessment.