C oncept M ath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models (2024)

Abstract: This paper introduces ConceptMath, a bilingual (English and Chinese), fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models (LLMs). Unlike traditional benchmarks that evaluate general mathematical reasoning with an average accuracy, ConceptMath systemically organizes math problems under a hierarchy of math concepts, so that mathematical reasoning can be evaluated at different granularity with concept-wise accuracies. Based on our ConcepthMath, we then evaluate a broad range of LLMs, and we observe existing LLMs, though achieving high average accuracies on traditional benchmarks, exhibit significant performance variations across different math concepts and may even fail catastrophically on the most basic ones. Besides, we also introduce an efficient fine-tuning strategy to enhance the weaknesses of existing LLMs. Finally, we hope ConceptMath could guide the developers to understand the fine-grained mathematical abilities of their models and facilitate the growth of foundation models. Code is available at https://github.com/conceptmath/conceptmath.

## introduction mathematical reasoning is a crucial capability for large language models (llms). recent advancements in llms, including anthropic @xcite , @xcite , and llama @xcite , have demonstrated impressive mathematical reasoning on existing benchmarks with high average accuracies on datasets like gsm8k @xcite . although these benchmarks are able to measure the overall mathematical reasoning capabilities of llms on average, they fail to probe the fine-grained failure modes of mathematical reasoning on specific * first three authors contributed equally. # corresponding author: jiaheng liu. figure 1 : the concept-wise accuracies of llama2-13b and the fine-tuned version based on our efficient fine-tuning method (i.e., llama2-ft). mathematical concepts. for example, fig. 1 shows that the performance of llama2-13b varies significantly across different concepts and fails on simple concepts like rational number and cylinders. it is crucial to know these specific failure modes of the language model, especially in some practical applications where we need to focus on specific mathematical abilities. for example, for financial analysts, calculation, measurement, and statistics are the concepts of most interest while others like geometry are not as important. moreover, the mathematics system, by its nature, is more fine-grained than holistic. it is typically organized into distinct math concepts foot_0 , and humans develop comprehensive mathematical capabilities through a concept-by-concept, curriculum-based learning process @xcite . these issues underscore the core motivation of this paper: the need for a finegrained benchmark that evaluates concept-wise mathematical reasoning of llms. therefore, first, we introduce conceptmath, the first bilingual (english and chinese), concept-wise benchmark for measuring mathematical reasoning. conceptmath gathers math concepts from four educational systems, resulting in four distinct mathematical concept systems: english elementary, english middle, chinese elementary, and chinese middle foot_1 . each of these concept systems organizes around 50 atomic math concepts under a threelevel hierarchy and each concept includes approximately 20 mathematical problems. overall, con-ceptmath comprises a total of 4011 math word problems across 214 math concepts, and fig. 2 shows the diagram overview of conceptmath. second, based on our conceptmath, we perform extensive experiments to assess the mathematical reasoning of existing advanced llms, including 2 close-sourced llms and 17 opensourced llms. these evaluations were performed in zero-shot, chain-of-thought (cot), and fewshot settings. to our surprise, even though most of the evaluated llms claim to achieve high average accuracies on traditional mathematical benchmarks (e.g., @xcite ), they fail catastrophically across a wide spectrum of mathematical concepts. third, to make targeted improvements on underperformed math concepts, we propose an efficient fine-tuning strategy by first training a concept classifier and then crawling a set of samples from a large open-sourced math dataset @xcite for further llms finetuning. in fig. 1 , for llama2-ft, we observe that the results of these weaknesses improved a lot after using the efficient fine-tuning method. in summary, our contributions are as follows: ## conceptmath conceptmath is the first bilingual, concept-wise benchmark for measuring mathematical reasoning. in this section, we describe the design principle, dataset collection process, dataset statistics and an efficient fine-tuning strategy to enhance the weaknesses identified by our conceptmath. ## experiments in this section, we perform extensive experiments to demonstrate the effect of our conceptmath. ## related work large language models for mathematics. large language models (llms) such as gpt-3.5 and gpt-4 have exhibited promising capabilities in complex tasks @xcite @xcite @xcite . however, the proficiency of open-source alternatives like llama @xcite and llama2 @xcite remains notably inferior on these datasets, particularly in handling ## conclusion we introduce a new bilingual concept-wise math reasoning dataset called conceptmath to assess models across a diverse set of concepts. first, con-ceptmath covers more than 200 concepts across elementary and middle schools for mainstream english and chinese systems. second, we extensively evaluate existing llms by three prompting methods, which can guide further improvements for these llms on mathematical abilities. third, we analyze the contamination rates, error cases and provide a simple and efficient fine-tuning strategy to enhance the weaknesses.