CHAMP : A Competition-level Dataset for Fine-Grained Analyses of LLM s’ Mathematical Reasoning Capabilities (2024)

Abstract: Recent large language models (LLMs) have shown indications of mathematical reasoning ability on challenging competition-level problems, especially with self-generated verbalizations of intermediate reasoning steps (i.e., chain-of-thought prompting). However, current evaluations mainly focus on the end-to-end final answer correctness, and it is unclear whether LLMs can make use of helpful side information such as problem-specific hints. In this paper, we propose a challenging benchmark dataset for enabling such analyses. The Concept and Hint-Annotated Math Problems (CHAMP) consists of high school math competition problems, annotated with concepts, or general math facts, and hints, or problem-specific tricks. These annotations allow us to explore the effects of additional information, such as relevant hints, misleading concepts, or related problems. This benchmark is difficult, with the best model only scoring 58.1% in standard settings. With concepts and hints, performance sometimes improves, indicating that some models can make use of such side information. Furthermore, we annotate model-generated solutions for their correctness. Using this corpus, we find that models often arrive at the correct final answer through wrong reasoning steps. In addition, we test whether models are able to verify these solutions, and find that most models struggle.

## introduction recent large language models (llms) have demonstrated impressive performance on many tasks that previously required specialized models or were thought to be out of reach of conventional neural networks. one such capability is mathematical reasoning: llms can often solve simple math problems and make reasonable attempts at challenging, competition-level problems. in addition to model scaling @xcite , there are two key factors behind the progress: sophisticated prompting for each problem, we write the relevant and helpful concepts (e.g., a 3 ± b 3 = (a + b)(a 2 ± ab + b 2 )), and hints (e.g, express x 3 as the product of two factors involving y). b : in our experiments, to investigate a model's ability to understand and use the additional c & h information, we design 17 prompts to evaluate ten models: gpt-3.5 / 4 / 4 turbo, palm 2 medium, llama 2 7b / 70b, llama 3 8b / 70b, mistral 7b and mixtral 8x22b. c : for each problem, we manually judge two model-generated solutions on their correctness, and further annotate the first wrong step of the reasoning (red highlights), if present. d : this corpus thus serves as a novel dataset for benchmarking and evaluating the solution verification ability of llms. using this dataset, we design 17 different prompts, and evaluate various proprietary and opensource models including gpt-3.5 / 4 / 4 turbo (openai, 2023), palm 2 medium @xcite foot_0 , llama 2 7b / 70b @xcite , llama 3 8b / 70b (ai, 2024), mistral 7b (jiang et al., 2023) and mixtral 8x22b (team, 2024). while we observe a diverse range of behaviors across different models and prompts, we find the accuracy of the best setting to be 67.0% only (measured by the final answer correctness, ignoring any possible errors in intermediate reasoning steps), and gains from the additional concept and hint information vary across models. the results indicate a large room for improvement with competitionlevel math for llms, and moreover highlight the utility of champ for developing and benchmarking future models. for each problem, we further analyze solutions generated by these models by manually annotating the first wrong step in the reasoning process, such as an arithmetic error or question misunderstanding, or validating that the solution is fully correct. this annotation serves two purposes. first, it concretely identifies how much the final answer accuracy (which is the predominant practice for math problem evaluations) over-estimates a model's ability to generate fully correct solutions: indeed, we find that in many instances, a model gets the answer "right" despite generating wrong reasoning steps, indicating that these models are potentially relying on shortcut heuristics. second, we can evaluate the verification ability of any model, i.e., how well it can reason about a given solution and identify any errors, where we find that most models struggle. the above evaluations suggest key deficiencies in current llms and highlight the value of these annotations as an additional benchmarking resource. in summary, we evaluate four model capabilities using our dataset: generating correct final answer, generating correct full solution, using the contextual c & h information, and verifying a given solution. our findings uncover new strengths and limitations of current models, and give directions for future work on improving them. a key distinction of champ compared to other math datasets is the information associated with each problem. in addition to the problem text and its solution-which are common components of such datasets-we annotate relevant concepts and hints and further label them on each solution step, with problems relating to each other via common math concepts (e.g., fermat's little theorem). in this way, champ enables fine-grained evaluations of mathematical problem solving abilities of llms that are not possible with other datasets, for example allowing for different types of additional information to be made available in the context through prompting. we observe that many techniques seek to improve an llm's mathematical reasoning ability, such as encouraging chain-of-thought generation @xcite , selecting a final result from multiple sampled outputs @xcite , and using external tools such as a calculator or python interpreter @xcite to eliminate arithmetic errors. these directions can be combined with our experimental setup. ## the champ dataset this section describes the dataset structure and construction. due to the high level of math expertise required, the dataset curation is carried out exclusively by the paper authors. problems. we select problems from the book problem-solving strategies by @xcite , a classic piece of material for high-school math com- problem id: p_inequality_36 problem: for non-negative a, b, c, d, what is the smallest value of (a + c)(b + d) -√ ab -√ cd -1? concepts and hints: h1. compare (a + c)(b + d) with √ ab + √ cd by squaring both terms. c1. (@xmath0. c2. for non-negative x, y, we have (x + y) ≥ 2 √ xy and (x + y)/2 ≥ √ xy, with equality if and only if @xmath1. c3. for non-negative x, y, √ @xmath2. answer: -1 solution step: 1. we have (a + c)(b for each problem, we manually verify and write out the full detailed step-wise solution in natural language, as the solution manual often skips steps and occasionally contains typographical errors. we also provide an explicit final answer that can be checked against. ## experiment 2: solution verification the first wrong step (fws) annotation allows for evaluating whether llms can read a solution to a problem and verify its correctness, which is a much more difficult task than comparing the final answer against the ground-truth answer. in this set of experiments, we study how well llms can judge model-generated solutions, as well as the (error-free) ground truth solution. ## discussion this paper presents champ, the concept and hint-annotated math problems dataset, along with annotations of logical correctness of model-generated solutions to each problem. the unique construction of champ enables previously under-explored studies regarding context-related reasoning abilities as well as verification abilities of llms. we investigate the mathematical reasoning abilities of 10 proprietary and open-source llms of various sizes and release dates through champ. even the best models are far from perfect, and many models are unable to incorporate useful concepts and problem-specific hints. even when the final answer accuracy is high, a closer inspection of the reasoning steps reveals that many models may be accidentally arriving at the correct answer. furthermore, all models struggle at solution verification, indicating that these models often generate but do not understand their solutions, similar to some of the findings of recent work @xcite . these results advocate for a more fine-grained and multi-faceted investigation of llms' mathematical reasoning capabilities. ## limitations while champ reveals intriguing behaviors of different models, there are several limitations. for one, we relied on extensive manual annotation for high-quality labeling, and hence the dataset currently contains only 270 problems. there is also the risk of dataset contamination; i.e., the models we evaluate may have been trained on the original versions of the problems from champ. however, to mitigate this we rewrote the problems to fit the design of the dataset, minimizing the impact of memorization. finally, while we rely on automatic evaluation for final answer accuracy to enable scalable evaluation, this may not be perfect. our manual grading results suggest, though, that gpt-4's automatic grading has a high accuracy of 97%. before after prove that, for a, b, @xmath3, 3 √ @xmath4, b, c, what is the smallest value of √ ab + bc + ac/ 3 √ abc? (answer: √ 3) the product of three positive reals is 1. their sum is greater than the sum of their reciprocals. prove that exactly one of these numbers @xmath5. the product of three positive real numbers is 1, and their sum is greater than the sum of their reciprocals. how many of them can be greater than 1? (answer: 1)