M ath F ish: Evaluating Language Model Math Reasoning via Grounding in Educational Curricula (2024)

Abstract: To ensure that math curriculum is grade-appropriate and aligns with critical skills or concepts in accordance with educational standards, pedagogical experts can spend months carefully reviewing published math problems. Drawing inspiration from this process, our work presents a novel angle for evaluating language models’ (LMs) mathematical abilities, by investigating whether they can discern skills and concepts enabled by math content. We contribute two datasets: one consisting of 385 fine-grained descriptions of K-12 math skills and concepts, or *standards*, from Achieve the Core (*ATC*), and another of 9.9K math problems labeled with these standards (*MathFish*). We develop two tasks for evaluating LMs’ abilities to assess math problems: (1) verifying whether a problem aligns with a given standard, and (2) tagging a problem with all aligned standards. Working with experienced teachers, we find that LMs struggle to tag and verify standards linked to problems, and instead predict labels that are close to ground truth, but differ in subtle ways. We also show that LMs often generate problems that do not fully align with standards described in prompts, suggesting the need for careful scrutiny on use cases involving LMs for generating curricular materials. Finally, we categorize problems in GSM8k using math standards, allowing us to better understand why some problems are more difficult to solve for models than others.

## introduction when assessing mathematical reasoning in large language models (lms), a common approach is to test their problem solving abilities. math is a popular domain for model evaluation @xcite @xcite , as problem instances can be designed to target specific abilities. however, many datasets contain only coarse-grained information on what skills each problem assesses, such as general arithmetic or operation types @xcite . in practice, curricular experts' categorizations of math are finegrained, mapping materials to specific skills, such as multiplication procedures for fractions, or concepts, such as area and volume. our work bridges this gap and examines a novel angle for evaluating lms' mathematical understanding. we ask, how well can models identify specific skills and concepts that students learn or practice when completing math problems? first, we contribute english datasets of 9.9k human-written math problems (mathfish ) scaffolded by 385 k-12 u.s. curriculum standards in achieve the core (atc ). these standards are informed by human learning progressions, and commonly used in real-world reviews of math content. in education, materials have focused alignment with a standard if they enable students to learn the full intent of concepts/skills described by that standard. 1 identifying alignment can thus inform educators whether a set of materials adequately targets core learning goals for students. second, we provide a fine-grained assessment of lms' abilities to reason about math pedagogy, skills, and concepts, by asking them to recognize whether a standard aligns with a given problem ( §4, §5). we experiment with two task formats: one in which we verify whether a single standard aligns with a problem, and another in which we tag each problem with any standards. our experiments demonstrate that models achieve reasonable accuracy, but are not yet at expert-level performance across both task formats. third, to further illustrate the utility of these task formats, we apply the best-performing models and prompting approaches on two case studies. in one, we work with k-12 math teachers with curriculum review experience to apply verification on lmgenerated problems ( §4.2). we find that a gpt-4 verifier tends to overestimate full standards alignment of generated problems compared to teachers. in the other, we tag gsm8k @xcite , a widely used grade school math dataset, with standards ( §5.2). we find that gsm8k only covers around a third of all k-12 standards, and that lms struggle more to solve problems tagged with higher grade levels' standards. as lms are deployed in more real-world use cases, it is increasingly important to evaluate models with potential users. throughout this paper, we work with curriculum specialists and teachers to center their voices when evaluating lm capabilities. these educational experts informed us that identifying math standards in curricular materials is time-intensive, and reviewing a set of published materials can take 6-8 months to complete. this motivates us to investigate whether models can support reviewers by combing through subtle differences across standards. altogether, we hope that these datasets, along with the tooling we contribute for transforming data into task formats for lms, can facilitate a more granular understanding of lm reasoning around skills and concepts in math content. lms for improving math education. lms have increasing potential for improving math education through applications like automated feedback to educators @xcite @xcite , automated tutoring @xcite , and lesson planning @xcite . evaluating models with education domain experts before real-world deployment is crucial to best align with educational needs and mitigate potential harms @xcite @xcite . in our setting, we work with domain experts in a manner that centers real-world educational needs ( §4.2). ## verifying standards alignment this section focuses on our first task format, where models verify the alignment of individual standards against each problem. we first examine how models perform on mathfish problems ( §4.1), and then apply our best-performing verifier on generated problems ( §4.2). ## tagging problems with standards as we introduced in §3, tagging navigates a hierarchical decision tree, where each branch presents models a problem and a list of d, c, or s descriptions. we shuffle the ordering of options in each level of the tree to avoid position bias @xcite . the d level consists of 12 options, while the c level and s level each have on average 13.7 options and 2.6 options, respectively. we also give models the option to also respond with "none" at each level, indicating that none of the listed options apply to an example problem. prompt templates and our rationale behind their design can be found in appendix d. in this section, we discuss experiments we run on mathfish problems ( §5.1) and apply our best tagger on gsm8k ( §5.2). ## conclusion our work investigates how well lms can identify fine-grained skills and concepts that students learn when completing math problems. we contribute a dataset of 385 expert-designed mathematical standards layered with natural language descriptions and organizational metadata (atc ), a dataset of 9.9k standards-labeled problems drawn from reputable curricula (mathfish ), and annotations of problem and standard pairs by experienced teachers. in addition, we provide tooling to transform data into tasks that assess lms' abilities to reason about mathematical language and concepts/skills. we find that though lms still do not reach expertlevel performance on verification, tagging, and generation tasks, they show promise and utility for assisting curriculum review and disaggregating math problem solving benchmarks.