MATHWELL : Generating Educational Math Word Problems Using Teacher Annotations (2024)

Abstract: Math word problems are critical K-8 educational tools, but writing them is time consuming and requires extensive expertise. To be educational, problems must be solvable, have accurate answers, and, most importantly, be educationally appropriate. We propose that language models have potential to support K-8 math education by automatically generating word problems. However, evaluating educational appropriateness is hard to quantify. We fill this gap by having teachers evaluate problems generated by LLMs, who find existing models and data often fail to be educationally appropriate. We then explore automatically generating *educational* word problems, ultimately using our expert annotations to finetune a 70B language model. Our model, MATHWELL, is the first K-8 word problem generator targeted at educational appropriateness. Further expert studies find MATHWELL generates problems far more solvable, accurate, and appropriate than public models. MATHWELL also matches GPT-4’s problem quality while attaining more appropriate reading levels for K-8 students and avoiding generating harmful questions.

## introduction math word problems (mwp) are natural language math questions paired with numerical answers and are critical tools for k-12 math education @xcite @xcite . traditionally, teachers hand write mwps customized to their students' interests, which has been shown to improve students' learning, test performance, and general interest in math @xcite @xcite . however, teachers' time pressure is often so severe that they must use boilerplate question sets. we propose that large language models (llms) are poised to enhance math education by generating customized, 1 https://github.com/bryanchrist/mathwell figure 1 : generating educational math word problems with language models. to be educational, problems must simultaneously be solvable, accurate, and educationally appropriate. we aim to explore and enhance llms' capacity to generate educational mwps. there have been many works using traditional nlp methods to generate mwps @xcite @xcite @xcite @xcite @xcite . however, they all rely on input reference mwps or equations, ultimately largely rephrasing training data. to use them, teachers would need to manually curate problem sets, which is time consuming. and being traditional approaches, these methods are also not promptable, limiting personalization and mwp diversity. ## related work math qa datasets there are many datasets for training and evaluating llms on grade school math reasoning. several popular datasets include gsm8k @xcite , numglue @xcite , gsm-hard @xcite , as-div @xcite and svamp @xcite . recent work has also developed new grade school math evaluation datasets or large training datasets with different solution rationales for existing datasets, particularly gsm8k, including both human written @xcite and synthetic @xcite @xcite @xcite data. however, because existing datasets are designed primarily for training and evaluating grade school math reasoning, they are not aligned with training educational grade school mwp generators. grade school mwp training data for educationally appropriate generators must contain high-quality grammar, be written at an appropriate mathematical difficulty and reading level for k-8 students, include questions similar to those students would encounter in the classroom, and be comprised of questions that are appropriate for an education setting. further, to encourage effective solution generation, we use program of thought (pot) solutions written as python functions (see appendix c.7 for details), as pot outperforms chain of thought (cot; @xcite ) for open-ended questions, which are the type of problems we seek to generate @xcite @xcite . we consider high-quality grammar and similarity to word problems students encounter in the classroom baseline criteria any potentially relevant training data must possess. existing datasets that contain these baseline characteristics and are aligned with one or more of the other four criteria are gsm-hard @xcite , gsm8k @xcite , mathinstruct gsm8k @xcite , asdiv @xcite , and svamp @xcite , although none of these datasets contain all four criteria. ## methods math word problems (mwps) are natural language questions paired with numerical answers. we study generating these pairs with llms, prompting them to write new problems without augmenting handpicked equations or reference problems. we evaluate these problems based on both the human and automatic evaluation criteria defined below. human evaluation criteria for modelgenerated mwps to be educational, they must meet three criteria: 1) solvability, where questions are possible to solve and have one correct answer. 2) accuracy, where generated answers must be correct. 3) appropriateness, where mwps should be questions teachers would feel comfortable giving to k-8 students. generally, appropriate mwps should make sense, avoid grammatical errors or conflicting information, and be about topics and include mathematical operations appropriate for k-8 students in a school setting. because this is hard to define, we emphasize real teacher evaluations of generated mwps. ideal mwps are accurate, solvable, and appropriate, thereby meeting all criteria. we refer to such problems as mac. automatic evaluation criteria reading level automatically assesses if mwps are written appropriately. like @xcite , we use @xcite to evaluate reading level. fkgl is a function of the total words, sentences, and syllables in a text, and the score represents a u.s. grade level (aggarwal; @xcite . thus, a fkgl score above 8 for a mwp would be considered inappropriate for an educational k-8 mwp generator. lower reading levels for mwps are preferred because high reading levels are known to harm student performance, especially for students who are already struggling @xcite . negative fkgl scores are possible and denote text that is easy to read due to having short words and sentences. we also calculate each mwp's average token length. longer token length can be a proxy for mwp complexity, as longer questions tend to include more mathematical operations. we follow prior works in using two automatic metrics to compare the quality of our synthetic mwps to human-written mwps: perplexity (ppl) and bertscore @xcite . lower ppl implies better outputs and we calculate ppl using llama-2 (70b). to show that ppl is not biased towards llama-2 outputs, we report gpt-2 ppl in ## evaluating mathwell next, we thoroughly evaluate mathwell's generated mwps using both human and automatic evaluations. we compare 250 randomly-selected mathwell outputs to those of open and closed source models. for closed-source models, we evaluate gpt-3.5 turbo and gpt-4 turbo. gpt-3.5 represents a closed-source model with similar capability to llama-2 (70b) @xcite , while gpt-4 is a much more capable model @xcite . for open-source models, we evaluate llemma (34b) @xcite and mammoth (70b) @xcite , which have similar capabilities to llama-2 (70b) but are math specific. llama-2 serves as a baseline to ensure task-specific finetuning improves performance. we prompt each model using examples from egsm and ask it to create a question/answer pair on a random topic k-8 students are interested in (see appendix f for a full list of topics). our 250 evaluation sample for each model is roughly 2.5-5 times larger than the human evaluation samples used in other mwp generator studies @xcite @xcite @xcite @xcite . ## controllability being able to control the math operations and topics present in mwps generated by mathwell would maximize the model's educational applicability by allowing teachers to generate mwps that target specific concepts they are teaching. to test mathwell's controllability, we prompted the model to generate questions involving only one of the operations from the list of addition, subtraction, multiplication, and division. each prompt included 8-shot in-context examples and we generated 100 questions for each operation, totaling 400 generated questions. see appendix a for the full prompt. we then measured the percent of generated questions that successfully include only the intended operator. we compare this to the distribution of questions produced by the unprompted model that only contain one of the operators. as shown in ## conclusions we explore educational reference-free k-8 mwp generation and create the first dataset to train models for this task and the only one with teacher annotations. we demonstrate the quality of this dataset by using it to finetune mathwell, the first reference-free educational mwp generator. our evaluations show that mathwell outperforms other open-source llms at reference-free mwp generation, matching the performance of gpt-4 while generating questions with a more appropriate reading level and that do not contain harmful inappropriate content. further, we show that our dataset, egsm, is comparable to existing math qa data while containing questions that are more educationally appropriate. these findings suggest that reference-free mwp generation is a feasible and practical alternative to traditional reference-dependent generators. future research should train reference-free mwp generators to create questions aligned with specific grade levels and develop automated classification methods to reduce human annotation costs.