It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset (2023)

Abstract: The math word problem (MWP) is a complex task that requires natural language understanding and logical reasoning to extract key knowledge from natural language narratives. Previous studies have provided various MWP datasets but lack diversity in problem types, lexical usage patterns, languages, and annotations for intermediate solutions. To address these limitations, we introduce a new MWP dataset, named DMath (Diverse Math Word Problems), offering a wide range of diversity in problem types, lexical usage patterns, languages, and intermediate solutions. The problems are available in English and Korean and include an expression tree and Python code as intermediate solutions. Through extensive experiments, we demonstrate that the DMath dataset provides a new opportunity to evaluate the capability of large language models, i.e., GPT-4 only achieves about 75% accuracy on the DMath dataset.

## introduction the math word problem (mwp) is a challenging and intriguing task that requires a deep understanding of natural language narratives and logical reasoning @xcite . the narratives typically involve several numbers and unknown variables. also, some problems demand commonsense reasoning. solving these problems requires a comprehensive understanding of world knowledge and the reasoning between mathematical representations within a given context. as the first step to developing mwp models, it is vital to collect a representative subset of mwp to construct mwp benchmark datasets. although many existing studies @xcite @xcite have published various mwp corpora, they mostly lack the result of chatgpt (gpt-3.5-turbo; openai) and gpt-4 (openai, 2023), using a few-shot cot @xcite prompting method, indicate that dmath is the most challenging benchmark. gpt-4 shows about 90% accuracy on three existing datasets and approximately 75% on dmath. with the advent of large language models (llms), they have shown remarkable performances in various reasoning tasks. notably, the adoption of @xcite has yielded an impressive 92% accuracy on the gsm8k dataset @xcite , which is widely used as a representative mwp benchmark. this result has led to the belief that llms demonstrate exceptional proficiency in mathematical reasoning. however, our empirical findings show that the true capability of llms has not been revealed because of the limited mwp benchmarks. as illustrated in figure 1 , gpt-4 achieves more than 90% accuracy on the existing three mwp datasets, but its accuracy drops to approximately 75% on our dataset with multi-aspect diversity. this significant gap motivates us to develop more diverse datasets with greater complexity and challenges in real-world problems than existing ones. dataset language annotation data size cld # eq. templates # total ops mawps (koncel-kedziorski et al., 2016) en expression tree 1,921 0.26 39 4 mathqa (amini et al., 2019) en expression tree 37,259 0.05 6,599 58 asdiv-a (miao et al., 2020) en expression tree 1,218 0.50 19 4 svamp @xcite en expression tree 1,000 0.22 26 4 gsm8k @xcite en natural language 8,792 0.81 -4 math23k @xcite zh expression tree 23,160 -2,187 4 ape210k @xcite 2020 ) to quantify the diversity of lexical usage patterns. the higher, the more diverse. the underlined numbers represent the top-3 values in each column. let en, ko, and zh denote english, korean, and chinese, respectively, and 'ops' denotes operators. in this study, we introduce a multi-aspect diverse mwp dataset, called dmath, which has the following key features: (1) dmath fully covers problem types across five categories outlined in the math education guideline of grade schools in the usa @xcite and @xcite . these categories include arithmetic (ari), comparison (com), correspondence (cor), geometry (geo), and possibility (pos). thus, various mathematical reasoning skills can be evaluated on dmath. (2) dmath consists of about 10,000 problems manually created by 43 human workers covering various lexical usage patterns. this diversity in lexical usage patterns helps evaluate the general performance of mwp models. (3) dmath supports bilingual input languages, i.e., english and korean, to evaluate the effect of input languages for mwp models. (4) we offer the annotation of expression trees and python code as intermediate solutions to evaluate the effect of expression forms in training mwp models. to annotate the expression trees, we adopt 50 operators, some newly designed to handle the operators for list structures. (details are given in @xcite to analyze the performance of various mwp models on our dataset, we employ fine-tuning and prompting approaches. for the fine-tuning method, we employ three representative mwp models with pre-trained language models, i.e., roberta @xcite , gpt-2 @xcite , and codegpt @xcite . for the prompting method, we adopt representative llms, i.e., gpt-3 @xcite , chatgpt (gpt-3.5-turbo; openai), and gpt-4 (openai, 2023), with various reasoning prompting methods, i.e., zero-shot @xcite , zero-shot cot @xcite , few-shot cot @xcite , and pal @xcite . ## dataset formulation the mwp sample comprises a natural narrative (input) and an answer (output). the input includes natural sentences, symbols, and math equations. the output represents a unique numeric or string value. besides, each sample is associated with its question (english) you want to pick 2 fruits out of an apple, a peach, a pear and place them in different fruit baskets. what is the number of cases? question (korean) 사과, 복숭아, 배 중 2개를 뽑아 서로 다른 과일 바구니에 담으려고 합니다. 가능한 경우의 수는 모두 몇 가지입니까? human solution len([apple, peach, pear]) permutation 2 equation [list_sol] apple peach pear [list_eol] [list_len] 2 [perm] code @xmath0.append(a) li.append(b) li.append(c) @xmath1, elem in enumerate(range(d)): @xmath2: example of a math word problem with two languages (english and korean), two expression forms (an equation and its corresponding python code), a human solution, and the answer. more examples of dmath for each category can be found in appendix a. based on these reasoning tasks, we categorize five problem types, i.e., arithmetic (ari) for arith- for detail, a template represents the process of performing an operation. if the order or the type of operators is different, we regard that they are different templates. diversity in language narratives. existing datasets @xcite @xcite are primarily based on monolingual language, i.e., english or chinese. our dataset provides bilingual languages, i.e., english and korean. it is beneficial for validating the effect of languages on llms. diversity in solution forms. to extend the usage, we annotate two expression formats, an expression tree and python code, as illustrated in the math equation has been widely used as an expression tree @xcite . some categories require us to enumerate and select a specific number/string from a list/sequence of numbers/strings. geo also demands extensive knowledge of geometric shapes like triangles, rectangles, and circles. to annotate our collected problems, we introduce 50 operators. (see appendix b.) following are some rules for defining these operators. first, we create operator names to match actual math symbols as closely as possible. (e.g., ## dataset construction while other mwp datasets are typically collected from the web and annotated with relevant equations, the dmath dataset exclusively comprises human-generated problems. the linguistic diversity is higher than others regarding natural languages and expression trees. specifically, we provide a comprehensive overview of collecting, augmenting, annotating, validating, and translating the dmath dataset. collection. instead of curating existing math problems, we ask human workers to create new problems. we recruit 43 undergraduate and graduate students and let them develop new and diverse grade school-level math problems. each worker contributes to creating about 100-200 samples, resulting in a total collection of 4,184 seed samples. these samples were distributed across five math categories, with 960 for arithmetic, 1,092 for comparison, 943 for correspondence, 401 for geometry, and 788 for possibility. augmentation. human workers manually augment seed samples using three policies: paraphrasing, domain word changes, and expanding equation trees (using a few more equations from the original sentence problem or adding other equations). we carefully augment seed samples to minimize linguistic similarities and superficial differences. human workers manually boost 1-3 samples from each seed sample using different expressions. we also verify the similarity between seed and augmented samples using bleu @xcite scores and remove some samples with high similarities. the statistics for bleu score between seeds and augmented problems are 0.0 for min, 0.24 for mean, 0.23 for median, and 0.92 for max. refer to the detailed histogram in appendix c. annotation. we request the worker to annotate its answer and the intermediate solution as an expression tree using pre-defined operators. this expression tree follows the postfix notation and consists of numbers, strings, and operators, as exemplified in we go through these procedures to reduce potential risks to the correctness of the annotated data. for step (1) and step (2), we ensure that the questions assigned to human workers do not overlap so that they do not receive the same questions in each step, which can act as a cross-check. we employ code to check step (3). for step (4), 2-3 human workers are assigned per question. more detailed procedures are as follows. step (1): question-answer validation. after the annotation process, we assign natural language questions to human workers and ask them to solve the provided questions. there are two cases where their answers differ from the annotated answers. the first is when the natural language problem is unsolvable, and the second is when the annotated answer is incorrect. when a worker encounters these errors while solving a problem, the worker corrects the error manually. in the first case, the worker should correct the natural language question. in the second case, the worker should correct the annotated answer. besides, two or more workers cross-check the corrected questions to ensure no errors. step (2): question-expression tree validation. natural language questions that are solvable and correctly annotated with correct answers are given to human workers along with expression trees. the human workers check whether the given natural language question and the expression tree are correctly paired. if not, they modify the expression tree to match the natural language problem. ## experimental setup datasets. we use three datasets to compare with dmath, which are gsm8k @xcite , svamp @xcite . gsm8k is a commonly used dataset for evaluating math reasoning and has high quality. svamp is a variation of the previous dataset (mawps) @xcite , and it allows us to evaluate the robustness of the methods. asdiv is a highly diverse dataset, showing the importance of increasing the lexical diversity of the problem. we did not run experiments on the mathqa @xcite ) dataset because we found that it has incorrect answers, as also discussed in existing studies @xcite @xcite . also, we did not compare dmath with math @xcite since there is a discrepancy in domain knowledge levels. we aimed to tackle math problems at the elementary school level (grades 1-6), while math is designed for the high school level (grades 8-12). see more details in appendix d. baselines. we evaluate the mwp solvers with the fine-tuning method and the prompting method. for the fine-tuning method, we use pre-trained language models. we use two representative models, roberta @xcite and gpt-2 @xcite for the expression tree output and use codegpt @xcite for the python code output. they are fine-tuned with dmath train data. for more information on how the framework works, refer to appendix e. for the prompting method, we use gpt-based models. we use babbage, curie, davinci @xcite , gpt-3.5-turbo (openai), and gpt-4 (openai, 2023) model using api provided by openai. we denote babbage as gpt-3 . besides, we use four prompting methods: zero-shot @xcite , zero-shot framework # params. ## empirical results overall performance results. second, the worst problem categories differ across mwp models. both geo and pos rely on world knowledge, impacting the performance of fine-tuning models, i.e., roberta large and gpt-2 large , differently. fine-tuned models excel at pos due to condensed operations but struggle with expression tree-based problems in geo. conversely, prompting models, i.e., chatgpt and gpt2023 ) mean natural language prompts. we set few-shot cot @xcite as nl prompts method and pal @xcite as python code prompt. we choose roberta base @xcite as the comparison model for codegpt sm all @xcite because it has the most similar parameter sizes. third, the prompting methods in llms also incur distinct results. chatgpt and gpt-4 show the lowest performance for the zero-shot prompting. they can be significantly improved by using cot prompting. however, when model sizes are too small, the impact of few-shot cot prompting is insignificant, implying that model capacity is an important factor in prompting-based models. we further analyze these results by problem type. for ari types, 8.97% of problems are correct using only natural language prompts and 16.44% are correct using only python code prompts. for com, the corresponding percentages are 29.34% and 14.07%, respectively. ari and com show the largest performance differences among the five categories between natural language prompt and python code prompt. python code prompts perform better on ari types because llm often needs correction in its calculations. the higher accuracy of natural language prompts in com types can be attributed to the concise language expression of large and small relationships between two objects, compared to python code. lastly, the fine-tuning and prompting methods show similar tendencies in model performance. this tendency indicates that various forms of expression have their preferred categories. to elaborate, when we state that form of expression a outperforms form b, it does not imply that every type within a uniformly surpasses every type within b. rather, it suggests that certain types within a outperform b, while some types in b excel over a. the length of the output sequence emerges as the most significant impact. in detail, for ari and pos types, codegpt sm all using python code performs better, and for com, cor, and geo types, roberta base model using expression tree performs better. the average token length and std. for cor types with expression tree are (6.61, 4.13), which is lower than the pos type of (10.81, 4.51). that of pos type with python code is (129.54, 57.55), which is lower than cor type of (147.61, 205.04). in common, geo types seem to have lower token lengths but lower performance due to the lack of domain knowledge of pre-trained models. comparison on different datasets. instead of intentionally inflating the difficulty to undermine model performance, we have noted significant performance differences between models on our dataset. specifically, dmath exhibits average deviations of 13.07% and 15.35% in zero-shot cot and few-shot cot prompting methods, respectively, on the gpt-4 model. this suggests that our dataset contains problems that can be incrementally solved as model capacity improves. the problems are challenging yet reasonable and tailored to grade school levels. in brief, our dataset pinpoints areas few-shot selection. the few-shot prompt is given dmath and other datasets (gsm8k, svamp, and asdiv) differently. other datasets used the example of @xcite as in previous studies. unlike them, dmath is a mix of different mathematical categories. we consider evaluating dmath with the same prompt as other arithmetic mathematical reasoning datasets to be unfair because there are several problem types in dmath. if we give few-shot examples of only arithmetic type, then the model will not do well except arithmetic. therefore, various prompt selection methods were taken, and the highest-performance prompt method, one category random method was used as the final few-shot prompt examples in this paper. (see more details in appendix f.) we validate the best-performing prompt method by comparing five few-shot cot prompts. model prompt method language en ko chatgpt few-shot cot 60.99 46.03 gpt-4 75.18 65.37 qualitative analysis of gpt-4 failure cases. we conduct a qualitative analysis on failure cases of gpt-4 with few-shot cot. (see several examples in appendix g.) for ari, com, and geo examples, as the length of the reasoning path increases, gpt-4 generates incorrect answers more frequently. for cor, the problems about the number of digits in a number mostly incur incorrect answers. for geo, gpt-4 failed to elicit certain domain knowledge, e.g., the rectangle that can be made to have the largest area given a perimeter is a square. for pos, we observe gpt-4 found difficulties in solving problems related to the number of cases. following studies like @xcite @xcite enriched this further, but they were limited by their reliance on multiple choice and a lack of diverse lexical usage patterns. recent works @xcite targeted a broader diversity, but more enhancement is needed. our dataset addresses this by providing diversity in problem types, lexical usage patterns, languages, and expression forms. math word problem solving models. early mwp models used symbolic approaches @xcite , but their domain specificity limited their use. later, template-based and statistical methods @xcite @xcite @xcite improved performance but faced challenges with diverse datasets @xcite . deep neural networks became popular, with rnn-based models @xcite , tree generation models @xcite @xcite , and relation extraction models @xcite . recent models leverage pre-trained language models like bert @xcite to enhance performance @xcite @xcite . large language models have also shown significant performance improvements with techniques like step-by-step reasoning @xcite , calculation verifiers @xcite , and various prompting methods @xcite @xcite . recent studies @xcite @xcite indicate that large language models can improve reasoning capabilities autonomously without supervised data, surpassing current techniques in accuracy. ## conclusion this paper proposed dmath, a collection of 10k high-quality grade school-level math word problems. it mainly pursued multi-aspect diversity for problem types, lexical usage patterns, languages, and expression forms. we evaluated various mwp models on dmath and other datasets. we observed that the logical reasoning capability of the mwp model was mainly focused on arithmetic reasoning. because of various features, our dataset is more beneficial for inspecting the diverse reasoning abilities of models. our dataset can also help evaluate mwp models for diverse aspects.