A Diverse Corpus for Evaluating and Developing E nglish Math Word Problem Solvers (2020)

Abstract: We present ASDiv (Academia Sinica Diverse MWP Dataset), a diverse (in terms of both language patterns and problem types) English math word problem (MWP) corpus for evaluating the capability of various MWP solvers. Existing MWP corpora for studying AI progress remain limited either in language usage patterns or in problem types. We thus present a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem types taught in elementary school. Each MWP is annotated with its problem type and grade level (for indicating the level of difficulty). Furthermore, we propose a metric to measure the lexicon usage diversity of a given MWP corpus, and demonstrate that ASDiv is more diverse than existing corpora. Experiments show that our proposed corpus reflects the true capability of MWP solvers more faithfully.

## introduction human math/science tests have been considered more suitable for evaluating ai progress than the turing test @xcite . among them, math word problems (mwps) are frequently chosen to study natural language understanding and simulate human problem solving @xcite @xcite , because the answer is not a span within the given problem text that can be directly extracted. current mwp corpora can be classified into four categories: (1) the number word problem corpus @xcite , which contains number word problems only; (2) the arithmetic word problem corpora @xcite , which involve the four basic arithmetic operations ## problem type a problem type (pt) indicates a crucial math operation pattern for solving an mwp. as mwps of the same problem type share a similar pattern (in language usages, logic representation, or inferences), they thus indicate stereotypical math operation patterns that could be adopted to solve an mwp @xcite . in asdiv, each mwp is annotated with a specific pt taught at elementary schools. some examples of selected pts are shown in ## asdiv math word problem corpus this corpus was designed based on the following guidelines: (1) the corpus should be as diverse as possible in terms of lexicon usage so that the answer is less likely to be obtained via mechanical/statistical pattern matching without understanding the content. (2) the corpus should cover most pts found in primary school so that it can approximate real human tests. (3) the corpus should be annotated with sufficient information so that it can be used not only to assess the capability of various systems but also to facilitate system development. ## experiments to study the correlation between cld and system performance, we selected three sota mwp solvers to conduct the experiments: two based on statistical models, lca++ @xcite and unitdep @xcite ; and one using a neural network which adopts two-layer gate-feedforward networks for a goal-driven tree-structured approach (gts) @xcite . since the selected mwp solvers solve only arithmetic mwps, we first collected 4,117 mwps from mathqa to construct a subset that its associated formulas satisfy the following two conditions: (1) they involve only arithmetic operations; and (2) they contain neither external constants (which would necessitate external domain knowledge to solve the problem and is out of the scope of this work) nor reused operands (which rarely occur and would complicate the solution procedure). we filtered out inconsistent problems (specified in section 3.2) and termed the remaining 3,000 mwps as mathqa-c dataset (-c for consistent) to evaluate the performance. similarly, we extracted a subset of 1,218 mwps that involve only arithmetic operations (and also satisfy the constraints mentioned above) from asdiv, and termed this the asdiv-a dataset (-a for arithmetic). the clds for mathqa-c and asdiv-a were found to be 0.08 and 0.50, respectively. also, @xmath0. afterwards, we tested the solvers against three mwp corpora: mathqa-c, asdiv-a, and asdiv. mathqa-c is reported with 5-fold cross-validation accuracy. for asdiv-a and asdiv, we randomly split the mwps of each pt into five nearly equallysized subsets, and report the 5-fold cross-validation accuracy. for gts system, we repeated the experiment 5 times and obtained the averaged answer accuracy. ## conclusion and future work we present an mwp corpus which not only is highly diverse in terms of lexicon usage but also covers most problem types taught in elementary school. each mwp is annotated with the corresponding problem type, equation, and grade level, which are useful for machine learning and assessing the difficulty level of each mwp. we also propose a metric to measure the diversity of lexicon usage of a given corpus. in terms of this metric, we show that in comparison with those corpora widely adopted to compare systems, ours is more suitable for assessing the real performance of an mwp solver. last, we conduct experiments to show that a low-diverse mwp corpora will exaggerate the true performance of sota systems (we are still far behind human-level performance), and that grade level is a useful index for indicating the difficulty of an mwp.