Are NLP Models really able to Solve Simple Math Word Problems? (2021)

Abstract: The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered “solved” with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

## introduction a math word problem (mwp) consists of a short natural language narrative describing a state of the world and poses a question about some unknown quantities (see mwps come in many varieties. among the simplest are the one-unknown arithmetic word problems where the output is a mathematical expression involving numbers and one or more arithmetic operators (+, -, * , /). problems in tables 1 and 6 are of this type. more complex mwps may have systems of equations as output or involve other operators or may involve more advanced topics and specialized knowledge. recently, researchers have started focusing on solving such mwps, e.g. multiple-unknown linear word problems @xcite , geometry @xcite and probability @xcite , believing that existing work can handle one-unknown arithmetic mwps well @xcite . in this paper, we question the capabilities of the state-of-the-art (sota) methods to robustly solve even the simplest of mwps suggesting that the above belief is not well-founded. in this paper, we provide concrete evidence to show that existing methods use shallow heuristics to solve a majority of word problems in the benchmark datasets. we find that existing models are able to achieve reasonably high accuracy on mwps from which the question text has been removed leaving only the narrative describing the state of the world. this indicates that the models can rely on superficial patterns present in the narrative of the mwp and achieve high accuracy without even looking at the question. in addition, we show that a model without word-order information (i.e., the model treats the mwp as a bag-of-words) can also solve the majority of mwps in benchmark datasets. the presence of these issues in existing benchmarks makes them unreliable for measuring the performance of models. hence, we create a challenge set called svamp (simple variations on arithmetic math word problems; pronounced swamp) of one-unknown arithmetic word problems with grade level up to 4 by applying simple variations over word problems in an existing dataset (see below, we summarize the two broad contributions of our paper. ## deficiencies in existing datasets here we describe the experiments that show that there are important deficiencies in mawps and asdiv-a. ## svamp the efficacy of existing models on benchmark datasets has led to a shift in the focus of researchers towards more difficult mwps. we claim that this efficacy on benchmarks is misleading and sota mwp solvers are unable to solve even elementary level one-unknown mwps. to this end, we create a challenge set named svamp containing simple one-unknown arithmetic word problems of grade level up to 4. the examples in svamp test a model across different aspects of solving word problems. for instance, a model needs to be sensitive to questions and possess certain reasoning abilities to correctly solve the examples in our challenge set. svamp is similar to existing datasets of the same level in terms of scope and difficulty for humans, but is less susceptible to being solved by models relying on superficial patterns. our work differs from adversarial data collection methods such as adversarial nli @xcite in that these methods create examples depending on the failure of a particular model while we create examples without referring to any specific model. inspired by the notion of normative evaluation (linzen, 2020), our goal is to create a dataset of simple problems that any system designed to solve mwps should be expected to solve. we create new problems by applying certain variations to existing problems, similar to the work of @xcite . however, unlike their work, our variations do not check for linguistic capabilities. rather, the choice of our variations is motivated by the experiments in section 4 as well as certain simple capabilities that any mwp solver must possess. ## final remarks going back to the original question, are existing nlp models able to solve elementary math word problems? this paper gives a negative answer. we have empirically shown that the benchmark english mwp datasets suffer from artifacts making them unreliable to gauge the performance of mwp solvers: we demonstrated that the majority of problems in the existing datasets can be solved by simple heuristics even without word-order information or the question text. the performance of the existing models in our proposed challenge dataset also highlights their limitations in solving simple elementary level word problems. we hope that our challenge set svamp, containing elementary level mwps, will enable more robust evaluation of methods. we believe that methods proposed in the future that make genuine advances in solving the task rather than relying on simple heuristics will perform well on svamp despite being trained on other datasets such as asdiv-a and mawps. in recent years, the focus of the community has shifted towards solving more difficult mwps such as non-linear equations and word problems with multiple unknown variables. we demonstrated that the capability of existing models to solve simple one-unknown arithmetic word problems is overestimated. we believe that developing more robust methods for solving elementary mwps remains a significant open problem.