Adversarial Examples for Evaluating Math Word Problem Solvers (2021)

Abstract: Standard accuracy metrics have shown that Math Word Problem (MWP) solvers have achieved high performance on benchmark datasets. However, the extent to which existing MWP solvers truly understand language and its relation with numbers is still unclear. In this paper, we generate adversarial attacks to evaluate the robustness of state-of-the-art MWP solvers. We propose two methods, Question Reordering and Sentence Paraphrasing to generate adversarial attacks. We conduct experiments across three neural MWP solvers over two benchmark datasets. On average, our attack method is able to reduce the accuracy of MWP solvers by over 40% on these datasets. Our results demonstrate that existing MWP solvers are sensitive to linguistic variations in the problem text. We verify the validity and quality of generated adversarial examples through human evaluation.

## introduction a math word problem (mwp) consists of a natural language text which describes a world state involving some known and unknown quantities. the task is to parse the text and generate equations that can help find the value of unknown quantities. solving mwp's is challenging because apart from understanding the text, the model needs to identify the variables involved, understand the sequence of events, and associate the numerical quantities with their entities to generate mathematical equations. an example of a simple mwp is shown in existing mwp solvers can be categorized into statistical learning based @xcite and deep learning based solvers . however, recent deep learning based approaches @xcite @xcite have established their superiority over statistical learning based solvers. here, we will briefly review some recent mwp solvers. initially, @xcite modelled the task of mwp as a sequence to sequence task and utilized recurrent neural nets (rnns) to learn problem representations. building upon this, @xcite focused on learning representations for mathematical operators and numbers, @xcite utilized tree structure to develop decoders for mwp solvers. more recently, to learn accurate relationship between numerical quantities and their attributes @xcite modelled encoder as a graph structure. all such mwp solvers have achieved high per-formance on benchmark datasets. however, the extent to which these solvers truly understand language and numbers remains unclear. prior works on various nlp tasks have shown that deep neural networks (dnns) attend to superficial cues to achieve high performance on benchmark datasets. recently, @xcite proposed a challenge test set called svamp which demonstrate that existing mwp solvers rely on shallow heuristics to achieve high performance. instead of relying on standard accuracy metrics, many works have used adversarial examples @xcite to evaluate the robustness of neural nlp models. adversarial examples are generated by making small changes to the original input such that the adversarial example is (1) semantically similar to the original input, (2) is grammatically correct and fluent and (3) deceives the dnns to generate an incorrect prediction. in @xcite authors crafted adversarial attacks to test the robustness of qa systems. prior works in @xcite uses adversarial examples to show deficiencies of nli models. similarly, @xcite uses adversarial examples to develop robust dialogue and neural machine translation models. recently, there has been a plethora of work @xcite @xcite @xcite to evaluate text classification systems against adversarial examples. although adversarial examples are commonly used for various nlp tasks, there has been no work that uses adversarial examples to evaluate mwp solvers. in this paper, we bridge this gap and evaluate the robustness of state-of-the-art mwp solvers against adversarial examples. generating adversarial attacks for mwp is a challenging task as apart from preserving textual semantics, numerical value also needs to be preserved. the text should make mathematical sense, and the sequence of events must be maintained such that humans generate the same equations from the problem text. standard adversarial generation techniques like synonym replacement @xcite are not suitable for mwp as the fluency of the problem statement is not preserved. similarly, paraphrasing techniques like back-translation @xcite are not ideal as they generate syntactically uncontrolled examples. we propose two methods to generate adversarial examples on mwp solvers, (1) question reorder-ing -it transforms the problem text by moving the question part to the beginning of the problem and (2) sentence paraphrasing -it paraphrases each sentence in the problem such that the semantic meaning and the numeric information remains unchanged. our results demonstrate that current solvers are not robust against adversarial examples as they are sensitive to minor variations in the input. we hope that our insights will inspire future work to develop more robust mwp solvers. our contributions are as follows: ## future work and conclusion the experiments in this paper showcase that nlp models do not understand mwp entirely and are not robust enough for practical purposes. our work encourages the development of robust mwp solvers and techniques to generate adversarial math examples. we believe that the generation of quality mwp's will immensely help develop solvers that genuinely understand numbers and text in combination. future works could focus on creating more such techniques for adversarial examples generation and making robust mwp solvers.