Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solvers (2022)

Abstract: Existing Math Word Problem (MWP) solvers have achieved high accuracy on benchmark datasets. However, prior works have shown that such solvers do not generalize well and rely on superficial cues to achieve high performance. In this paper, we first conduct experiments to showcase that this behaviour is mainly associated with the limited size and diversity present in existing MWP datasets. Next, we propose several data augmentation techniques broadly categorized into Substitution and Paraphrasing based methods. By deploying these methods we increase the size of existing datasets by five folds. Extensive experiments on two benchmark datasets across three state-of-the-art MWP solvers shows that proposed methods increase the generalization and robustness of existing solvers. On average, proposed methods significantly increase the state-of-the-art results by over five percentage points on benchmark datasets. Further, the solvers trained on the augmented dataset performs comparatively better on the challenge test set. We also show the effectiveness of proposed techniques through ablation studies and verify the quality of augmented samples through human evaluation.

## introduction a math word problem (mwp) consists of natural language text which describes a world state involving some known and unknown quantities, followed by a question text to determine the unknown values. the task is to parse the problem statement and generate equations that can help find the value of unknown quantities. an example of a simple mwp is shown in training deep neural models effectively requires large number of data points @xcite . constructing large datasets which are annotated, labeled and have mwps of similar difficulty level is a very expensive and tedious task. to address these key challenges, we resort to data augmenta-tion techniques. our motivation behind generating augmentations is that humans require sufficient practice to understand mwps. humans learn to solve mwps by going through a variety of similar examples and slowly become capable enough to tackle variations of similar difficulty levels. we aim to generate augmentations such that sufficient linguistic variations of a similar problem are present in the dataset. these variations will make the solver more robust in tackling mwp, increase their reasoning ability and numerical understanding. data augmentation for mwps is a challenging task as we need to preserve the equation labels while generating new samples @xcite . the generated samples should be (1) semantically similar to their original counterpart, (2) must have the same numerical values and preserve relationship with their respective entities and (3) should maintain the same sequence of events in the problem text. existing augmentation methods (wei and zou) cannot be directly applied due to the above mentioned reasons. our methods can be broadly classified as follows: to ensure high quality augmentations foot_0 , we propose a selection algorithm which selects samples that have high similarity with original problem and incur high loss values when tested on existing solvers. this algorithm helps selecting only those samples that can make existing solvers more robust. further, we also verify the validity and the quality of generated augmentations through human evaluation. most of the existing mwp datasets are either in languages other than english or contain problems of varying difficulty levels @xcite @xcite @xcite . we focus on strengthening existing english language datasets which can facilitate the development of better mwp solvers. we consider datasets containing mwp that can be solved using linear equations in one variable. these datasets include mawps @xcite and asdiv-a (miao et al., 2020) both having 2, 373 and 1, 213 problems respectively. following are the key contributions made in this paper: although, these datasets have larger problem set but they are noisy and contain problems of varied difficulty levels. text data augmentation: to effectively train deep learning models, large datasets are required. data augmentation is a machine learning technique that artificially enlarges the amount of training data by means of label preserving transformations (taylor and nitschke, 2018). @xcite hypothesize that textual data augmentation would only be helpful if the generated data contains new linguistic patterns that are relevant to the task and have not been seen in pre-training. in nlp, many techniques have been used for generating augmentations, (wei and zou) introduced noise injection, deletion, insertion and swapping of words in text. @xcite used recurrent neural networks and generative adversarial networks for short-text augmentation @xcite . recently, hard label adversarial attack models have also been used @xcite . other frequently used methods include inducing spelling mistakes @xcite , synonym replacement @xcite , identifying close embeddings from a defined search space @xcite , round trip translations @xcite , paraphrasing techniques @xcite and words predicted by language model @xcite among many others. these methods are specific to the task at hand and needs to be adapted such that the generated augmentations bring diversity in the concerned dataset. ## proposed augmentation approach data augmentation generates new data by modifying existing data points through transformations based on prior knowledge about the problem domain. we introduce carefully selected transformations on well known text augmentation techniques to develop examples suited for the task of mwp. these transformations help in increasing the diversity and size of problem set in existing datasets. ## future work and conclusion we showcase that the existing mwp solvers are not robust and do not generalize well on even simple variations of the problem statement. in this work, we have introduced data augmentation techniques for generation of diverse math word problems. we were able to enhance the size of existing dataset by 5 folds and significantly increase the performance of state-of-the-art solvers by over 5 percentage points. future works could focus on developing techniques to generate data artificially and making robust mwp solvers.