Math Word Problem Solving by Generating Linguistic Variants of Problem Statements (2023)

Abstract: The art of mathematical reasoning stands as a fundamental pillar of intellectual progress and is a central catalyst in cultivating human ingenuity. Researchers have recently published a plethora of works centered around the task of solving Math Word Problems (MWP) — a crucial stride towards general AI. These existing models are susceptible to dependency on shallow heuristics and spurious correlations to derive the solution expressions. In order to ameliorate this issue, in this paper, we propose a framework for MWP solvers based on the generation of linguistic variants of the problem text. The approach involves solving each of the variant problems and electing the predicted expression with the majority of the votes. We use DeBERTa (Decoding-enhanced BERT with disentangled attention) as the encoder to leverage its rich textual representations and enhanced mask decoder to construct the solution expressions. Furthermore, we introduce a challenging dataset, ParaMAWPS, consisting of paraphrased, adversarial, and inverse variants of selectively sampled MWPs from the benchmark Mawps dataset. We extensively experiment on this dataset along with other benchmark datasets using some baseline MWP solver models. We show that training on linguistic variants of problem statements and voting on candidate predictions improve the mathematical reasoning and robustness of the model. We make our code and data publicly available.

## introduction math word problem solving is a long-standing research problem in artificial general intelligence (agi) and a lot of studies about this topic, from both industry and academia, have been published recently. a typical math word problem (mwp) takes the form of a written narrative that articulates a problem scenario and poses a question regarding one or more unknown quantities. a language model capable of solving such problems has problem: 69 handbags are sold for @xmath07 each. how much did the boutique earn after selling all the handbags? expression: @xmath2: 3354 to translate the human-readable problem statement to a valid mathematical expression that can be evaluated to obtain the numeric answer. an example of a classic mwp is portrayed in a lot of challenges manifest while designing an automated system for solving these problems @xcite . the primary challenge is to understand the quantities in the problem and capture their complex mathematical interconnections from a linear textual sequence written in natural language. there exists a diverse range of mwps with differing difficulty levels, i.e., varying numbers of unknown values, and depth of the relationships between quantities, which require good mathematical reasoning ability to solve. furthermore, the absence of crucial information and the presence of irrelevant information in the problem statements proves to be quite a challenge for the solver models @xcite . other challenges include learning to tackle the chronological and temporal ambiguities of the events happening in the problem statements and dealing with mwps that significantly differ from the training set in terms of semantic and syntactic structure. to address the problem outlined in deberta (decoding-enhanced bert with disentangled attention) @xcite is currently one of the most popular language models due to its effectiveness in achieving state-of-the-art results on a variety of natural language processing tasks, including language translation, text classification, and question answering. in our work, we find that the deberta model achieves value accuracies of 63.5% and 91.0% on the svamp dataset @xcite and the mawps dataset @xcite respectively. it falls behind the current sota accuracy of roberta-deductreasoner @xcite by a slight margin of 1 ± 0.20% on the mawps dataset, but exceeds its accuracy of 47.3 ± 0.20% on the svamp dataset. our code and data are publicly available at - https://github.com/starscream-11813/ variational-mathematical-reasoning ## problem formulation a math word problem s is a sequence of word tokens and numeric values, where the v @xmath3, . . . , v m } denotes the word tokens in s and the set n @xmath4, . . . , n l } denotes the set of numeric quantities in s. the set of word tokens v s consists of entities such as names of people, objects, units, and rates while the set of quantities n s consists of the numerical amount relevant to those entities. the goal of an mwp solver model is to map s to a valid mathematical expression e, consisting of the quantities in (n s ∪ c), where c is a set of constants, and the fundamental mathematical operators @xmath5, -, ×, ÷}, which can be evaluated to obtain the correct answer. ## methodology figure-1 in appendix-a shows an overview of our proposed architecture. given a problem statement s, we prompt the paraphraser model to generate k linguistic variants of s which are, s 1 , s 2 , . . . , s k . these k variant problems along with the seed problem s consists of quantities that are tagged appropriately using quantity tags. each of the k + 1 text sequences is then tokenized and the content embeddings h and positional embeddings p of the tokens are fed to the deberta model. the disentangled self-attention mechanism of deberta's encoder utilizes h and p to generate the output h output , which is a contextual representation of the content of each problem statement. h output , along with the relative positional embeddings p and absolute positional embeddings i of each of the problem statements are used by the transformer layers of enhanced mask decoder (emd) of deberta to generate the k + 1 predicted equations e 1 , e 2 , . . . , e k+1 . these equations are then simplified and the equation that is predicted the most number of times is elected as the final prediction of the model. this majority voting module is used only during the validation/testing phase and for inference. during the training phase, the k + 1 problem statements are deemed as stand-alone training samples and the negative log-likelihood loss (nllloss) is calculated using the predicted equations and the ground-truth equation. consequently, if the training set of the dataset used to train the model consists of n samples, it is as if the model is trained with (k + 1) × @xmath6. the knowledge points gathered after being trained on an extra kn samples contributes to the robustness of the model. ## conclusion and future work in this paper, we propose the idea of an mwp solving framework that utilizes the paraphrased linguistic variations of problem texts to train a de-berta model that generates candidate solution expressions and finalizes the predicted math expression by employing majority voting on a set of simplified candidate expressions. our findings demonstrate that incorporating linguistic variants of problem statements during training and utilizing a voting mechanism for candidate predictions enhance the model's mathematical reasoning and overall robustness. we also introduce a large-scale, diverse, and challenging singleequation mwp dataset, paramawps, consisting of paraphrased, inverse, and adversarial variants of selectively sampled datapoints from mawps, as a formidable evaluation test-bed and a proper benchmark for training mwp solver models. we wish to experiment further with harder problem text variations (e.g. grammatical errors) and conduct a thorough error analysis of the models for identifying their lapses in mathematical reasoning and discovering more scopes of improvement. we also aim to expand our research to encompass the intricate realms of multi-equation, multi-step deduction, and domain-knowledge problems. we hope our approach and findings will pave the way to more scholarly works on the vistas of agi and in tandem be deemed a noteworthy and meaningful contribution to this domain of research.