Solving Arithmetic Word Problems Using Transformer and Pre-processing of Problem Texts (2020)

Abstract: This paper outlines the use of Transformer networks trained to translate math word problems to equivalent arithmetic expressions in infix, prefix, and postfix notations. We compare results produced by a large number of neural configurations and find that most configurations outperform previously reported approaches on three of four datasets with significant increases in accuracy of over 20 percentage points. The best neural approaches boost accuracy by 30% on average when compared to the previous state-of-the-art.

## introduction students are exposed to simple arithmetic word problems starting in elementary school, and most become proficient in solving them at a young age. however, it has been challenging to write programs to solve such elementary school level problems well. even simple word problems, consisting of only a few sentences, can be challenging to understand for an automated system. solving a math word problem (mwp) starts with one or more sentences describing a transactional situation. the sentences are usually processed to produce an arithmetic expression. these expressions then may be evaluated to yield a numerical value as an answer to the mwp. recent neural approaches to solving arithmetic word problems have used various flavors of recurrent neural networks (rnn) and reinforcement learning. however, such methods have had difficulty achieving a high level of generalization. often, systems extract the relevant numbers successfully but misplace them in the generated expressions. they also get the arithmetic operations wrong. the use of infix notation also requires pairs of parentheses to be placed and balanced correctly, bracketing the right numbers. there have been problems with parentheses placement. question: at the fair adam bought 13 tickets. after riding the ferris wheel he had 4 tickets left. if each ticket cost 9 dollars, how much money did adam spend riding the ferris wheel? some possible expressions that can be produced: to start, correctly extracting the numbers in the problem is necessary. figure 1 gives examples of some infix representations that a machine learning solver can potentially produce from a simple word problem using the correct numbers. of the expressions shown, only the first one is correct. the use of infix notation may itself be a part of the problem because it requires the generation of additional characters, the open and closed parentheses, which must be placed and balanced correctly. the actual numbers appearing in mwps vary widely from problem to problem. real numbers take any conceivable value, making it almost impossible for a neural network to learn representations for them. as a result, trained programs sometimes generate expressions that have seemingly random numbers. for example, in some runs, a trained program could generate a potentially inexplicable expression such as (25.01 4) ⇤ 9 for the problem given in figure 1 , with one or more numbers not in the problem sentences. to obviate this issue, we replace the numbers in the problem statement with generic tags like hii, hqi, and hxi and save their values as a preprocessing step. this approach does not take away from the generality of the solution but suppresses fertility in number generation leading to the introduction of numbers not present in the question sentences. we extend the prepro-cessing methods to ease the understanding through simple expanding and filtering algorithms. for example, some keywords within sentences are likely to cause the choice of operators for us as humans. by focusing on these terms in the context of a word problem, we hypothesize that our neural approach will improve further. in this paper, we use the transformer model @xcite to solve arithmetic word problems as a particular case of machine translation from text to the language of arithmetic expressions. transformers in various configurations have become a staple of nlp in the past three years. we do not augment the neural architectures with external modules such as parse trees or deep reinforcement learning. we compare performance on four individual datasets. in particular, we show that our translation-based approach outperforms stateof-the-art results reported by @xcite @xcite @xcite by a large margin on three of four datasets tested. on average, our best neural architecture outperforms previous results by almost 10%, although our approach is conceptually more straightforward. we organize our paper as follows. the second section presents related work. then, we discuss our approach. we follow by an analysis of baseline experimental results and compare them to those of other recent approaches. we then take our best performing model and train it with motivated preprocessing, discussing changes in performance. we follow with a discussion of our successes and shortcomings. finally, we share our concluding thoughts and end with our direction for future work. ## related work past strategies have used rules and templates to match sentences to arithmetic expressions. some such approaches seemed to solve problems impressively within a narrow domain but performed poorly otherwise, lacking generality @xcite @xcite . @xcite used feature extraction and template-based categorization by representing equations as expression forests and finding a close match. such methods required human intervention in the form of feature engineering and the development of templates and rules, which is not desirable for expandability and adaptability. hosseini et al. @xcite performed statistical similarity analysis to obtain acceptable results but did not perform well with texts that were dissimilar to training examples. existing approaches have used various forms of auxiliary information. hosseini et al. @xcite used verb categorization to identify important mathematical cues and contexts. mitra and baral @xcite used predefined formulas to assist in matching. @xcite parsed the input sentences, enumerated all parses, and learned to match, requiring expensive computations. roy and roth @xcite performed searches for semantic trees over large spaces. some recent approaches have transitioned to using neural networks. semantic parsing takes advantage of rnn architectures to parse mwps directly into equations, or expressions in a mathspecific language @xcite . rnns have shown promising results, but they have had difficulties balancing parentheses. sometimes rnn models incorrectly choose numbers when generating equations. rehman et al. @xcite used part-of-speech tagging and classification of equation templates to produce systems of equations from third-grade level mwps. most recently, sun et al. @xcite used a bidirectional lstm architecture for math word problems. huang et al. @xcite used a deep reinforcement learning model to achieve character placement in both seen and new equation templates. wang et al. @xcite also used deep reinforcement learning. we take a similar approach to @xcite in preprocessing to prevent ambiguous expression representation. this paper builds upon @xcite , extending the capability of similar transformer networks and solving some common issues found in translations. here, we simplify the tagging technique used in @xcite , and apply preprocessing to enhance translations. ## approach we view math word problem solving as a sequenceto-sequence translation problem. rnns have excelled in sequence-to-sequence problems such as translation and question answering. the introduction of attention mechanisms has improved the performance of rnn models. vaswani et al. @xcite introduced the transformer network, which uses stacks of attention layers instead of recurrence. applications of transformers have achieved state-of-the-art performance in many nlp tasks. we use this architecture to produce character sequences that are arithmetic expressions. the models we experiment with are easy and efficient to train, allowing us to test several configurations for a comprehensive comparison. we use several configurations of transformer networks to learn the prefix, postfix, and infix notations of mwp equations independently. prefix and postfix representations of equations do not contain parentheses, which has been a source of confusion in some approaches. if the learned target sequences are simple, with fewer characters to generate, it is less likely to make mistakes during generation. simple targets also may help the learning of the model to be more robust. ## experiment 2: preprocessing for improved results we use various additional preprocessing methods to improve the training and testing of mwp data. one goal of this section is to improve the notably low performance on the ai2 tests. we introduce eight techniques for improvement and report our results as an average of 3 separate training and testing sessions. these techniques are also tested together in some cases to observe their combined effects. ## conclusions and future work in this paper, we have shown that the use of transformer networks improves automatic math word problem-solving. we have also shown that postfix target expressions perform better than the other two expression formats. our improvements are well-motivated but straightforward and easy to use, demonstrating that the well-acclaimed transformer architecture for language processing can handle mwps well, obviating the need to build specialized neural architectures for this task. in the future, we wish to work with more complex mwp datasets. our datasets contain basic arithmetic expressions of +, -, *, and /, and only up to 3 of them. for example, datasets such as dol-phin18k @xcite , consisting of webanswered questions from yahoo! answers, require a wider variety of language to be understood by the system. we wish to use other architectures stemming from the base transformer to maximize the accuracy of the system. for our experiments, we use the 2017 variation of the transformer @xcite , to show that generally applicable neural architectures work well for this task. with that said, we also note the importance of a strategically designed architecture to improve our results. to further the interest in automatic solving of math word problems, we have released all of the code used on github. 2