Generating Equation by Utilizing Operators : GEO model (2020)

Abstract: Math word problem solving is an emerging research topic in Natural Language Processing. Recently, to address the math word problem-solving task, researchers have applied the encoder-decoder architecture, which is mainly used in machine translation tasks. The state-of-the-art neural models use hand-crafted features and are based on generation methods. In this paper, we propose the GEO (Generation of Equations by utilizing Operators) model that does not use hand-crafted features and addresses two issues that are present in existing neural models: 1. missing domain-specific knowledge features and 2. losing encoder-level knowledge. To address missing domain-specific feature issue, we designed two auxiliary tasks: operation group difference prediction and implicit pair prediction. To address losing encoder-level knowledge issue, we added an Operation Feature Feed Forward (OP3F) layer. Experimental results showed that the GEO model outperformed existing state-of-the-art models on two datasets, 85.1% in MAWPS, and 62.5% in DRAW-1K, and reached comparable performance of 82.1% in ALG514 dataset.

## introduction math word problem solving is an emerging research topic in natural language processing. the problem solving task involves dealing with numbers or numeric context information, which is useful in understanding language. recently, to address the math word problem solving task, researchers applied neural models with the encoder-decoder architecture, which is mainly used in machine translation tasks. such models generate equations as outputs using given natural language sentence sequences as inputs @xcite . the state-of-the-art neural models use hand-crafted features and are based on generation methods. in the primary benchmark datasets, such hand-crafted featurebased models have recently reported performance levels of about 80%, while neural models remain at 60-70% @xcite . due to the relatively low performance of neural models, models that utilize hand-crafted features still retain state-of-the-art (sota) performance in many datasets @xcite . however, defining and building hand-crafted features is costly. in this paper, we propose the geo (generation of equations by utilizing operators) model that does not use hand-crafted features and addresses two issues that are present in existing neural models: 1. missing domain-specific knowledge features and 2. losing encoder-level knowledge. the first issue of missing domain-specific knowledge feature occurs when neural models don't make enough use of domain-specific information of the target domain. for our case of math word problem solving, domain-specific knowledge is mathematical information. considering how humans extract and process mathematical information from a word problem when building an equation for the problem, people identify main mathematical elements, such as numbers and operations @xcite . existing neural models so far assumes that such main mathematical elements would be automatically identified in the model training process. yet, explicitly utilizing information about numbers and operations can be used to achieve better model performance. therefore, in the geo model, we will introduce two auxiliary tasks that enable the encoder to learn information about numbers and operators explicitly. the second issue of losing encoder-level knowledge occurs when decoder-learned hidden states are not utilized in the generation process in an encoder-decoder model. in the case of math problems, when information is lost in the generation process, the decoder may generate an irrelevant number that does not appear in the problem. to solve this issue, @xcite introduced a method of copying numbers in the questions. however, this method has a limitation in that it is difficult to apply the method when an answer equation contains a number that does not appear explicitly in the problem. therefore, to address this problem, we will introduce an additional layer, operation feature feed forward (op3f) layer that combines the encoder hidden states and the decoder hidden states in the geo model. the op3f layer will allow the encoder-decoder model to reflect the information learned from the encoder during the generation process. the geo model is a neural model that is capable of sufficiently learning mathematical information without using hand-crafted features. our experimental results show that the geo model outperforms the existing neural models. in addition, the geo model yields a comparable performance as the hand-crafted feature-based model. the contributions of our research are summarized as follows: the following sections are constructed as follows: first, in section 2, we introduce the previous attempts for math word problem solving. next, in section 3, we explain the geo model's architecture and its components. in section 4 and 5, we report the experimental setup and results along with a discussion of our results. lastly, in section 6, we conclude our study and present future research directions. ## related work our goal is to improve the performance of the neural model to be similar or better compared to the hand-crafted feature (hcf)-based model by addressing the two problems of the existing neural model, missing domain-specific knowledge, and losing encoder-level knowledge. in this section, we present prior models that were used in solving math word problems and explore directions of improvement for existing models. in math word problem solving tasks, earlier machine learning studies utilized hcfs that reflect the domain-specific knowledge to extract mathematical information. kushman's study @xcite , which was the first attempt in solving the math problems using machine learning techniques, showed that math word problems could be solved by using expert-defined hcfs. following kushman's research, studies reported that improved performance can be obtained by defining and utilizing features related to domain-specific knowledge @xcite . in particular, the mixedsp @xcite , which utilized context-related/question-related hcfs, has been reported as sota of 83.0% in the benchmark dataset of alg514. recently, with the emergence of neural models, researchers solved the math problem task by using neural models with only automatically extracted features, which is cost efficient compared to expertdesigned hcfs @xcite . yet, such models did not show comparable performance to the hcf-based models. for example, @xcite 's deep neural solver (dns), which utilizes word embedding and gated recurrent unit (gru) network for automatic feature extraction, reported performance of 70.1% in the alg514 dataset, which is lower than the performance of an hcf-based model with a performance of 79.7%. the cass model @xcite also showed similar trends in performance between pure neural models and hcf-based models. cass reported 44.8% performance for alg514 in the pure neural model, whereas the model reported an 82.5% performance for the same model with hcf-based domain-specific features. a possible explanation for why the neural models yield lower performance than the hcf-based models can be that the neural models could not sufficiently utilize the domain-specific knowledge feature, which the hcf-based models could usefully utilize. considering that the performance improvement occurred when hcfs were additionally given to the same model in several studies, it could be said that the model did not lack the learning ability but could not extract the domain-specific knowledge features. thus, if a technique can be devised in which the neural model can automatically extract such domain-specific knowledge features, there is still a possibility that the neural model will perform better or comparable to the existing hcf-based models. the fine-tuning technique of the language model, which has been actively applied to neural models in recent years, offers useful implications for finding ways to automatically extract domain-specific knowledge features. the fine-tuning technique is a technique that adjusts the language model, which is a pretrained representation using a large amount of corpus to address a specific task. the fine-tuning technique has been reported to yield good performance in various natural language processing tasks @xcite @xcite ). in fact, some studies have shown that applying the technique contributes to performance improvement in the math word solving task. for example, @xcite reported the highest performance in the korean language for two datasets of cc and il using bert @xcite , a widely used language model. however, it is insufficient to conclude that fine-tuning the pretrained language model would be useful for extracting domain-specific knowledge features. this is because there is a study showing that the pretrained language model has decreased the performance @xcite , and there has been a report that the pretrained language model did not have much effect in cases with a little amount of data @xcite . therefore, one should examine whether the impact of fine-tuning a language model can be used to extract the domain-specific knowledge feature in the math word problem solving task. in the geo model, two auxiliary tasks are designed to learn domain-specific knowledge features in the fine-tuning process of the language model. although neural models can properly learn domain-specific knowledge features, the learned features must be utilized without loss in the model's architecture, in order to contribute to performance improvement. however, in the encoder-decoder structure which is recently used to solve math word problems, if the length of the equation is complex, the information learned by the model through the encoder could be lost in the decoder's equation generation process. to avoid this losing encoder-level knowledge issue, a study such as @xcite applied techniques such as recursive neural network or attention method, which are known to respond relatively well to long sequences. however, considering that there are reports of degrading performance for long equation generation in this study as well, it is expected that further exploration to solve this problem is required. so in the geo model, we try to devise a new layer for encoder-decoder architecture that addresses the above issue. ## generating equation by utilizing operators (geo) model we propose the geo model, which is a variant of the encoder-decoder model. figure 1 demonstrates the geo model's architecture. the input for the geo is the preprocessed math word problem and the output is an answer equation which is in the form of the index for numeric constants (inc) postfix equation, which can be used to generate the solution to the given word problem. the geo model has two components which does not exist in a pure encoder-decoder model. first, the geo has auxiliary task layers component, which addresses the missing domain-specific knowledge features. second, the geo utilizes an operation feature feed forward (op3f) layer component that applies encoder-level knowledge to decoder hidden states. in the following subsections, we will explain the data preprocessing step along with the main components of the geo, which are the base model, auxiliary task layers, and the op3f layer. ## experimental setup to evaluate and understand the geo's performance, we conducted two experiments using three benchmark datasets: alg514, draw-1k, and mawps. experiment 1 was designed to compare the problemsolving accuracy of the geo against the sota models that either uses hcf or pure neural model. experiment 2 was designed to ablate how each component of the geo contributes to performance. ## conclusion in this paper, we propose a novel encoder-decoder model, the geo model for math word problem solving. the geo model's performance accuracy outperformed the existing state-of-the-art models in two out of the three datasets. also, our ablation study showed that two components of the geo model, two auxiliary tasks and the op3f layer, address the raised two issues: missing domain-specific knowledge features and missing encoder-level knowledge. as future work, we plan to apply our auxiliary tasks and op3f layer to another task, especially including number information and abstraction is needed.