A Knowledge-Aware Sequence-to-Tree Network for Math Word Problem Solving (2020)

Abstract: With the advancements in natural language processing tasks, math word problem solving has received increasing attention. Previous methods have achieved promising results but ignore background common-sense knowledge not directly provided by the problem. In addition, during generation, they focus on local features while neglecting global information. To incorporate external knowledge and global expression information, we propose a novel knowledge-aware sequence-to-tree (KA-S2T) network in which the entities in the problem sequences and their categories are modeled as an entity graph. Based on this entity graph, a graph attention network is used to capture knowledge-aware problem representations. Further, we use a tree-structured decoder with a state aggregation mechanism to capture the long-distance dependency and global expression information. Experimental results on the Math23K dataset revealed that the KA-S2T model can achieve better performance than previously reported best results.

## introduction math word problem solving has attracted increasing attention, and many math word problem solving systems have been developed. early statistical learning methods @xcite @xcite extracted templates or features from problems and generated corresponding math expressions based on these templates or features. these methods require a large number of manually formulated features or can only be applied to small application problems in small areas. in recent years, many methods @xcite @xcite have been developed that apply neural networks to analyze math word problems, with promising results. these methods use end-to-end models to directly generate the corresponding math expressions from the problem text. although previous methods have achieved promising results, several problems remain that need to be addressed: 1) background knowledge and common sense should be incorporated. for example, as shown in figure 1 , both apples and oranges are fruit. humans are naturally aware of this common-sense information, but it is difficult for the model to learn this information from the problem text alone. 2) when generating expressions, sequence-to-sequence (seq2seq) methods tend to focus on local features and ignore global information. for example, the root node "/" of the expression tree in figure 1 is directly adjacent to its right child "4", but they are eight steps apart in the pre-order expression sequence. @xcite proposed a sequence-to-tree (seq2tree) method for generating an expression tree in pre-order based on the parent node and the left sibling tree of each node. however, global information is still not being considered in the generated expression tree. to overcome these problems, we propose a novel knowledge-aware sequence-to-tree (ka-s2t) method for exploring how to better utilize external knowledge and capture the global expression information. the proposed model connects related entities and categories based on external knowledge bases to capture common-sense information and obtain better interaction between words. in addition, we designed a tree-structured decoder to capture the long-distance dependency and global expression information. ka-s2t updates all nodes in the generated expression at each time step, whereby the node state is updated by a recursive aggregation of its neighboring nodes. through multiple iterations of aggregation, the model can use global information associated with the generated expression to generate the next node and thereby achieve better predictions. the main contributions of this paper can be summarized as follows: ## models in this section, we define the problem and present our proposed ka-s2t model. as shown in figure 2 , we first use a bidirectional long short-term memory (lstm) network to encode the math word problem sequences (section 2.2). then, we construct an entity graph based on external knowledge to model the relationships between different entities and categories in the problem (section 2.3), and use a two-layer graph attention network (gat) to obtain knowledge-aware problem representations (section 2.4). finally, we used a tree-structured decoder with a state aggregation mechanism to generate pre-order traversal math expression trees (section 2.5). ## related work solving math word problems has long been a challenging task @xcite @xcite and has attracted the attention of many researchers. some methods on math word problem solving incorporate extra features by manually crafting fine-grained templates or defining math concepts. @xcite formulated finegrained templates and aligned numbers in math word problems to those candidate templates. @xcite developed declarative rules to transform math concepts into expressions. these methods require manually formulated features and may be difficult to apply to math word problems in different domains. recently, many studies have used deep learning methods to incorporate external knowledge from the knowledge base into many nlp tasks, such as dialogue systems @xcite and reading comprehension @xcite . these methods extend knowledge triples into natural language sequences or build multi-hop inference graphs based on relationships in the knowledge base, and have achieved promising results. in this paper, we model the entities in the problem and their categories as entity graphs and use graph attention to generate knowledge-aware problem representations. seq2seq neural networks @xcite ) have achieved promising results on math word problem solving. for instance, @xcite used a seq2seq model to generate math expressions. @xcite incorporated reinforcement learning into the model to construct a math expression step by step. @xcite used a data-driven approach to semantically parsing text into math expressions. recently, tree-structured decoder was used to further improve the seq2seq framework. @xcite propose a seq2tree model to generate expression tree in a goal-driven manner based on the parent node and left sibling tree of each node. @xcite propose a tree-structured decoding method with an auxiliary stack that generates the abstract syntax tree of the equation in a top-down manner. in this paper, we generated the pre-order math expression tree based on parent node state of each node and recursively aggregated neighbors of each node in the partial expression tree to incorporate global information. ## conclusion in this study, we proposed a novel knowledgeaware sequence-to-tree model that can automatically solve math word problems. we used an entity graph to incorporate common sense knowledge from external knowledge bases into the proposed model. in addition, we proposed a tree-structured decoder with a state aggregation mechanism for generating math expressions. our experimental results confirmed that our ka-s2t model outperformed other state-of-the-art models.