Improving Math Word Problems with Pre-trained Knowledge and Hierarchical Reasoning (2021)

Abstract: The recent algorithms for math word problems (MWP) neglect to use outside knowledge not present in the problems. Most of them only capture the word-level relationship and ignore to build hierarchical reasoning like the human being for mining the contextual structure between words and sentences. In this paper, we propose a R easoning with P re-trained K nowledge and H ierarchical S tructure ( RPKHS ) network, which contains a pre-trained knowledge encoder and a hierarchical reasoning encoder. Firstly, our pre-trained knowledge encoder aims at reasoning the MWP by using outside knowledge from the pre-trained transformer-based models. Secondly, the hierarchical reasoning encoder is presented for seamlessly integrating the word-level and sentence-level reasoning to bridge the entity and context domain on MWP. Extensive experiments show that our RPKHS significantly outperforms state-of-the-art approaches on two large-scale commonly-used datasets, and boosts performance from 77.4% to 83.9% on Math23K, from 75.5 to 82.2% on Math23K with 5-fold cross-validation and from 83.7% to 89.8% on MAWPS. More extensive ablations are shown to demonstrate the effectiveness and interpretability of our proposed method.

## introduction math word problem (mwp) is a reasoning task for answering a mathematical query based on the problem description, which is an interdisciplinary research topic to bridge the mathematics and natural language processing. as shown in ## related work the mwp is the task of translating a short paragraph consisting with multiple short sentences into target mathematical equations. previous approaches usually solve the mwp by using rulebased methods @xcite , statistical machine learning methods @xcite @xcite , semantic parsing methods @xcite @xcite and deep learning methods @xcite @xcite @xcite . recently, the deep learning based methods have been paid more attention for their significant improvement. @xcite proposed a seq2seq-based model to directly map the linguistic text to a solution. @xcite and @xcite implicitly modeled tree-based structure for decoding the mwp expressions, while @xcite @xcite optimized the decoder via explicit tree structure. some research focused on graph structure on word-level reasoning. for example, @xcite built two customized graphs for enriching the quantity representations in the problem. @xcite presented a graph-to-tree encoder-decoder framework for grammar parsing. however, they ignore the sentence-level relationship and the correlation between word and sentence. different from the previous methods, we propose to use hierarchical reasoning containing word-level and sentence-level reasoning. besides, we are the first ones to explore the effect of implicit knowledge from the pre-trained neural network weights on the task of math word problems. ## experiments in this section, we first introduce the data that we use and the state-of-the-art baselines that we compare against. then we show the implementation details of our experiments. next, we demonstrate our results in comparison with other methods and provide extensive analyses. finally, we conduct ablation studies and show some visualizations to investigate the effectiveness of our proposed components of our model (reasoning with pre-trained knowledge and hierarchical structure, rpkhs). 4.1 datasets and evaluation datasets. we evaluate our proposed rpkhs and compare it with other state-of-the-art methods on two commonly-used datasets, namely mawps (koncel-kedziorski et al., 2016) with 2,373 problems and math23k (wang et al., 2018b) containing 23,162 problems. evaluation. as other works do @xcite , for two datasets, we also measure the performance of our proposed method via the solution accuracy. for the math23k dataset, there are two settings for evaluation on the previous methods. one is evaluating the model on the test set (denoted as "math23k" in we adopt plateau learning rate scheduler that reduces the learning rate by half every 20 epoch. our model is trained for 80 epochs. the beam size is set to be 5 in beam search to generate expression trees, which is inspired by the gts @xcite . word-level sentence-level pre-trained math23k (%) 74.9 √ 75.8 √ 76.1 √ 80.1 √ √ 79.8 √ √ 81.4 √ √ 82.3 √ √ √ 83.9 ## conclusion we propose reasoning with pre-trained knowledge and hierarchical structure to jointly incorporate implicit knowledge and hierarchical representation into the neural network, which can be achieved by two encoders. a pre-trained knowledge encoder uses implicit knowledge for enhancing textual representation. a hierarchical reasoning encoder bridges the entity and context domain on mwp by building hierarchical reasoning between word-level and sentence-level reasoning. extensive experiments show that the proposed model achieves a new state-of-the-art performance.