Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network (2022)

Abstract: In this paper, we study how to continually pre-train language models for improving the understanding of math problems. Specifically, we focus on solving a fundamental challenge in modeling math problems, how to fuse the semantics of textual description and formulas, which are highly different in essence. To address this issue, we propose a new approach called COMUS to co ntinually pre-train language models for m ath problem u nderstanding with s yntax-aware memory network. In this approach, we first construct the math syntax graph to model the structural semantic information, by combining the parsing trees of the text and formulas, and then design the syntax-aware memory networks to deeply fuse the features from the graph and text. With the help of syntax relations, we can model the interaction between the token from the text and its semantic-related nodes within the formulas, which is helpful to capture fine-grained semantic correlations between texts and formulas. Besides, we devise three continual pre-training tasks to further align and fuse the representations of the text and math syntax graph. Experimental results on four tasks in the math domain demonstrate the effectiveness of our approach. Our code and data are publicly available at the link: blue https://github.com/RUCAIBox/COMUS .

## introduction understanding math problems via automated methods is a desired machine capacity for artificial intelligence assisted learning. such a capacity is the key to the success of a variety of educational applications, including math problem retrieval @xcite , problem recommendation @xcite , and problem solving @xcite . given [math] 0.6 of the value root find sin sqrt divide tan math syntax graph equal sin x ⋯ plus times root advcl nsubj comp obj ccomp obl nmod case det 2 2 x x textual description: given that sin x is … find the value of @xmath0 . ## preliminaries in this section, we formulate the problem statement and then introduce the math syntax graph. problem statement. generally, a math problem consists of a textual description d and several formulas {f 1 , f 2 , • • • , f m }. the textual description provides necessary background information for the math problem. it is formally denoted as a sequence of tokens @xmath1, t 2 , • • • , t l }, where t i is either a word token or a mathematical symbol (e.g., a number or an operator). the formulas describe the relationship among mathematical symbols, which is the key to understand and solve the math problem. each formula consists of a sequence of mathematical symbols, denoted as based on the above notations, this work focuses on continually pre-training a plm on unsupervised math problem corpus for domain adaptation. after that, the plm can be fine-tuned on various tasks in the math domain (e.g., knowledge point classification), and improve the task performance. ## methodology as shown in figure 2 , our approach aims to effectively encode the textual description and formulas, and fuse these two kinds of information for understanding math problems. in what follows, we first present the base models for encoding math problems, and then introduce the devised syntax-aware memory network and continual pre-training tasks. ## related work in this section, we review the related work from the following two aspects, namely math problem understanding and continual pre-training of language models. ## conclusion and future work in this paper, we proposed comus, a continual pre-training approach for math problem understanding. by integrating the formulas with the syntax tree of mathematical text, we constructed the math syntax graph and designed the syntax-aware memory network to fuse the semantic information from the text and formulas. in the memory network, we treated tokens from the text and triplets from the graph as the queries and slot entries, respectively, and modeled the semantic interaction between tokens and their semantic-related nodes via multi-view read and write operations. besides, we devised three continual pre-training tasks to further enhance and align the representations of the textual description and math syntax graph of the math problem. experimental results have shown that our approach outperforms several competitive baselines on four tasks in the math domain. in future work, we will consider applying our method to solve more difficult math-related tasks, e.g., automatic math problem solving and analysis generation. besides, we will also consider incorporating external math domain knowledge into our model to improve the understanding of mathematical logic and numerical reasoning.