G raph MR : Graph Neural Network for Mathematical Reasoning (2021)

Abstract: Mathematical reasoning aims to infer satisfiable solutions based on the given mathematics questions. Previous natural language processing researches have proven the effectiveness of sequence-to-sequence (Seq2Seq) or related variants on mathematics solving. However, few works have been able to explore structural or syntactic information hidden in expressions (e.g., precedence and associativity). This dissertation set out to investigate the usefulness of such untapped information for neural architectures. Firstly, mathematical questions are represented in the format of graphs within syntax analysis. The structured nature of graphs allows them to represent relations of variables or operators while preserving the semantics of the expressions. Having transformed to the new representations, we proposed a graph-to-sequence neural network GraphMR, which can effectively learn the hierarchical information of graphs inputs to solve mathematics and speculate answers. A complete experimental scenario with four classes of mathematical tasks and three Seq2Seq baselines is built to conduct a comprehensive analysis, and results show that GraphMR outperforms others in hidden information learning and mathematics resolving.

## introduction mathematical reasoning aims to infer satisfactory solutions for the given mathematical problem based on logical rules. it plays a pivotal role in computer algebra @xcite @xcite (e.g., automated theorem proving), design formulas in numerical programs, and complete scientific computations. with the creation of massive data of logical expressions, neural networks (nns) based on statistical learning @xcite have shown success on mathematical reasoning tasks countered to rule-based approaches. for example, @xcite ; @xcite applied tree neural networks to process logic entailment and boolean arithmetic problems. @xcite released elaborated nns for symbolic integration and differential equations, shown greater competitiveness than algebra systems. however, much of the sequence-to-sequence (seq2seq) methods up to now @xcite @xcite have not explored the structural information behind expressions, such as precedence and associativity. for example, modern seq2seq models treat structured symbolic expressions with specific syntax as serialized natural language sentences, which result treat x+y and y+x as different expressions. moreover, most of these works focus on single tasks @xcite , lack the capability to handle complex computation. this study seeks to excavate the implicit information hidden in mathematical problems and examine the capacities of graph-to-sequence architecture to solve large-scope mathematical problems, such as algebra, arithmetic, and polynomial. the first step in this study was to investigate efficient graph substitutes of mathematical expressions. we discuss the representation ability of abstract syntax tree (ast) and directed acyclic graph (dag) @xcite , which reveal the structural information of expressions while preserving the semantics of operators and arithmetic entities. then numeral decomposition strategy is presented to replace graph node normalization of ast/dag. subsequently, we introduce graphmr, a graph-to-sequence approach, which can effectively learn preserved information in graph inputs. to display the performance and generalization ability of our proposal, we conduct complete experiments on four classes of tasks: poly1, poly6, mbasim, and mathematics. the large-scale evaluations show that graphmr achieves a better performance than state-of-the-art seq2seq models. furthermore, the experiment result shows that dag achieves the same accuracy on benchmarks with a smaller graph size than ast. to summarize, main contributions of this dissertation are as follows: the results showed that graphmr outperforms state-of-the-art seq2seq models. foot_0 2 background 2.1 mathematical reasoning with neural networks in recent years, there has been an increasing amount of research on mathematical reasoning using sequence-to-sequence neural networks. allamanis et al. (2017) tried to use parse trees to represent symbolic expressions and solved them by tree neural networks. a major problem with the parse tree is that it limits the representation capability of the model. their method can only handle small-size expressions and has an exponential explosion of semantic space to be represented. evans et al. (2018) proposed a framework called possible-worldnet and have shown its ability to solve logical entailment problems. the main disadvantage is that they failed to prove the model's generalization ability in other mathematical problems such as polynomials or arithmetics. more recently, lample and charton (2019); piotrowski et al. (2019) released elaborated synthesized datasets for symbolic integration and differential equations. their experiments have shown that seq2seq models outperform commercial cas such as matlab or mathematica. however, they considered expressions as sequences of independent symbols, which missed structural information hidden in symbolic expressions. conversely, other seq2seq related works @xcite @xcite are more focused on natural language understanding. their tasks are represented as mathematical query, which is more like neural machine translation. ## methodology to bring out the structural information of mathematics, we firstly introduced two effective graph representations, namely ast and dag. a numeral decomposition strategy was then discussed to reduce node feature complexity. following the graph generation process was then briefly described. finally, graphmr was depicted to collect and learn the hidden information from input graphs with its encoder and to give out a satisfactory solution through its decoder. ## conclusion this essay has studied the symbolic reasoning problem through graph neural networks. ast and dag were discussed to represent mathematical questions, which can preserve the structural and semantic information of the expressions. then we introduced a novel method, graphmr, based on graph2seq to address mathematical reasoning problems. evaluation results show that graphmr outperforms state-of-the-art seq2seq methods in poly1 poly6 mbasimp mathematics 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ast dag ast dag figure 6: comparison of accuracies of graphmr under different configuration. ast † and dag † represents prediction accuracies without numeral decomposition.