Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022)

Abstract: Solving math word problems requires deductive reasoning over the quantities in the text. Various recent research efforts mostly relied on sequence-to-sequence or sequence-to-tree models to generate mathematical expressions without explicitly performing relational reasoning between quantities in the given context. While empirically effective, such approaches typically do not provide explanations for the generated expressions. In this work, we view the task as a complex relation extraction problem, proposing a novel approach that presents explainable deductive reasoning steps to iteratively construct target expressions, where each step involves a primitive operation over two quantities defining their relation. Through extensive experiments on four benchmark datasets, we show that the proposed model significantly outperforms existing strong baselines. We further demonstrate that the deductive procedure not only presents more explainable steps but also enables us to make more accurate predictions on questions that require more complex reasoning.

## introduction math word problem (mwp) solving @xcite is a task of answering a mathematical question that is described in natural language. solving mwp requires logical reasoning over the quantities presented in the context @xcite to compute the numerical answer. various recent research efforts regarded the problem as a generation problem -typically, such models focus on generating the complete target mathematical expression, often represented in the form of a linear sequence or a tree structure @xcite . figure 1 (top) depicts a typical approach that attempts to generate the target expression in the 1 our code and data are released at https://github. com/allanj/deductive-mwp . question: in a division sum , the remainder is 8 and the divisor is 6 times the quotient and is obt--ained by adding 3 to the thrice of the remainder. what is the divident? form of a tree structure, which is adopted in recent research efforts @xcite @xcite . specifically, the output is an expression that can be obtained from such a generated structure. we note that, however, there are several limitations with such a structure generation approach. first, such a process typically involves a particular order when generating the structure. in the example, given the complexity of the problem, the decision of generating the addition ("+") operation as the very first step could be counter-intuitive and does not provide adequate explanations that show the reasoning process when being presented to a human learner. furthermore, the resulting tree contains identical sub-trees ("8 × 3 + 3") as highlighted in blue dashed boxes. unless a certain specifically designed mechanism is introduced for reusing the already generated intermediate expression, the approach would need to repeat the same effort in its process for generating the same sub-expression. solving math problems generally requires deductive reasoning, which is also regarded as one of the important abilities in children's cognitive development @xcite . in this work, we propose a novel approach that explicitly presents deductive reasoning steps. we make a key observation that mwp solving fundamentally can be viewed as a complex relation extraction problem -the task of identifying the complex relations among the quantities that appear in the given problem text. each primitive arithmetic operation (such as addition, subtraction) essentially defines a different type of relation. drawing on the success of some recent models for relation extraction in the literature @xcite , our proposed approach involves a process that repeatedly performs relation extraction between two chosen quantities (including newly generated quantities). as shown in figure 1 , our approach directly extracts the relation ("multiplication", or "×") between 8 and 3, which come from the contexts "remainder is 8" and "thrice of the remainder". in addition, it allows us to reuse the results from the intermediate expression in the fourth step. this process naturally yields a deductive reasoning procedure that iteratively derives new knowledge from existing ones. designing such a complex relation extraction system presents several practical challenges. for example, some quantities may be irrelevant to the question while some others may need to be used multiple times. the model also needs to learn how to properly handle the new quantities that emerge from the intermediate expressions. learning how to effectively search for the optimal sequence of operations (relations) and when to stop the deductive process is also important. in this work, we tackle the above challenges and make the following major contributions: ## related work early efforts focused on solving mwp using probabilistic models with handcrafted features @xcite . @xcite and @xcite designed templates to find the alignments between the declarative language and equations. most recent works solve the problem by using sequence or tree generation models. @xcite proposed the math23k dataset and presented a sequence-to-sequence (seq2seq) approach to generate the mathematical expression @xcite . other approaches improve the seq2seq model with reinforcement learning @xcite , template-based methods @xcite , and group attention mechanism @xcite . @xcite proposed a goal-driven tree-structured (gts) model to generate the expression tree. this sequence-to-tree approach significantly improved the performance over the traditional seq2seq approaches. some follow-up works incorporated external knowledge such as syntactic dependency @xcite or commonsense knowledge @xcite . @xcite modeled the equations as a directed acyclic graph to obtain the expression. @xcite and @xcite adopted a graph-to-tree approach to model the quantity relations using the graph convolutional networks (gcn) @xcite . applying pre-trained language models such as bert @xcite was shown to significantly benefit the tree expression generation @xcite @xcite @xcite . different from the tree-based generation models, our work is related to deductive systems @xcite where we aim to obtain step-by-step expressions. recent efforts have also been working towards this direction. @xcite constructed a dataset to provide explanations for expressions at each step. @xcite created the mathqa dataset annotated with step-by-step operations. the annotations present the expression at each intermediate step during problem-solving. our deductive process (figure 1 ) attempts to automatically obtain the expression in an incremental, step-by-step manner. our approach is also related to relation extraction (re) @xcite , a fundamental task in the field of information extraction that is focused on identifying the relationships between a pair of entities. recently, zhong and chen (2021) designed a simple and effective approach to directly model the relations on the span pair representations. in this work, we treat the operation between a pair of quantities as the relation at each step in our deductive reasoning process. traditional methods @xcite applied rule-based approaches to extract the mathematical relations. mwp solving is typically regarded as one of the system 2 tasks @xcite , and our current approach to this problem is related to neural symbolic reasoning @xcite . we design differentiable modules @xcite in our model ( §3.2) to perform reasoning among the quantities. ## approach the math word problem solving task can be defined as follows. given a problem description @xmath0, w 2 , ⋯, w n } that consists of a list of n words and q @xmath1, q 2 , ⋯, q m }, a list of m quantities that appear in s, our task is to solve the problem and return the numerical answer. ideally, the answer shall be computed through a mathematical reasoning process over a series of primitive mathematical operations @xcite as shown in figure 1 . such operations may include "+" (addition), "-" (subtraction), "×" (multiplication), "÷" (division), and " * * " (exponentiation). in our view, each of the primitive mathematical operations above can essentially be used for describing a specific relation between quantities. fundamentally, solving a math word problem is a problem of complex relation extraction, which requires us to repeatedly identify the relations between quantities (including those appearing in the text and those intermediate ones created by relations). the overall solving procedure requires in-voking a relation classification module at each step, yielding a deductive reasoning process. in practice, some questions cannot be answered without relying on certain predefined constants (such as π and 1) that may not have appeared in the given problem description. we therefore also consider a set of constants @xmath2, c 2 , ⋯, c |c| }. such constants are also regarded as quantities (i.e., they would be regarded as {q m+1 , q m+2 , . . . , q m+|c| }) which may play useful roles when forming the final answer expression. ## experiments datasets we conduct experiments on four datasets across two different languages: mawps @xcite , math23k @xcite , mathqa @xcite , and svamp @xcite . the dataset statistics can be found in ## conclusion and future work we provide a new perspective to the task of mwp solving and argue that it can be fundamentally regarded as a complex relation extraction problem. based on this observation, and motivated by the deductive reasoning process, we propose an end-toend deductive reasoner to obtain the answer expression in a step-by-step manner. at each step, our model performs iterative mathematical relation extraction between quantities. thorough experiments on four standard datasets demonstrate that our deductive reasoner is robust and able to yield new state-of-the-art performance. the model achieves particularly better performance for complex questions that involve a larger number of operations. it offers us the flexibility in interpreting the results, thanks to the deductive nature of our model. future directions that we would like to explore include how to effectively incorporate commonsense knowledge into the deductive reasoning process, and how to facilitate counterfactual reasoning @xcite .