Question Directed Graph Attention Network for Numerical Reasoning over Text (2020)

Abstract: Numerical reasoning over texts, such as addition, subtraction, sorting and counting, is a challenging machine reading comprehension task, since it requires both natural language understanding and arithmetic computation. To address this challenge, we propose a heterogeneous graph representation for the context of the passage and question needed for such reasoning, and design a question directed graph attention network to drive multi-step numerical reasoning over this context graph. Our model, which combines deep learning and graph reasoning, achieves remarkable results in benchmark datasets such as DROP.

## introduction machine reading comprehension (mrc) aims to develop ai models that can answer questions for text documents. recently, the performance of mrc in public datasets has been improved dramatically due to the advanced pre-trained models, such as bert @xcite , roberta @xcite and albert @xcite . however, pre-trained models are not explicitly aware of the concepts of numerical reasoning since numeracy supervision signals are rarely available during pretraining. the representations from these pre-trained models fall short in their ability to support downstream numerical reasoning. yet such ability is critical for the comprehension of financial news and scientific articles, since basic numerical operations, such as addition, subtraction, sorting and counting, need to be conducted to extract the essential information @xcite . recently, @xcite proposed a numericallyaware qanet (naqanet), which treats the span extractions, counting, and numerical addition/subtraction separately. however, this work is preliminary in the sense that the model neglects the relative magnitude between numbers. to improve this method, @xcite proposed numnet, which constructs a number comparison graph that encodes the relative magnitude information between numbers on directed edges. although numnet achieves superior performance than other numerically-aware models @xcite @xcite , we argue that numnet is insufficient for sophisticated numerical reasoning, since it lacks two critical ingredients for numerical reasoning: the number type and entity information play essential roles in numerical comprehension and reasoning. as per the study in the cognitive system -"this abstract, notation-independent appreciation of numbers develops gradually over the first several years of life ... human infants appreciate numerical quantities at a non-symbolic level: they know approximately how many objects they see before them even though they do not understand number words or arabic numerals.", the concept of discrete number is gradually developed through the reallife experience @xcite . the association among the numbers and entities is a strong regularization for learning the numerical reasoning model: the comparison and addition/subtraction between numbers are typically applied to those with the same type or referring to the same entity. to illustrate it, we show two concrete examples of numerical reasoning over texts in the question. the entities in the question may occur in several sentences in the passage, indicating how each number is related to each other through these bridging entities, which helps the qa model better collect and aggregate the information for numerical reasoning. we also observe that when the question entities co-occur in a single sentence (the last sentence in this example), this could be a hint that the answer can be derived from that sentence. the second example illustrates the case in span extraction. similarly, the model is benefited when the correlations between the numbers and "stephen gostkowski" are explicitly provided. to explicitly integrate the type and entity information into the model, we construct a heterogeneous directed graph where the nodes consist of entities and different types of numbers, and the edges can encode different types of relations. the corresponding graph of the example in based on this heterogeneous graph, we propose a question directed graph attention network (qdgat) for the task of numerical mrc. as the answer-related numbers can be directed by the question, qdgat incorporates the contextual encoding of the question in the graph reasoning process. more specifically, qdgat employs a contextual encoder, such as bert @xcite and roberta @xcite , to extract the representations of the numbers and entities in both the question and the passage, serving as the initial embeddings of each node in the graph. with the heterogeneous graph, qdgat learns to collect information from the graph conditioned on the question for numerical reasoning. each node is also described by a context-aware representation conditioned on the question, and the representations are updated through a message-passing iteration. after multiple iterations of message passing with graph neural networks, qdgat gradually aggregates the node information to answer the question. in this sense, qdgat abstracts the representation of passage and question in a way more consistent with human perception and reasoning, making the model produces a more interpretable reasoning pattern. we evaluate qdgat on two benchmark datasets: the drop dataset @xcite which requires discrete reasoning over the content of paragraph, and a subset of the race dataset @xcite that contains the number-related questions. experimental results indicate that qdgat achieves remarkable performance on the drop dataset, currently ranked as top 1 for all released models. and also rank first compared with other models that use the identical pre-training model. numerical reasoning in mrc. numerical reasoning has been studied when solving arithmetic word problems (awp). however, existing awp models only worked on small datasets, and the arithmetic expression must be clearly given. numerical reasoning in mrc is more challenging since the numbers and reasoning rules are extracted from raw text, which requires a more sophisticated model. naqanet improved the output layer of qanet to predict the answers from the arithmetic computation over numbers. in addition to naqanet, genbert @xcite injects numerical skills into bert by generating numerical data. @xcite provides a semantic parser that points to locations in the text that can be used in further numerical operations. bert-calculator @xcite defines a set of executable programs and learns to choose one to derive numerical answers. numnet @xcite uses a numerically-aware graph neural network to encode numbers, which made further progress on the drop dataset. however, the graph in numnet contains only numbers and ignores their types and context information which play a key point in numerical reasoning. our model differs from numnet in two aspects: (1) we use a heterogeneous graph containing entities and different types of numbers to encode the relations among the entities and numbers, rather than the relations from numerical comparison; (2) we use the question embedding to modulate the attention over graph neighbors and update the representation to achieve reasoning. ## method in this section, we first introduce the machine reading comprehension task requiring numerical reasoning. then the framework of our model is provided, followed by detailed descriptions about its components. ## conclusion in this work, we propose a novel method named qdgat for numerical reasoning in the machine reading comprehension task. our method not only builds a more compact graph containing different types of numbers, entities, and relations, which can be a general method for other sophisticated reasoning tasks but also conditions the reasoning directly on the question language embedding, which modulates the attention over graph neighbors and change messages being passed iteratively to achieve reasoning. the experimental results verify the effectiveness of our method. in the future, we plan to extend our model to learn the heterogeneous graph automatically, which assures more flexibility for numerical reasoning. we would also explore to learn the types of numbers and entities together the reasoning modules using variational autoencoder techniques (kingma and welling, 2014), which may help the ner system better adapt to the numerical reasoning task.