Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension (2020)

Abstract: Natural Questions is a new challenging machine reading comprehension benchmark with two-grained answers, which are a long answer (typically a paragraph) and a short answer (one or more entities inside the long answer). Despite the effectiveness of existing methods on this benchmark, they treat these two sub-tasks individually during training while ignoring their dependencies. To address this issue, we present a novel multi-grained machine reading comprehension framework that focuses on modeling documents at their hierarchical nature, which are different levels of granularity: documents, paragraphs, sentences, and tokens. We utilize graph attention networks to obtain different levels of representations so that they can be learned simultaneously. The long and short answers can be extracted from paragraph-level representation and token-level representation, respectively. In this way, we can model the dependencies between the two-grained answers to provide evidence for each other. We jointly train the two sub-tasks, and our experiments show that our approach significantly outperforms previous systems at both long and short answer criteria.

## introduction machine reading comprehension (mrc), a task that aims to answer questions based on a given document, has been substantially advanced by recently released datasets and models @xcite @xcite @xcite @xcite . natural questions (nq, @xcite , a newly released benchmark, makes it more challenging by introducing much longer documents than existing datasets and questions that are from real user queries. besides, unlike conventional mrc tasks (e.g. @xcite , in nq, answers are provided in a two-grained format: long answer, which is typically a paragraph, and short answers, which are typically one or more entities inside the long answer. figure 1 shows an example from nq dataset. existing approaches on nq have obtained promising results. for example, @xcite builds a pipeline model using two separate models: the decomposable attention model @xcite to select a long answer, and the document reader model @xcite to extract the short answer from the selected long answer. despite the effectiveness of these approaches, they treat the long and short answer extraction as two individual sub-tasks during training and fail to model this multi-grained characteristic of this benchmark, while we argue that the two sub-tasks of nq should be considered simultaneously to obtain accurate results. according to @xcite , a valid long answer must contain all of the information required to answer the question. besides, an accurate short answer should be helpful to confirm the long answer. for instance, when humans try to find the two-grained answers in the given wikipedia page in figure 1 , they will first try to retrieve paragraphs (long answer) describing the entity bowling hall of fame, then try to confirm if the location (short answer) of the asked entity exists in the paragraph, which helps to finally decide which paragraph is the long answer. in this way, the two-grained answers can provide evidence for each other. to address the two sub-tasks together, instead of using conventional documents modeling methods like hierarchical rnns @xcite @xcite , we propose to use graph attention networks @xcite and bert @xcite , directly model representations at tokens, sentences, paragraphs, and documents, the four different levels of granularity to capture hierarchical nature of documents. in this way, we directly derive scores of long answers from its paragraph-level representations and obtain scores of short answers from the start and end positions on the token-level representations. thus the long and short answer selection tasks can be trained jointly to promote each other. at inference time, we use a pipeline strategy similar to @xcite , where we first select long answers and then extract short answers from the selected long answers. experiments on nq dataset show that our model significantly outperforms previous models at both long and short answer criteria. we also analyze the benefits of multi-granularity representations derived from the graph module in experiments. to summarize, the main contributions of this work are as follows: we will release our code and models at https: //github.com/dancingsoul/nq_bert-dm. 2 preliminary ## approach in this section, we will explain our model. the main idea of our model lies in multi-granularity document modeling with graph attention networks. the overall architecture of our model is shown in figure 2 . ## experiments in this section, we will first describe the data preprocessing details, then give the experimental results and analysis. we also conduct an error analysis and two case studies in the appendix. ## related work machine reading comprehension has been widely investigated since the release of large-scale datasets @xcite @xcite @xcite . lots of work has begun to build end-to-end deep learning models and has achieved good results @xcite @xcite @xcite . they normally treat questions and documents as two simple sequences regardless of their structures and focus on incorporating questions into the documents, where the attention mechanism is most widely used. @xcite proposes a model for multiparagraph reading comprehension using tf-idf as the paragraph selection method. @xcite focuses on modeling a passage at word and sentence level through hierarchical attention. previous work on document modeling is mainly based on a two-level hierarchy @xcite @xcite @xcite . the first level encodes words or sentences to get the low-level representations. moreover, a highlevel encoder is applied to obtain document representation from the low-level. in these frameworks, information flows only from low-level to high-level. @xcite proposed a graph neural network model for summarization and this framework allows much complex information flows between nodes, which represents words, sentences, and entities in the graph. graph neural networks have shown their flexibility in a variant of nlp tasks @xcite @xcite . a recent approach that began with graph attention networks @xcite , which applies attention mechanisms to graphs. @xcite proposed knowledge graph attention networks to model the information in the knowledge graph, @xcite proposed gated attention networks, which use a convolutional sub-network to control each attention head's importance. we model the hierarchical nature of documents by representing them at four different levels of granularity. besides, the relations between nodes are represented by different types of edges in the graph. ## conclusion in this work, we present a novel multi-grained mrc framework based on graph attention networks and bert. we model documents at different levels of granularity to learn the hierarchical nature of the document. on the natural questions dataset, which contains two sub-tasks predicting a paragraph-level long answer and a token-level short answer, our method jointly trains the two sub-tasks to consider the dependencies of the twograined answers. the experiments show that our proposed methods are effective and outperform the previously existing methods by a large margin. improving our graph structure of representing the document as well as the document-level pretraining tasks is our future research goals. besides, the currently existing methods actually cannot process a long document without truncating or slicing it into fragments. how to model long documents is still a problem that needs to be solved.