Capturing Global Structural Information in Long Document Question Answering with Compressive Graph Selector Network (2022)

Abstract: Long document question answering is a challenging task due to its demands for complex reasoning over long text. Previous works usually take long documents as non-structured flat texts or only consider the local structure in long documents. However, these methods usually ignore the global structure of the long document, which is essential for long-range understanding. To tackle this problem, we propose Compressive Graph Selector Network (CGSN) to capture the global structure in a compressive and iterative manner. The proposed model mainly focuses on the evidence selection phase of long document question answering. Specifically, it consists of three modules: local graph network, global graph network and evidence memory network. Firstly, the local graph network builds the graph structure of the chunked segment in token, sentence, paragraph and segment levels to capture the short-term dependency of the text. Secondly, the global graph network selectively receives the information of each level from the local graph, compresses them into the global graph nodes and applies graph attention to the global graph nodes to build the long-range reasoning over the entire text in an iterative way. Thirdly, the evidence memory network is designed to alleviate the redundancy problem in the evidence selection by saving the selected result in the previous steps. Extensive experiments show that the proposed model outperforms previous methods on two datasets.

## introduction long document question answering (ldqa) is a task to select relevant evidence and answer questions over long text @xcite . compared to the traditional qa tasks, whose input is often under 512 tokens foot_1 , the input of ldqa can be more than 20k tokens. ldqa methods can be divided into two categories: end-to-end methods and select-then-read methods. the end-to-end methods usually take a question and a long text as input to select evidence and produce the answer in one step. for example, @xcite use longformer-encoder-decoder (led) model to select evidence in the encoder part and generate answers in the decoder part. the select-then-read methods firstly apply an evidence selection model to obtain evidence pieces in a long document and then use an answer generation model to generate answers given the evidence pieces and the question. these methods mainly focus on the evidence selection phase. for example, @xcite and @xcite select paragraphs in an open domain retrieving manner. @xcite and @xcite build structure on the chunked documents for evidence selection. @xcite model information flows among chunks to enhance the ability of the model in selecting the evidence. however, most of the two kinds of works ignore the global structure of a long document when selecting evidence pieces, which is crucial to long-range understanding. therefore, improvement on the evidence selection phase is needed. motivated by the human reading process: selectively memorizing the important pieces of information and integrating them, we propose an evidence selection model in the select-thenread method, named compressive graph selector network (cgsn). it aims to capture the global structural information in a compressive and iterative manner. specifically, the model is composed of three modules: the local graph network, the global graph network and the evidence memory network. firstly, the local graph takes a segment 3 of the document as input and implements graph attention among tokens, sentences, paragraphs and the segment itself. secondly, the global graph module selectively receives the information from the local graph and compresses it with the stored information via multi-head attention. then, the graph attention is applied to the global graph to integrate the global structural information, which is written back to the local graph nodes to enhance the expression of local nodes for evidence selection. thirdly, the evidence memory network receives and summarizes the evidence selecting results and sends them into the global network to alleviate the evidence redundancy problem. extensive experiments on two datasets show that cgsn outperforms previous methods in the evidence selection phase. using the same answer generator as the previous methods do, cgsn further reaches the best results in the answer generation phase. our contributions are as follows: long document question answering. long document question answering aims to answer the question with the comprehension of the long document and applies multi-hop reasoning among retrieved evidence paragraphs. dasigi et al. (2021) take advantage of the pre-trained model led (beltagy et al., 2020) and treat the input as a long sequence to predict the evidence paragraphs and generate the answer. zheng et al. (2020a) and ainslie et al. (2020) model the structure on the chunked document to select the evidence paragraph. although ainslie et al. (2020) claims that they explicitly model the structure of long documents, the input of their model is limited in 4k tokens, which can be regarded as a relatively long chunk. gong et al. (2020) use the recurrent mechanism to enable information flow through different chunks for evidence selection. karpukhin et al. (2020) and zhu et al. (2021) search for relevant evidence from 3 a 'segment' is a series of paragraphs in a document. ## compressive graph selector network in this section, we first formalize the long document question answering (ldqa) task, and then introduce the proposed evidence selection model, i.e. compressive graph selector network (cgsn). as for the answer generator, we use a vanilla led as the answer generator and describe the implementation details in appendix c. finally, we discuss the advantages of the select-then-read methods over the end-to-end methods. segment paragraph sentence token local graph nodes ffnn p1 p2 pn ... evidence memory . gate doc paragraph sentence global graph nodes local graph global graph outputs [cls] question [sep] paragraph1... [cls] question [sep] paragraphn [sep] @xmath0 ## experiments in this section, firstly, we discuss the main result of the cgsn model on qasper and hotpotqa-doc, and then further explore the proposed model. ## conclusion to solve the problem of lacking global structure in ldqa methods, we propose compress graph selector network to capture the global structure over the long document when selecting evidence pieces. extensive experiments demonstrate the strong per-formance of the model.