A tten W alker: Unsupervised Long-Document Question Answering via Attention-based Graph Walking (2023)

Abstract: Annotating long-document question answering (long-document QA) pairs is time-consuming and expensive. To alleviate the problem, it might be possible to generate long-document QA pairs via unsupervised question answering (UQA) methods. However, existing UQA tasks are based on short documents, and can hardly incorporate long-range information. To tackle the problem, we propose a new task, named unsupervised long-document question answering (ULQA), aiming to generate high-quality long-document QA instances in an unsupervised manner. Besides, we propose AttenWalker, a novel unsupervised method to aggregate and generate answers with long-range dependency so as to construct long-document QA pairs. Specifically, AttenWalker is composed of three modules, i.e. span collector, span linker and answer aggregator. Firstly, the span collector takes advantage of constituent parsing and reconstruction loss to select informative candidate spans for constructing answers. Secondly, with the help of the attention graph of a pre-trained long-document model, potentially interrelated text spans (that might be far apart) could be linked together via an attention-walking algorithm. Thirdly, in the answer aggregator, linked spans are aggregated into the final answer via the mask-filling ability of a pre-trained model. Extensive experiments show that AttenWalker outperforms previous methods on NarrativeQA and Qasper. In addition, AttenWalker also shows strong performance in the few-shot learning setting.

## introduction textual question answering (qa) is the task of answering questions given textual documents as the context. previous works can be divided into short-document qa foot_1 methods @xcite today's increasing flood of information on the web creates a need for ... the document is first fed into an led pre-trained model @xcite (the upper half). then, the acquired token-level attention graph (not shown here) is converted into a spanlevel graph (the lower half) via the method described in section 3.4. spans (which might be far apart) are then linked if their edge weight is high. for example, the span "the main contributions" walks through 1,065 tokens and links with "a single-layer forward recurrent neural network", which is then linked with "long short-term memory" since their high weight edges (0.53 and 0.48). other spans do not connect with them due to their low edge weights to these spans. and long-document qa methods @xcite . short-document methods approach, and even outperform humans due to the availability of largescale short-document qa datasets @xcite . despite that, long-document methods still lag behind humans by a large margin since annotating long-document qa datasets @xcite is time-consuming and costly. intuitively, the high cost of annotating longdocument qa pairs can be alleviated in an unsupervised manner. however, there are only shortdocument unsupervised question answering (uqa) works @xcite , which aim to construct a large number of short-document qa pairs in an unsupervised manner and train a qa model with these qa pairs. @xcite first propose the uqa task and use unsupervised neural translation to construct qa pairs in a short passage. @xcite raise the unsupervised shortdocument multi-hop question answering (umqa) task and design a question generation method to build multi-hop questions within two short passages. to break the document length limitation and incorporate long-range information, we propose a more challenging task, i.e. unsupervised long-document question answering (ulqa) task, to generate high-quality long-document qa pairs and train a competitive qa model without any human-labeled long-document qa pairs. the core challenge of this task is in the modeling of long-range dependency without supervision. to address this issue, we study an attention-driven method to incorporate meaningful long-range information in the constructed qa pairs. figure 1 illustrates a motivating example of the attention flow in a long document. it is observed that, by walking through the attention edges of a pre-trained model, related spans would be linked and long-range dependency in the document could be constructed. therefore, long-range information could be also incorporated into qa pairs through these walkable attention patterns among text spans. thus, we propose attenwalker, a novel unsupervised framework to generate long-range dependent answers in long-document qa pairs. specifically, atten-walker comprises three modules: span collector, span linker and answer aggregator. firstly, the span collector takes advantage of the constituent parsing and reconstruction ability of a pre-trained model to select informative candidate spans. secondly, related spans that might be far apart could be connected through local or global attention edges of a long-document pre-trained model. thirdly, collected spans are aggregated through the reconstruction ability of a pre-trained model. extensive experiments on qasper @xcite and narrativeqa @xcite show that the proposed attenwalker can effectively model long-range dependency in long-document qa. besides, attenwalker also shows strong performance in the few-shot learning setting. our contributions are as follows: unsupervised question answering unsupervised question answering (uqa) (lewis et al., 2019) targets at alleviating the data scarcity problem in qa datasets. it focuses on generating qa pairs without supervision and training a qa model on them. lewis et al. (2019) firstly propose the uqa task. based on a pure short document, they extract answers via named entity tools and propose a novel cloze translation method to make alignment between cloze question and natural question so as to generate plenty of natural questions. then, the constructed (context, question, answer) triples are used to train a qa model. li et al. (2020) use cited documents to generate questions so that the overlapping problem between the generated question and the raw context could be alleviated. nie et al. (2022a) propose to mine answers beyond named entities in the synthetic qa dataset and improve the model's ability in dealing with diverse answers. pan et al. (2021) propose the first unsupervised multi-hop qa framework via multi-hop question generation. however, most of these methods focus on the short-document scenario, while the longdocument setting is still unexplored. long-document question answering longdocument question answering (long-document qa) aims to answer questions based on the understanding of a long sequence of text. previous methods can be divided into end-to-end methods and selectthen-read methods. end-to-end methods (dasigi et al., 2021) apply sparse attention models to directly answer the question given a long document. dasigi et al. (2021) uses the longformer-encoder-decoder model to make long-range reasoning on a long document and then answer a question. caciularu et al. (2022) uses a sequence-level objective to improve evidence verification. for the select-thenread methods, nie et al. (2022b) propose a compressive graph selector network to select questionrelated snippets from the long document and then use the selected short snippets for answer generation. however, despite competitive performances span collector span linker answer aggregator long document today's increasing flood of information on the web creates a need ... (6109 tokens) constituent parser long short-term memory t5 reconstructor the main contributions a single-layer forward recurrent neural network long document with partitioned spans ... ... longformer model the main contributions a single-layer forward recurrent neural network long short-term memory attention graph walking bart aggregator aggregating the main contributions were to develop a single-layer forward recurrent neural network for sentence information the answer linked linked figure 2: an overview of attenwalker. it consists of three modules, including span collector, span linker, and answer aggregator. on long-document qa, these methods heavily rely on supervised qa data and can hardly apply to the low-resource setting. ## attenwalker in this section, we first formalize the task of longdocument qa. after that, the proposed atten-walker is described in detail. ## experimental setup we evaluate the proposed attenwalker on qasper @xcite and narrativeqa @xcite . in particular, for qasper, the answer types in this dataset can be extractive, abstraction, yes/no, or unanswerable. yet, according to our analysis (appendix a), qa instances with yes/no or unanswerable answers cannot properly evaluate the ability of long document reasoning. therefore, we only focus on the extractive and abstractive qa instances in this work. the datasets splitting and processing details are in appendix c.1. we use the documents in the qasper training set to construct qa pairs for training the qa model and do qasper-related experiments. the long documents in the training set of narrativeqa are used similarly. the dataset construction details can be found in appendix c.2. what's more, the setting of the long document qa model trained on the constructed dataset can be referred to c.3. ## experiment in this section, we first discuss the main results of attenwalker on qasper and narrativeqa, and then further analyze the proposed method. ## conclusion we study a new task, named unsupervised longdocument question answering, and propose atten-walker, an unsupervised method to incorporate long-range information in qa pairs via graph walking. extensive experiments show the strong performance of the proposed method. we believe that this work can be an important step in the long-document reasoning with a low-resource setting.