R o R : Read-over-Read for Long Document Machine Reading Comprehension (2021)

Abstract: Transformer-based pre-trained models, such as BERT, have achieved remarkable results on machine reading comprehension. However, due to the constraint of encoding length (e.g., 512 WordPiece tokens), a long document is usually split into multiple chunks that are independently read. It results in the reading field being limited to individual chunks without information collaboration for long document machine reading comprehension. To address this problem, we propose RoR, a read-over-read method, which expands the reading field from chunk to document. Specifically, RoR includes a chunk reader and a document reader. The former first predicts a set of regional answers for each chunk, which are then compacted into a highly-condensed version of the original document, guaranteeing to be encoded once. The latter further predicts the global answers from this condensed document. Eventually, a voting strategy is utilized to aggregate and rerank the regional and global answers for final prediction. Extensive experiments on two benchmarks QuAC and TriviaQA demonstrate the effectiveness of RoR for long document reading. Notably, RoR ranks 1st place on the QuAC leaderboard ( https://quac.ai/ ) at the time of submission (May 17th, 2021).

## introduction the task of machine reading comprehension (mrc), which requires machines to answer questions through reading and understanding a given document, has been a growing research field in natural language understanding @xcite @xcite @xcite . transformer-based pre-trained models have been widely proven to be effective in a range of natu-ral language processing tasks, including the mrc task @xcite @xcite . typically, these models consist of a stack of transformer blocks that only encode a length-limited sequence (e.g., 512). however, the input sequences in some mrc tasks may exceed the length constraint. for example, each instance in open-domain mrc usually consists of a collection of passages, such as trivi-aqa @xcite , one of the most popular open-domain mrc datasets, containing 6,589 tokens on average. in addition, for conversational mrc task, such as quac @xcite , existing methods incorporate conversation history by prepending the previous utterances to the current question, which is packed with the document into a length input (707 tokens on average). to handle a long document that exceeds the length constraint, a commonly used approach is to split a document into multiple individual chunks and then predict answers from each chunk separately. the highest scoring span in these answers is selected as the final answer. this approach is straightforward but results in two problems: (1) the reading field is limited to the regional chunk instead of the complete document; and (2) the scores of the answers are not comparable as they are not globally normalized over chunks. to address these problems, we propose ror, a read-over-read pipeline, which is able to expand the reading field from chunk-level to documentlevel. ror contains a chunk reader and a document reader, both of which are based on the pre-trained model. specifically, the chunk reader first predicts the regional answers from each chunk. these answers are then compacted into a new document through a minimum span coverage algorithm guaranteeing that its sequence length is shorter than the limitation (i.e., 512). by this means, all regional answers can be normalized in one document. this document serves as the highly-condensed version of the original document, which is further read by the document reader to predict a set of global answers. as the chunk reader and global reader provide high confidence answers from different views, we fully leverage both of them for final answer prediction. specifically, after predicting the regional and global answer spans, a voting strategy is proposed and utilized to rerank them. this voting strategy is based on the idea that a candidate regional or global answer span overlapped more with the others is more likely to be correct. the contributions are summarized as follows: ## related work mrc is a fundamental task in natural language understanding that aims to determine the correct answers to questions after reading a given passage @xcite @xcite . the best performing models in various mrc tasks are commonly based on the pre-trained language models (plms) within the typical encoding limit of 512 tokens. however, the input sequence in some mrc tasks usually exceeds the length limit, such as conversational mrc and open-domain mrc. conversational mrc, which extends the traditional single-turn mrc, requires the models to additionally understand the conversation history @xcite @xcite @xcite as dialog and conversational recommendation systems @xcite . a straightforward but effective approach of modeling the history is to prepend the previous dialogs to the current question, which will compose a lengthy input sequence with the relatively long document @xcite . open-domain mrc is a task of answering questions using a large collection of passages @xcite @xcite . the main challenge of this task is that the sequence length of multiple passages relevant to each question far exceeds the length limit of 512 tokens. for example, documents in triviaqa @xcite contain 6,589 tokens on average. to enable the plms to encode long documents, a common approach is to chunk the document into overlapping chunks of length 512, then process each chunk separately, which inevitably causes the two problems aforementioned. another intuitive approach is to increase the encoding length of the plms. for example, the recently proposed plms longformer @xcite and big bird @xcite , specifically for long document modeling, have extended the encoding length from 512 to 4,096. however, their encoding length is fixed. the two problems caused by chunking still exist when encoding the sequences longer than 4,096. in contrast, our proposed model ror is flexible which is able to encode sequences of arbitrary length. moreover, ror is assembleable and its encoder can be replaced with any plms, such as bert and longformer. theoretically, hierarchical models can be adapted to long document mrc @xcite @xcite . however, deploying the large transformer-based plms as the encoders of hierarchical models can be prohibitively costly. typically, hierarchical models parallelly encode the splitted chunks of a long document with multiple transformers, which requires extremely large gpu support. in contrast, ror only needs to read a chunk at each encoding process, then gradually predict all answers from chunk to document. therefore, ror is able to deal with a long document without consuming too much computing resources and can be more widely used than hierarchical models. ## conclusion in this work, a read-over-read (ror) pipeline is proposed for long document mrc, which contains an enhanced chunk reader to predict the regional answers and a document reader to predict the global answers. moreover, a voting strategy is designed to optimize the process of answer aggregation in ror. comprehensive empirical studies on quac and triviaqa demonstrate the effectiveness of ror, which comprehensively improves the performances of the plms on long document reading. meanwhile, electra-ror achieves state-of-the-art over all evaluation metrics on quac leaderboard.