Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension (2020)

Abstract: In this paper, we study machine reading comprehension (MRC) on long texts: where a model takes as inputs a lengthy document and a query, extracts a text span from the document as an answer. State-of-the-art models (e.g., BERT) tend to use a stack of transformer layers that are pre-trained from a large number of unlabeled language corpora to encode the joint contextual information of query and document. However, these transformer models can only take as input a fixed-length (e.g., 512) text. To deal with even longer text inputs, previous approaches usually chunk them into equally-spaced segments and predict answers based on each segment independently without considering the information from other segments. As a result, they may form segments that fail to cover complete answers or retain insufficient contexts around the correct answer required for question answering. Moreover, they are less capable of answering questions that need cross-segment information. We propose to let a model learn to chunk in a more flexible way via reinforcement learning: a model can decide the next segment that it wants to process in either direction. We also apply recurrent mechanisms to enable information to flow across segments. Experiments on three MRC tasks – CoQA, QuAC, and TriviaQA – demonstrate the effectiveness of our proposed recurrent chunking mechanisms: we can obtain segments that are more likely to contain complete answers and at the same time provide sufficient contexts around the ground truth answers for better predictions.

## introduction teaching machines to read, process, and comprehend natural language is a coveted goal of machine reading comprehension (mrc) problems @xcite @xcite @xcite . many existing mrc datasets have a similar task definition: given a document and a question, the goal is to extract a span from the document (in most cases) or instead generate an abstractive answer to answer the question. there is a growing trend of building mrc readers @xcite @xcite based on pre-trained language models @xcite , such as gpt @xcite and bert @xcite . these models typically consist of a stack of transformer layers that only allow fixed-length (e.g., 512) inputs. however, it is often the case that input sequences exceed the length constraint, e.g., documents in the triviaqa dataset @xcite contain 2,622 tokens on average. some conversational mrc datasets such as coqa @xcite and quac @xcite often go beyond the length limit as we may need to incorporate previous questions as well as relatively long documents into the input to answer the current question. to deal with long text inputs, a commonly used approach firstly chunks the input text into equallyspaced segments, secondly predicts the answer for each individual segment, and finally ensembles the answers from multiple segments @xcite . however, there are two major limitations of this approach: first, a predetermined large stride size for chunking may result in incomplete answers, and we observe that models are more likely to fail when the answer is near the boundaries of a segment, compared to the cases when an answer is in the center of a segment surrounded by richer context (figure 1 ); second, we empirically observe that chunking with a smaller stride size contributes little to (sometimes even hurts) the model performance. a possible explanation is that predicting answer for each segment independently may cause incomparable answer scores across segments. a similar phenomenon is also observed in open-domain question answering tasks @xcite . considering the limitations mentioned above, we propose recurrent chunking mechanisms (rcm) on top of the transformer-based models for mrc tasks. there are two main characteristics of rcm. first, it could let the machine reader learn how to choose the stride size intelligently when reading a lengthy document via reinforcement learning, so it helps prevent extracting incomplete answers from a segment and retain sufficient contexts around the answer. second, we apply recurrent mechanisms to allow the information to flow across segments. as a result, the model can have access to the global contextual information beyond the current segment. figure 1 : the influence of the distance between the center of the answer span and the center of the segment. the test performance (in f1 score) is evaluated on coqa using a bert-large reader. the best performance is achieved when the chunk center coincides with the answer span center. within the distance @xmath0, while 99% answers are completely covered, the performance degrades as the segment center moves away from the answer center, and the segment contains fewer relevant contexts. when the distance reaches 96, more than half of the predicted spans are incomplete. in the experiments, we evaluate the proposed rcm 1 on three mrc datasets: coqa, quac, and triviaqa. experimental results demonstrate that rcm leads to consistent performance gains on these benchmarks. furthermore, it also generates segments that are more likely to cover the entire answer spans and provide richer contextual information around the ground truth answers. 1 the code is available at https://github.com/ hongyugong/rcm-question-answering.git . ## method the proposed recurrent chunking mechanisms (rcm) are built upon the pre-trained bert models. we will briefly introduce the basic model in section 2.1, and then the rcm approach in section 2.2 and 2.3. more details of our model in training and testing are presented in sections 2.4 and 2.5. ## discussion in this section, we will analyze the performance of the baseline bert-large model and our proposed recurrent chunking mechanisms. ## related work there is a growing interest in mrc tasks that require the understanding of both questions and reference documents @xcite @xcite @xcite . recent studies on pre-trained language models @xcite @xcite have demonstrated their great success in fine-tuning on mrc tasks. however these pre-trained nlp models (e.g., bert) only take as input a fixed-length text. variants of bert are proposed to process long documents in tasks such as text classification @xcite . to deal with lengthy documents in machine reading comprehension tasks, some previous studies skip certain tokens @xcite or select a set of sentences as input based on the given questions @xcite @xcite . however, they mainly focus on tasks in which most of the answers to given questions are formed by a single informative sentence. these previous approaches are less applicable to deal with those complicated questions that demand cross-sentences reasoning or have much lexical variability from their lengthy documents. ## conclusion in this paper, we propose a chunking policy network for machine reading comprehension, which enables a model learn to chunk lengthy documents in a more flexible way via reinforcement learning. we also add a recurrent mechanism to allow the information to flow across segments so that the model could have knowledge beyond the current segment when selecting answers. we have performed extensive experiments on three public datasets of machine reading comprehension: coqa, quac, and triviaqa. our approach outperforms benchmark models across different datasets.