VAULT : VA riable Unified Long Text Representation for Machine Reading Comprehension (2021)

Abstract: Existing models on Machine Reading Comprehension (MRC) require complex model architecture for effectively modeling long texts with paragraph representation and classification, thereby making inference computationally inefficient for production use. In this work, we propose VAULT: a light-weight and parallel-efficient paragraph representation for MRC based on contextualized representation from long document input, trained using a new Gaussian distribution-based objective that pays close attention to the partially correct instances that are close to the ground-truth. We validate our VAULT architecture showing experimental results on two benchmark MRC datasets that require long context modeling; one Wikipedia-based (Natural Questions (NQ)) and the other on TechNotes (TechQA). VAULT can achieve comparable performance on NQ with a state-of-the-art (SOTA) complex document modeling approach while being 16 times faster, demonstrating the efficiency of our proposed model. We also demonstrate that our model can also be effectively adapted to a completely different domain – TechQA – with large improvement over a model fine-tuned on a previously published large PLM.

## introduction machine reading comprehension (mrc) has seen great advances in recent years with the rise of pre-trained language models (plm) @xcite @xcite and public leaderboards @xcite @xcite @xcite . while some challenges @xcite focus on reading comprehension with shorter contexts, many others @xcite @xcite focus on longer contexts that cannot fit into a typical 512 sub-token transformer window. motivated by this, we focus on reading comprehension with long contexts. one newer approach to this task @xcite focuses on modeling document hierarchy to represent multi-grained information for answer extraction. although this approach creates a strong representation of the text, it suffers from a significant drawback. the graph-based methods @xcite are inefficient on parallel hardware, such as gpus, resulting in slow inference speed @xcite . motivated by this, in this paper, we propose a reading comprehension model that addresses the above issue and uses a more light-weight, parallelefficient (i.e. efficient on parallel hardware) paragraph representation based on long contextual representations for providing paragraph answers to questions. instead of modeling document hierarchy from tokens to document pieces, we first introduce a base model that builds on top of a large "longcontext" plm (we use longformer, @xcite to model longer contexts with lightweight representations of each paragraph. we note that while our approach could work with any plm, we expect it to perform better with models that can support long contexts and therefore see more paragraph representations at once @xcite . to provide our model a notion of paragraph position relative to a text we also introduce positionaware paragraph representations (papr) utilizing special markup tokens and provide them as input for efficient paragraph classification. this approach allows us to encode paragraph-level position in the text and teach the model to impute information on each paragraph into the hidden outputs for these tokens that we can exploit to determine in which paragraph the answer resides. we then predict the answer span from this identified paragraph. while previous mrc methods @xcite use ground-truth start and end span positions exclusively as training objectives when extracting answer spans from the context and consider all other positions as incorrect instances equally. however, spans that overlap with the ground-truth should be considered as partially correct. motivated by @xcite which proposes a new optimization criteria based on constructing prior distribution over synonyms for machine translation, we further propose to improve the above base model by considering the start and end positions of ground-truth answer spans as gaussian-like distributions, instead of single points, and optimize our model using statistical distance. we call this final model, vault (variable unified long text representation) as it can handle a variable number and lengths of paragraphs at any position with the same unified model structure to handle long texts. to evaluate the performance of vault, we select the new natural questions (nq, @xcite and techqa @xcite datasets. nq attempts to make machine reading comprehension (mrc) more realistic by providing longer wikipedia documents as contexts and real user search-engine queries as questions, and aims at avoiding observation bias: high lexical overlap between the question and the answer context which can happen frequently if the question is created after the user sees the paragraph @xcite @xcite @xcite @xcite . the task introduces the extraction of long answers (henceforth la; typically paragraphs) besides also requiring short answers (henceforth sa) similar to squad @xcite . in figure 1 we examine an example from nq along with the answers of vault and @xcite . we see that while vault can extract answers from the very bottom of a page -if relevant -the existing system suffers from positional bias. it often predicts answers from the first paragraph of wikipedia (a region which often contains the most relevant information). we evaluate our model for domain adaptation on techqa, a recently introduced challenging dataset for qa on technical support articles where answers are typically 3-5 times longer than standard mrc datasets @xcite . empirically we first show that vault achieves comparable performance on nq with @xcite )'s document modeling architecture based on graph neural networks while being 16 times faster, demonstrating the efficiency of our proposed model. secondly, we show the generalization of our model architecture for domain adaptation on techqa. our experiments show that our model pre-trained on nq can be effectively adapted to techqa outperforming a standard fine-tuned model trained on a large plm such as roberta. to summarize, our contributions include: ## related work machine reading comprehension has been widely modeled as cloze-type span extraction @xcite @xcite . in nq, we need to identify answers in two levels, long and short answers. @xcite adapt a span extraction model for short answer extraction. @xcite ) construct complex networks for paragraph-level representation to enhance long answer classification along with span extraction for short answers. in this work, we propose a more light-weight and parallel-efficient way for constructing paragraph-level representation and classification by using longer context and modeling the negative instance through gaussian prior optimization. using the hierarchical nature of a long document for question answering has been previously studied by @xcite , where they use a hierarchical approach to select candidate sentences and extract answers in those candidates. however, due to the limit of input length for large plms, existing methods @xcite @xcite slice long documents into document pieces and perform prediction for each piece separately. in our work, we show that by modeling longer input with position-aware paragraph representation coupled with gaussian prior optimization (which is novel for mrc), we can achieve comparable performance using much simpler architecture compared to previous models, which coincide with recent new plm for long inputs on question answering @xcite foot_0 . ## model architecture in this section, we introduce vault, our proposed model that uses a simple yet effective paragraph representation based on a longer context. vault starts from a base classifier that utilizes positionaware paragraph representations trained on top of a large plm: longformer @xcite . next, we further introduce our gaussian priorbased training objective that considers partial credits for positions near the ground-truth, instead of only focusing on one ground-truth position. we show an overview of vault on the example from figure 1 in figure 2 . ## conclusions in this work we introduce and examine a powerful yet simple model for reading comprehension on long texts which we call vault, based on the hypothesis that with a large sequence length long answers can be classified effectively without computationally heavy graph-based models. we validate our approach by showing it yields f1 scores competitive with heavier methods at a fraction of the decoding cost on two very different domain benchmark datasets that require reading long texts.