Long Context Question Answering via Supervised Contrastive Learning (2022)

Abstract: Long-context question answering (QA) tasks require reasoning over a long document or multiple documents. Addressing these tasks often benefits from identifying a set of evidence spans (e.g., sentences), which provide supporting evidence for answering the question. In this work, we propose a novel method for equipping long-context QA models with an additional sequence-level objective for better identification of the supporting evidence. We achieve this via an additional contrastive supervision signal in finetuning, where the model is encouraged to explicitly discriminate supporting evidence sentences from negative ones by maximizing question-evidence similarity. The proposed additional loss exhibits consistent improvements on three different strong long-context transformer models, across two challenging question answering benchmarks – HotpotQA and QAsper.

## introduction answering questions that require reasoning over a long sequence, such over long documents or multiple documents, is a challenging task @xcite . research in this domain mostly includes tasks that involve multiple text segments, over benchmarks like hotpotqa @xcite and qasper @xcite . hotpotqa is a multi-hop qa benchmark over multiple paragraphs from wikipedia, while qasper involves reading comprehension from long academic papers, where relevant information on a question could be spread across the paper. given the task complexity @xcite , benchmarks often provide an additional set of evidence spans, such as sentences or paragraphs, for a given question answer pair. this breaks down the long-context qa task, adding a preliminary evidence span detection, which is crucial for successfully finding the correct answer, and also potentially helps in model interpretability. in this work, we propose a method for improving long-context qa via leveraging such evidence spans, by maximizing their similarity with the question. since identifying the evidences provides relevant information for answering the question, prior work showed that jointly training models to perform evidence span extraction in addition to answer generation is important for achieving high performance @xcite . to jointly perform evidence extraction and question answering, models utilize sentence representations (marker tokens) in the input; the final layer representation corresponding to these markers is then passed through a classification layer and is optimized using the cross-entropy loss in conjunction with the answer extraction/generation loss. we conjecture and demonstrate (see driven by the intuition that questions should be similar to their supporting evidences, under some specific geometric sub-space, we propose a novel supervised contrastive learning objective for the finetuning stage, aiming to maximize similarity of question-evidence representations. contrastive learning has been recently applied to a variety @xmath0... </@xmath1... </@xmath2. </@xmath3... </@xmath4... a: the dataset comes with a ranked set of relevant documents. hence the baselines do not use a retrieval system. figure 1: demonstration of our method over an example instance from qasper @xcite . a long sequence is fed to a model, producing representations for the marker special tokens. then, these vectors are used to compute our contrastive objective. the token colored in blue represents the question, and the tokens colored in green (red) represent positive (negative) evidence sentences. the goal is to maximize the similarity between the blue vector and the green vectors. ## setup in this section, we define and elaborate the common setup and notations for long-context qa. assume that we are given a question q i and a context s @xmath5, ..., s m consisting of m sentences (s j can be also a document/paragraph/passage, depending on the dataset). from s i , the task is to identify the correct answer a i and a set of n evidence spans s + @xmath6, ..., s i n } where i j are indices of the sentences that are the supporting evidence for answering the question q i . as common in the input setup of long-context transformer models @xcite @xcite , which are the current state-of-the-art models for solving longcontext qa tasks, the question and context sentences are concatenated in a single long sequence with special sentence tokens specifying sentence boundaries. then the input is passed to the longcontext transformer, which is trained to jointly identify the evidence sentences and extract or generate the answer. concretely, for each example, we prepare the following concatenated input sequence: then, a qa loss, which we denote by l qa is applied over the contextualized representation of each sentence token, and is optimized using supervision. l qa depends on the dataset and can take the form of a multi-task objective, representing multiple tasks in the context of qa @xcite (particularly evidence extraction and answer generation). ## question-evidence contrastive loss in this section, we elaborate on our novel proposed contrastive loss term, combining question-evidence similarity maximization ( §3.1), and question-type aware projections ( §3.2). ## evaluation and analysis in this section, we provide details about the experiments that we conducted and their outcomes. ## conclusion in this work, we proposed an effective sequencelevel contrastive loss for improving the performance of long-range transformers in solving qa tasks that require reasoning over long contexts. we demonstrate consistent improvement when using our approach on three different models over two different benchmarks. for future work, we propose exploring variations of our proposed supervised loss on other long-context tasks, such as longdocument and multi-document summarization, and integrating our method into information retrieval re-ranker models. depends on the size of the model). in order to determine the temperature hyperparameter τ , we searched over {0.2, 0.4, 0.6, 0.8, 1.0} per question type (if applicable). we also applied dropout with a rate of @xmath7.1 over the linear projections, which consistently improved the results over all the benchmarks. finally, we searched for the best performing λ hyperparameter over the values of {0.2, 0.4, 0.6, 0.8, 1.0}.