Retrieval Data Augmentation Informed by Downstream Question Answering Performance (2022)

Abstract: Training retrieval models to fetch contexts for Question Answering (QA) over large corpora requires labeling relevant passages in those corpora. Since obtaining exhaustive manual annotations of all relevant passages is not feasible, prior work uses text overlap heuristics to find passages that are likely to contain the answer, but this is not feasible when the task requires deeper reasoning and answers are not extractable spans (e.g.: multi-hop, discrete reasoning). We address this issue by identifying relevant passages based on whether they are useful for a trained QA model to arrive at the correct answers, and develop a search process guided by the QA model’s loss. Our experiments show that this approach enables identifying relevant context for unseen data greater than 90% of the time on the IIRC dataset and generalizes better to the end QA task than those trained on just the gold retrieval data on IIRC and QASC datasets.

## introduction answering questions over a large text corpus typically requires retrieving information relevant to the question from the corpus, which is then used by a question answering (qa) model to arrive at the answer. recent work @xcite @xcite relies on retrieval models that learn dense representations of questions and retrieval candidates @xcite trained separately or jointly with the qa model. these learned retrieval models are more effective than those that use simple word overlap signals @xcite , but they require the positive retrieval targets for each question labeled. it is often difficult, if not impossible, to exhaustively label all the facts relevant to answering a question in a large corpus of text. consequently, even when the datasets provide retrieval labels, it is often the case that there exist alternative paths to the answer that ## method overview and problem our approach uses the standard two-step pipeline for open-domain qa seen in prior work. we first run a retrieval model that takes as input a question, q, and a large corpus of passages, c, and outputs a small subset of those passages, c ⊂ c, that contains sufficient information to answer the question. this subset is then passed to the second step: the qa model. this model takes as input the same question, q, and subset of passages, c, from the first step, and outputs an answer, a. depending on the data, this answer can take many forms, such as a span from the context, a number, yes/no, or none of these if the question is unanswerable. for each question, there may be many valid sets of context passages, where each set 1 contains all the information necessary to answer the question. we refer to individual sets as c * i , and the superset of all such sets as c * = {c * 1 . . . c * n }. as seen in figure 1 , these different context sets may express different reasoning paths reaching the answer, or they may contain different ways of expressing the same reasoning path. however, most datasets just contain annotations of one such set per question, c * i . our goal is to use these annotations to identify alternate, unannotated, relevant context, @xmath0, for each question. these additional contexts is used to augment the retrieval training data. approach the goal of the retrieval model is to identify context that maximizes the probability of the correct answer when given to the qa model. when supervised data, c * i , is available, 1 we apply our approach to datasets containing questions that require multiple facts to answer, so we label sets of facts. based on this, for each question, we search over the corpus for the top k contexts, ĉ1 . . . ĉk , and add them as additional data augmentation when training a new retrieval model: lastly, we train a final qa model using the gold context, including the results of this new retrieval model to incorporate the updated training and make it more robust to noise: labeling sets of facts because we apply our approach to datasets containing questions that require multiple facts to answer, we need to label sets of facts, not individual ones. for this reason, we train our base qa models conditioned on sets of facts, and while both labeling new contexts with the base qa model, and retrieving contexts, we use beam search to output sets of facts. in order to prevent the base qa model from memorizing the gold contexts, we use a 10-fold cross-labeling approach. foot_0 ## experiments we show the effect of our approach on two multihop qa datasets: iirc @xcite and qasc @xcite . ## related work most similar to our work are recent approaches using weak supervision for learning to retrieve for qa, using only questions and answers. @xcite pretrain a retrieval model using an inverse cloze task. @xcite more recently pro-posed to iteratively improve a retrieval model using hard-em. both approaches filter the data using the answer span heuristic. this heuristic breaks down on multi-hop questions, as well as questions that are not answerable by spans, such as true/false or discrete reasoning questions. @xcite and @xcite propose using knowledge distillation to incorporate qa information into a supervised retriever, and while assuming access to retrieval annotations, @xcite jointly learn retrieval and qa by marginalizing over potential contexts. all three of these approaches require encoding all potential contexts together with the question, whereas ours does not have that requirement, making ours more memory-efficient. ## conclusion this work shows that using the loss of a qa model trained on a partial set of labeled contexts to search for alternative contexts for retrieval is an effective method for augmenting the retriever's training data. our results present a more label-efficient training scheme for building supervised retrievers for qa. they also suggest that creators of datasets for open qa tasks that require supervised retrievers can better allocate their annotation budgets by obtaining retrieval labels for a small set of questions while maximizing the number of question-answer annotations.