To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering (2023)

Abstract: Recent advances in open-domain question answering (ODQA) have demonstrated impressive accuracy on general-purpose domains like Wikipedia. While some work has been investigating how well ODQA models perform when tested for out-of-domain (OOD) generalization, these studies have been conducted only under conservative shifts in data distribution and typically focus on a single component (i.e., retriever or reader) rather than an end-to-end system. This work proposes a more realistic end-to-end domain shift evaluation setting covering five diverse domains. We not only find that end-to-end models fail to generalize but that high retrieval scores often still yield poor answer prediction accuracy. To address these failures, we investigate several interventions, in the form of data augmentations, for improving model adaption and use our evaluation set to elucidate the relationship between the efficacy of an intervention scheme and the particular type of dataset shifts we consider. We propose a generalizability test that estimates the type of shift in a target dataset without training a model in the target domain and that the type of shift is predictive of which data augmentation schemes will be effective for domain adaption. Overall, we find that these interventions increase end-to-end performance by up to ~24 points.

## introduction general-purpose open-domain question answering (odqa; @xcite ; @xcite ; @xcite ) is an important task that automates reading and understanding a large corpus of documents to answer a given question succinctly. it is especially crucial in fields such as biomedicine, legal, news, etc., where more documents are added daily, outpacing the speed at which a user can process the information. current state-of-the-art odqa systems perform a two-stage pipeline process @xcite : 1) given a question top: average end-to-end performance of source domain model is quite poor when applied to ood datasets. source model (trained on general-purpose domain) performance improves when adapted to unseen target domain with interventions. bottom: drill-down of performance into zero and few-shot data augmentations averaged over target datasets exhibiting these shifts shows covariate and concept shifts respond to zero and fewshot data augmentations. target datasets with no shift do not improve much with any intervention while full shift benefits most from few-shot. and document corpus, a retriever @xcite @xcite selects relevant passages and 2) a question answering model, also known as a reader @xcite answers the given question based on the retrieved passages. this decoupling allows for independent advances in domain adaptation of general-purpose retrievers @xcite and question-answering @xcite models. to enable practical application, an odqa system should assist humans in keeping up with new knowledge without requiring annotations for every new domain or concept. for this, the system should be resilient to changes in the document, question, and answer distributions. unfortunately, the current work in odqa focus solely on wikipedia corpus and do not study effectiveness of a model trained on such a general-purpose domain when applied to an unseen domain. to gauge how likely it is for a source domain model to succeed on an unseen domain we need to understand its ability to work out-of-the-box or even adapt to a new target domain, under varying types and degrees of dataset shifts. @xcite . in this work, we study the challenges and interventions for generalizing odqa models to new domains via four contributions. first, to understand how well the state-of-the-art odqa system (trained on the general-purpose domain) performs on a variety of target distributions, we define a collection of datasets for evaluating domain generalization. we aggregate a set of seven odqa datasets spanning five different domains ( §2). we observe that the source odqa model does not generalize well (fig. 1 , top) on this collection ( §4). second, to automatically determine the type of data shift with only a small number of labeled target domain examples, we propose a generalizability test. this test assesses the type and degree of shift a new domain suffers with respect to the source domain ( §3). third, to understand the adaptability of the source model to a target domain, we analyze the performance of various intervention schemes, including existing zero-shot in-domain question generation and a novel few-shot language modelaided generation. these schemes create data akin to the target domain which is augmented with the source domain to learn an adapted version of the source model. overall, we observe improvement in performance across all the target datasets (fig. 1 ). the degree of improvement depends on the intervention scheme and underlying dataset shift ( §5). finally, we propose a simple and effective few-shot method that improves the performance by up to 24% in f1. this method prompts a large language model with 8 examples to generate examples for further adaptation. putting it all together, we use the generalizability test to gauge the type and degree of dataset shift in a target dataset. then, we empirically show that certain types of dataset shifts respond well to specific intervention schemes ( §5, fig. 1 ). this helps ascertain whether we can adapt a source model to unseen domain with minimal supervision. the resources used in this work are released at https: //github.com/ddua/adapt-or-annotate ## background and evaluation setup an odqa model learns interactions among three random variables: question (q), answer (a) and context corpus (c). for a given @xmath0, first the retriever r returns a set of passages, c @xmath1, c). these passages are then sent to a reader model m to obtain the final answer, â ← m(a|q, c q ). following prior work, we evaluate retriever performance with the acc@k metric, which computes if the oracle answer is found in the top-k retrieved passages foot_0 . we set @xmath2. for reader performance, we compute token-level f1 between the oracle and predicted answer. ## generalizability test there are many aspects that determine in what ways and to what extent one data distribution differs from another. it is often challenging to quantify the degree of generalizability or diverseness for a new domain without collecting enough samples to train a model in the new domain. to address this issue, we propose a method to assess the type and degree of diversity by utilizing only a few examples from the target domain as an evaluation set. ## how well do models generalize? we test the ood performance of the source model on target datasets and analyze the failures. ## interventions for improving adaptation domain adaptation is shown to be a causal intervention @xcite mechanism to effectively understand impact of an augmentation technique without much concern about spurious correlations. ## related work domain generalization in readers the most popular work in generalization in reading comprehension was introduced as part of the mrqa @xcite challenge, which focuses on transfer learning from multiple source datasets to unseen target datasets @xcite . this multi-task learning setup requires the model to perform complex reasoning at test time that may be unseen at training. however, in this work, we focus on the generalization capabilities of an end-to-end odqa setup that is able to understand passages in the new domain and not the ability to perform unseen reasoning. ## conclusion we investigate failures of odqa models under non-conservative dataset shift. we also propose a way to test compatibility of source model with new domains without much supervision. we establish how different dataset shift behave under a variety of intervention schemes. we hope future research will adopt these target datasets for evaluation.