R obust QA : Benchmarking the Robustness of Domain Adaptation for Open-Domain Question Answering (2023)

Abstract: Open-domain question answering (ODQA) is a crucial task in natural language processing. A typical ODQA system relies on a retriever module to select relevant contexts from a large corpus for a downstream reading comprehension model. Existing ODQA datasets consist mainly of Wikipedia corpus, and are insufficient to study models’ generalizability across diverse domains as models are trained and evaluated on the same genre of data. We propose **RobustQA**, a novel benchmark consisting of datasets from 8 different domains, which facilitates the evaluation of ODQA’s domain robustness. To build **RobustQA**, we annotate QA pairs in retrieval datasets with rigorous quality control. We further examine improving QA performances by incorporating unsupervised learning methods with target-domain corpus and adopting large generative language models. These methods can effectively improve model performances on **RobustQA**. However, experimental results demonstrate a significant gap from in-domain training, suggesting that **RobustQA** is a challenging benchmark to evaluate ODQA domain robustness.

## introduction open-domain question answering (odqa) is a crucial and practical nlp task. unlike traditional reading comprehension task where contexts are provided for a qa pair, in odqa, a retriever needs to first extract relevant passages from a large amount of documents; then a qa model provides answers based on these passages. due to the magnitude of the corpus, it is computationally prohibitive for a qa model to read through all documents. therefore, odqa becomes a popular research topic and is widely adopted in real-world applications. the practicality of odqa necessitates the evaluation of systems' out-of-domain (ood) performances because a real-world system needs to be robust when confronting domain drift. moreover, existing state-of-the-art (sota) odqa systems @xcite are based on neural networks, which are known to overfit training data and suffer from degradation when domain changes. for example, natural questions @xcite is the most commonly used odqa dataset, but a recent study shows that there is a significant amount of overlaps between its train and test sets. this partially explains why neural models trained only on nq can struggle in unseen domains @xcite . however, evaluating ood performances for odqa is currently not feasible in the research community due to the lack of a public multi-domain benchmark. existing popular odqa datasets such as @xcite rely on wikipedia or web documents. multi-domain evaluation datasets exist separately for each compo-nent of odqa. mrqa @xcite evaluates cross-domain reading comprehension performances, yet its samples are not created for opendomain settings. beir @xcite and @xcite provide crossdomain evaluation benchmarks for document retrieval or information retrieval (ir) systems, and they do not have downstream qa annotations. we present robustqa, the first multi-domain evaluation benchmark for odqa. our first contribution is that we leverage existing ood retrieval datasets, fiqa @xcite and lotte, and rigorously annotate answer spans from the retrieved documents. by further adapting two odqa datasets searchqa @xcite and @xcite , robustqa consists of eight domains over web-search, biomedicine, finance, lifestyle, recreation, technology, science, and writing, making it a diverse benchmark to evaluate odqa systems. besides benchmarking existing sota odqa systems with robustqa, we further study 1) incorporating unsupervised domain adaptation methods using target corpus pre-training and 2) leveraging prompt finetuning of large language models (llms) to improve reader performance. both methods provide overall gains on robustqa. but the performance gap between in-domain training data (nq) and robustqa is still significant, indicating our proposed benchmark datasets are challenging for odqa research. we summarize our contributions below. we briefly review odqa in this section. following the extractive qa set-up in the dpr paper @xcite , we denote a collection of documents as d. we split each document d @xmath0, p 2 , ...p m , ...p m }. denoting a token as w, a passage can be defined as p @xmath1,@xmath2, 0 ≤ @xmath3. we denote a question as q and a passage retriever as r. the task is to select k most relevant passages for q from c. formally, r(q, c) → c q . upon receiving k passages c q , a qa model predicts the most probable text span a @xmath4,j 1 :ja | w m,j 1 , . . . , w m,@xmath5, p @xmath6. we also test generative models for our task, but it is crucial to note that we are still confined to the extractive qa setting as generated texts need to be evaluated against the ground-truth answers, which are contained in the contexts. in real-world applications, a trained odqa model may encounter domain changes including token distribution shifts in c, or type and length changes in questions and answers. therefore, it is crucial to gauge domain robustness. however, a comprehensive evaluation benchmark does not exist, which we propose to address with robustqa. ## data creation existing public odqa data mostly leverage @xcite as a search corpus, and focus on factoid questions. to benchmark model robustness across a wider range of text genres and question types, we 1) annotate 6 datasets in finance, lifestyle, recreation, technology, science, and writing domains based on fiqa @xcite and @xcite ; 2) adapt two public available odqa datasets, searchqa @xcite and @xcite . the newly annotated data consist of a significant portion of challenging reasoning type of questions that cannot be answered with entities or short phrases. our data samples can be found in ## passage retrieval in this section, we describe the retrievers adopted in this work. we benchmarked five leading retrievers on robustqa. dpr adopts a bi-encoder architecture to encode a pair of question and passage independently. the passage relevancy score is calculated using vector similarity measures such as dot product. we use the best checkpoint (trained on nq) provided by the dpr paper @xcite . bm25 is a widely used sparse retriever, which matches keywords efficiently with an inverted index and can be seen as representing the question and context as weighted, high-dimensional sparse vectors. we use the bm25 implementation provided by the beir paper @xcite . bm25+ce incorporates cross-encoder (ce) architecture as passage re-rankers after obtaining bm25 results. however, reimers and gurevych (2019) points out that the ce architecture is computationally expensive as it requires both the question and passage to be fed into a language model for encoding. for this reason, in the beir paper, the authors input only the top 100 ranked passages returned by bm25 into ce for re-ranking, and we follow this practice. the original ce model is trained on ms marco (payal bajaj, 2016), and we also re-train ce using nq only from scratch, and denote this model as bm25+ce n q . colbertv2 @xcite . col-bert was initially proposed by (khattab and zaharia, 2020) using late interaction to decompose relevance modeling into token-level computations, which improves the expressivity of the query-document matching, but increases the storage requirements drastically. colbertv2 alleviates this issue with a residual compression mechanism and improves the quality of the retriever by distilling from a cross-encoder with hard-negative mining. we use the best checkpoint (trained on ms marco) provided by the paper. atlas @xcite ) jointly trains contriever @xcite , a dense retriever with bi-encoder architecture and t5 @xcite , a sequence-to-sequence language model as the reader. since we adopt atlas as one of the open-qa baseline models (sec. 5), we report its retriever performances here. ## open-domain qa to anchor our qa models against a widely tested baseline, we adopt the extractive qa model architecture used in the dpr paper. we further investigate whether we can improve extractive qa model's ood generalization by pretraining base models with unlabeled target domain corpus. as large language models (llms) are gaining research popularity nowadays, we also benchmark flan-t5 @xcite as a baseline. we also test the method that joingly trains llms with dense retriever @xcite , which is expected to have stronger performances than standalone qa models. ## results and analysis in this section, we show our benchmark results on robustqa for both passage retrieval and end-toend question answering. ## conclusion we propose robustqa, a benchmark consisting of samples across 8 different domains that better evaluates odqa systems' robustness on domain adaptation. after adopting sota odqa systems enhanced by unsupervised learning methods and llms, there still exists a significant performance gap between robustqa and the commonly used nq dataset, which suggests that robustqa is a more reliable and challenging benchmark to evaluate odqa systems' cross-domain performances.