Contrastive Domain Adaptation for Question Answering using Limited Text Corpora (2021)

Abstract: Question generation has recently shown impressive results in customizing question answering (QA) systems to new domains. These approaches circumvent the need for manually annotated training data from the new domain and, instead, generate synthetic question-answer pairs that are used for training. However, existing methods for question generation rely on large amounts of synthetically generated datasets and costly computational resources, which render these techniques widely inaccessible when the text corpora is of limited size. This is problematic as many niche domains rely on small text corpora, which naturally restricts the amount of synthetic data that can be generated. In this paper, we propose a novel framework for domain adaptation called contrastive domain adaptation for QA (CAQA). Specifically, CAQA combines techniques from question generation and domain-invariant learning to answer out-of-domain questions in settings with limited text corpora. Here, we train a QA system on both source data and generated data from the target domain with a contrastive adaptation loss that is incorporated in the training objective. By combining techniques from question generation and domain-invariant learning, our model achieved considerable improvements compared to state-of-the-art baselines.

## introduction question answering (qa) systems generate answers to questions over text. formally, such systems are nowadays trained end-to-end to predict answers conditional on an input question and a context paragraph (e.g., @xcite @xcite . therein, every qa sample is a 3-tuple consisting of a question, a context, and an answer. in this paper, we consider the subproblem of extractive qa, where the task is to extract answer spans from an unstructured context information for a given question as input. in extractive qa, both question and context are represented by running text, while the answer is defined by a start position and an end position in the context. an existing challenge for extractive qa systems is the distributional change between training data (source domain) and test data (target domain). if there is such a distribution change, the performance on test data is likely to be impaired. in practice, this issue occurs due to the fact that users, for instance, formulate text in highly diverse language or use qa for previously unseen domains @xcite . as a result, out-ofdomain (ood) samples occur that diverge from the training corpora of qa systems (i.e., which can be traced back to the invariance of the training data) and, upon deployment, lead to a drastic drop in the accuracy of qa systems. one solution to the above-mentioned challenge of a domain shift is to generate synthetic data from the corpora of the target domain using models for question generations and then use the synthetic data during training (e.g., @xcite . for this purpose, generative models have been adopted to produce synthetic data as surrogates from target domain, so that the qa system can be trained with both data from the source domain and synthetic data, which helps to achieve better results on the out-of-domain data distribution @xcite @xcite , see figure 1 for an overview of such approach. nevertheless, large quantities of synthetic data require intensive computational resources. moreover, many niche domains rely upon limited text corpora. their limited size puts barriers to the amount of synthetic data that can be generated and, as well shall see later, render the aforementioned approach for limited text corpora largely ineffective. in computer vision, some works draw upon another approach for domain adaptation, namely discrepancy reduction of representations @xcite @xcite @xcite . here, an adaptation loss or adversarial training approaches are often designed to learn domain-invariant features, so that the model can transfer learnt knowledge from the source domain to the target domain. however, the aforementioned approach for domain adaptation was designed for computer vision tasks, and, to the best of our knowledge, has not yet been tailored for qa. in this paper, we develop a framework for answering out-of-domain questions in qa settings with limited text corpora. we refer to our proposed framework as contrastive domain adaptation for question answering (caqa). caqa combines question generation and contrastive domain adaptation to learn domain-invariant features, so that it can capture both domains and thus transfer knowledge to the target distribution. this is in contrast to existing question generation where synthetic data is solely used for joint training with the source data but without explicitly accounting for domain shifts, thus explaining why caqa improves the performance in answering out-of-domain questions. for this, we propose a novel contrastive adaptation loss that is tailored to qa. the contrastive adaptation loss uses maximum mean discrepancy (mmd) to measure the discrepancy in the representation between source and target features, which is reduced while it simultaneously separates answer tokens for answer extraction. 1 the main contributions of our work are: 1. we propose a novel framework for domain adaptation in qa called caqa. to the best of our knowledge, this is the first use of contrastive approaches for learning domain-invariant features in qa systems. 2. our caqa framework is particularly effective for limited text corpora. in such settings, we show that caqa can transfer knowledge to target domain without additional training cost. 3. we demonstrate that caqa can effectively answer out-of-domain questions. caqa outperforms the current state-of-the-art baselines for domain adaptation by a significant margin. the performance of extractive question answering systems (e.g., @xcite @xcite ) is known to deteriorate when the training data (source domain) differs from the data used during testing (target domain) @xcite . approaches to adapt qa systems to a certain domain can be divided in (1) supervised approaches, where one has access to labeled data from the target domain (i.e., transfer learning; @xcite , or (2) unsupervised approaches, where no labeled information is accessible. the latter is our focus. ## discussion we now discuss variations in the performance across models and datasets. for this, we also investigate synthetic data generated by caqa manually (see appendix c). why is the performance sometimes below the upper bound (i.e., supervised training)? we see two explanations for the performance gap between supervised training and caqa (as well as the other baselines). (i) despite domain adaptation, some of the generated synthetic data cannot perfectly match the characteristics of the target domain but still reveal differences. we found this behavior, e.g., for naturalquestions. here, the average length of the synthetic answers are all below 3, as compared to 4.35 the training set of naturalquestions. this may lead to a performance gap at test time. (ii) the generated qa pairs are comparatively homogeneous and lack the diversity of the target domain. to examine this, we manually inspected synthetic samples from caqa (see appendix c). we found that the generated qa pairs cannot fully capture the diversity that is otherwise common in question formulation. for example, almost all questions in the synthetic data start with 'what', 'when', and 'who'. in contrast, in naturalquestions, we find many questions that we perceived as more diverse or event more difficult. examples are 'the court aquitted moninder singh pandher of what crime?' and 'why does queen elizabeth sign her name elizabeth r?'. such behavior is particularly exacerbated for naturalquestions, which was intentionally designed to introduce more variety in question formulation, and, hence, our contrastive domain adaptation approach might implicitly learn some of the characteristics (as compared to the state-of-the-art baselines). ## conclusion this work contributes a novel framework for domain adaptation of qa systems in settings with limited text corpora. we develop caqa in which we combine techniques from from question generation and domain-invariant learning to answer out-of-domain questions. different from existing works in question answering, we achieve this by proposing a contrastive adaptation loss. extensive experiments show that caqa is superior to other state-of-the-art approaches by achieving a substantially better performance on out-of-domain data.