QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation (2022)

Abstract: Question answering (QA) has recently shown impressive results for answering questions from customized domains. Yet, a common challenge is to adapt QA models to an unseen target domain. In this paper, we propose a novel self-supervised framework called QADA for QA domain adaptation. QADA introduces a novel data augmentation pipeline used to augment training QA samples. Different from existing methods, we enrich the samples via hidden space augmentation. For questions, we introduce multi-hop synonyms and sample augmented token embeddings with Dirichlet distributions. For contexts, we develop an augmentation method which learns to drop context spans via a custom attentive sampling strategy. Additionally, contrastive learning is integrated in the proposed self-supervised adaptation framework QADA. Unlike existing approaches, we generate pseudo labels and propose to train the model via a novel attention-based contrastive adaptation method. The attention weights are used to build informative features for discrepancy estimation that helps the QA model separate answers and generalize across source and target domains. To the best of our knowledge, our work is the first to leverage hidden space augmentation and attention-based contrastive adaptation for self-supervised domain adaptation in QA. Our evaluation shows that QADA achieves considerable improvements on multiple target datasets over state-of-the-art baselines in QA domain adaptation.

## introduction question answering (qa) is the task of finding answers for a given context and a given question. qa models are typically trained using data triplets consisting of context, question and answer. in the case of extractive qa, answers are represented as subspans in the context defined by a start position and an end position, while question and context are * both authors contributed equally to this research. a common challenge in extractive qa is that qa models often suffer from performance deterioration upon deployment and thus make mistakes for usergenerated inputs. the underlying reason for such deterioration can be traced back to the domain shift between training data (from the source domain) and test data (from the target domain) @xcite @xcite . existing approaches to address domain shifts in extractive qa can be grouped as follows. one approach is to include labeled target examples or user feedback during training @xcite @xcite . another approach is to generate labeled qa samples in the target domain for training @xcite @xcite . however, these approaches typically require large amounts of annotated data or extensive computational resources. as such, they tend to be ineffective in adapting existing qa models to an unseen target domain @xcite . only recently, a contrastive loss has been proposed to handle domain adaptation in qa @xcite . several approaches have been used to address issues related to insufficient data and generalization in nlp tasks, yet outside of qa. for example, augmentation in the hidden space encourages more generalizable features for training @xcite @xcite . for domain adaptation, there are approaches that encourage the model to learn domain-invariant features via a domain critic @xcite , or adopt discrepancy regularization between the source and target domains @xcite . however, to the best of our knowledge, no work has attempted to build a smooth and generalized feature space via hidden space augmentation and self-supervised domain adaption. in this paper, we propose a novel self-supervised qa domain adaptation framework for extractive qa called qada. our qada framework is designed to handle domain shifts and should thus answer out-of-domain questions. qada has three stages, namely pseudo labeling, hidden space augmentation and self-supervised domain adaptation. first, we use pseudo labeling to generate and filter labeled target qa data. next, the augmentation component integrates a novel pipeline for data augmentation to enrich training samples in the hidden space. for questions, we build upon multihop synonyms and introduce dirichlet neighborhood sampling in the embedding space to generate augmented tokens. for contexts, we develop an attentive context cutoff method which learns to drop context spans via a sampling strategy using attention scores. third, for training, we propose to train the qa model via a novel attention-based contrastive adaptation. specifically, we use the attention weights to sample informative features that help the qa model separate answers and generalize across the source and target domains. main contributions of our work are: 1 ## related work extractive qa has achieved significantly progress recently @xcite @xcite ). yet, the accuracy of qa models can drop drastically under domain shifts; that is, when deployed in an unseen domain that differs from the training distribution @xcite . 1 the code for our qada framework is publicly available at https://github.com/yueeeeeeee/self-supervised-qa . to overcome the above challenge, various approaches for qa domain adaptation have been proposed, which can be categorized as follows. data augmentation for nlp: data augmentation for nlp aims at improving the language understanding with diverse data samples. one approach is to apply token-level augmentation and enrich the training data with simple techniques (e.g., synonym replacement, token swapping, etc.) @xcite or custom heuristics @xcite . alternatively, augmentation can be done in the hidden space of the underlying model @xcite . for example, one can drop partial spans in the hidden space, which aids generalization performance under distributional shifts @xcite but in nlp tasks outside of qa. to the best of our knowledge, we are the first to propose a hidden space augmentation pipeline tailored for qa data in which different strategies are combined for question and context augmentation. contrastive learning for domain adaptation: contrastive learning is used to minimize distances of same-class samples and maximize discrepancy among classes @xcite . for this, different metrics are adopted to measure pair-wise distances (e.g., triplet loss) or domain distances with mmd @xcite . contrastive learning can also be used for domain adaptation by reducing the domain discrepancy: this "pulls together" intra-class features and "pushes apart" inter-class representations. here, several applications are in computer vision @xcite . in qa domain adaptation, contrastive learning was applied with averaged token features to separate answer tokens and minimize the discrepancy between source and target domain @xcite . however, our work is different in that we introduce a novel attention-based strategy to construct more informative features for discrepancy estimation and contrastive adaptation. ## setup we consider the following problem setup for qa domain adaptation, where labeled source data and unlabeled target data are available for training. our goal is to train a qa model f that maximizes the performance in the target domain using both source data and unlabeled target data @xcite @xcite . data: our research focuses on question answering under domain shift. let d s denote the source domain, and let d t denote the (different) target domain. then, labeled data from the source domain can be used for training, while, upon deployment, it should perform well on the data from the target domain. specifically, training is two-fold: we first pretrain a qa model on the source domain d s and, following this, the pretrained qa model is adapted to the target domain d t . the input data to each domain is as follows: model: the qa model can be represented with function f . f takes both a question and context as input and predicts an answer, i.e., ## conclusion in this paper, we propose a novel self-supervised framework called qada for qa domain adaptation. qada introduces: (1) hidden space augmentation tailored for qa data to enrich target training corpora; and (2) an attention-based contrastive adaptation to learn domain-invariant features that generalize across source and target domain. our experiments demonstrate the effectiveness of qada: it achieves a superior performance over state-ofthe-art baselines in qa domain adaptation.