Domain Adaptation for Question Answering via Question Classification (2022)

Abstract: Question answering (QA) has demonstrated impressive progress in answering questions from customized domains. Nevertheless, domain adaptation remains one of the most elusive challenges for QA systems, especially when QA systems are trained in a source domain but deployed in a different target domain. In this work, we investigate the potential benefits of question classification for QA domain adaptation. We propose a novel framework: Question Classification for Question Answering (QC4QA). Specifically, a question classifier is adopted to assign question classes to both the source and target data. Then, we perform joint training in a self-supervised fashion via pseudo-labeling. For optimization, inter-domain discrepancy between the source and target domain is reduced via maximum mean discrepancy (MMD) distance. We additionally minimize intra-class discrepancy among QA samples of the same question class for fine-grained adaptation performance. To the best of our knowledge, this is the first work in QA domain adaptation to leverage question classification with self-supervised adaptation. We demonstrate the effectiveness of the proposed QC4QA with consistent improvements against the state-of-the-art baselines on multiple datasets.

## introduction question answering (qa) or reading comprehension (rc) refers to the task of extracting answers from given context paragraphs based on input questions. qa systems predict the start and end positions of possible answer spans in given context documents upon input questions. in recent studies, qa systems have achieved significant improvements with transformer models and largescale datasets @xcite @xcite . once deployed, qa systems often experience performance deterioration upon user-generated questions. such performance drops can be traced back to domain shifts in two input elements: to alleviate the performance issue in qa domain adaptation, several approaches have been proposed to reduce the discrepancy between the source and target domains. integrating labeled target qa pairs in training can effectively improve the qa system in answering out-of-domain questions @xcite @xcite , where the target data can be humanannotated qa pairs or synthetic data using question generation methods. when only unlabeled questions are available (see figure 1 ), another possible approach is to reduce inter-domain discrepancy via domain-adversarial training @xcite . combined with pseudo labeling, qa systems demonstrate improved generalization in answering target domain questions @xcite . nevertheless, previous methods either require large amounts of annotated target data or exten-sive computing power @xcite @xcite . additionally, different types of qa pairs and their distributional changes are not taken into account. as a result, existing approaches are less effective for adapting qa systems to an unseen target domain. in this paper, we propose a domain adaptation framework for qa: question classification for question answering (qc4qa). unlike existing methods, we innovatively adopt a question classification (qc) model to classify input questions from both the source and target domains into different question classes. moreover, we pseudo label the target data using a pretrained qa system and perform distribution-aware sampling to build mini-batches that resemble the target question distribution. in the adaptation stage, we propose a self-supervised adaptation framework to minimize the domain gap, in which inter-domain and intra-class discrepancies are simultaneously regularized. this is in contrast to existing baselines (e.g., domain adversarial adaptation methods) where the source data is solely used for training but without explicitly accounting for domain shifts and question distribution changes @xcite @xcite . to the best of our knowledge, qc4qa is the first work that combines question classification and self-supervised adaptation for learning domain-invariant representation in qa domain adaptation. our main contributions are as follows 1 : to adapt qa systems for domain changes, methods for qa domain adaptation have been proposed in two different settings: (1) access to contexts and qa pairs from the target domain. here, partial access to target data is provided, or a question generation model is introduced for producing synthetic qa pairs. the target data is then used to train and improve adaptation performance @xcite ; (2) access to context paragraphs and unlabeled input questions from the target domain. here, unsupervised or selfsupervised adaptation can be used to improve the performance in the target domain @xcite . in this paper, we focus on the latter setting and study qa domain adaptation with access to target contexts and unlabeled questions. domain adaptation in computer vision: domain adaptation methods have been primarily studied for image classification problems. such approaches focus on minimizing the representation discrepancy between the source and target distributions. some methods design objective functions that encourage domain-invariant features in training @xcite . other methods leverage domain-adversarial training with a discriminator to implicitly impose regularization when source and target features are distinguishable @xcite , with successful applications in various vision tasks @xcite @xcite . domain adaptation in qa: various approaches are designed to improve qa performance by generating and refining synthetic qa pairs. based on target contexts, question generation models are introduced to produce a surrogate dataset, which is used to train qa systems @xcite @xcite . contrastive adaptaion minimizes inter-domain discrep-ancy with question generation and maximum mean discrepancy (mmd) distances @xcite . when unlabeled questions are accessible, domain-adversarial training can be applied to reduce feature discrepancy between domains @xcite . pseudo labeling and iterative refinements of such labels can be used for improved joint training @xcite . question classification (qc): classifying questions of different types is a common task in natural language processing. one of the widely-used question taxonomy trec divides questions into 6 coarse classes and 50 fine classes @xcite . early machine learning methods perform qc with hand-crafted features @xcite . neural networks improve the classification performance with sentence embeddings @xcite . however, the aforementioned approaches in qa domain adaptation encourage domain-invariant features without considering samples from different classes and their distributional changes. moreover, it is hitherto unclear how to estimate class discrepancies in qa, since class labels are not available in qa datasets. to solve this problem, we propose to use qc to divide qa pairs into different classes, where questions can be classified via an additional qc model or unsupervised clustering with minimum computational costs. we exploit the question classes by reducing the discrepancy among samples from the same class ('intra-class'). additionally, we design a distribution-aware sampling strategy in qc4qa to account for distributional changes between the source and target domains. by incorporating the discrepancy terms in the objective function, our self-supervised adaptation framework qc4qa achieves significant improvements against the state-of-the-art baseline methods. ## methodology model: the qa system can be represented with function f . f takes an input question x q and context document x c as input and yields answer prediction x a , namely x @xmath0, x c ). the output x a is represented as a subspan of x c and comprises of the answer start and end positions. objective: the objective is to learn a f * , which maximizes the performance in answering questions from the target domain d t . in other words, f * minimizes the negative log likelihood (i.e., cross entropy) for x t from the target domain distribution: ## conclusion in this paper, we propose a novel framework for qa domain adaptation. the proposed qc4qa combines question classification with self-supervised adaptation techniques. qc4qa leverages question classes to reduce domain discrepancies and resemble target data distribution in training. different from existing works, qc4qa achieves superior performance by introducing a simple question classifier and incorporating the question class information in the training objective. we demonstrate the efficiency and effectiveness of qc4qa compared to state-of-the-art approaches by achieving a substantially better performance on multiple datasets. despite having adopted question classification to adapt qa systems to unseen target domains, the proposed qc4qa has certain limitations. for example, we assume access to unlabeled questions in qa datasets and have not exploited the potential benefits of different question samples and question classes. for future work, we plan to relax our settings and explore question generation and question value estimation for qa domain adaptation.