Source-Free Domain Adaptation for Question Answering with Masked Self-training (2024)

Abstract: Previous unsupervised domain adaptation (UDA) methods for question answering (QA) require access to source domain data while fine-tuning the model for the target domain. Source domain data may, however, contain sensitive information and should be protected. In this study, we investigate a more challenging setting, source-free UDA, in which we have only the pretrained source model and target domain data, without access to source domain data. We propose a novel self-training approach to QA models that integrates a specially designed mask module for domain adaptation. The mask is auto-adjusted to extract key domain knowledge when trained on the source domain. To maintain previously learned domain knowledge, certain mask weights are frozen during adaptation, while other weights are adjusted to mitigate domain shifts with pseudo-labeled samples generated in the target domain. Our empirical results on four benchmark datasets suggest that our approach significantly enhances the performance of pretrained QA models on the target domain, and even outperforms models that have access to the source data during adaptation.

## introduction question-answering (qa) systems have achieved impressive performance in recent years with the development of pre-trained language models (plms). despite this, research shows that plms are sensitive to domain shifts, when training and evaluation datasets have different distributions @xcite . in addition, the performance of modern plms depends heavily on human-annotated in-domain data when it comes to specific tasks @xcite . the collection of high-quality datasets is timeconsuming and expensive, resulting in many realworld applications relying mostly on unlabeled data. a possible solution to these issues is to transfer knowledge from a relevant labeled source domain to an unlabeled target domain, which is referred to as unsupervised domain adaptation (uda). the key to implementing uda for qa is to extract common features from source and target contexts that provide sufficient information to determine where the answer is located. following this idea, previous work @xcite utilizes adversarial training to make the features extracted from the source and target indistinguishable from each other. recently, plms have become capable of recalling large amounts of information and performing a wide range of tasks @xcite . some recent studies @xcite @xcite incorporate self-training and multitasking training strategies into large plms for the task. by simultaneously performing the qa task on the source domain and the language modeling task on the target domain, it is assumed that plm learns qa knowledge while dealing with domain differences. however, for these methods to be effective, they must still have access to the source domain data when the models are adapted to target domains. this assumption is often not realistic in real-life applications, as some datasets contain sensitive information with limited access. in these cases, only models trained on these datasets are available to the public. for example, due to privacy concerns in the clinical domain, the organizers of semeval 2021 task 10 @xcite released models trained on mayo clinic clinical notes for public use, yet the original data remain confidential due to ''complicated data use agreements''. our study explores source-free uda, a setting where users can adapt models developed on private data to their own target domain without access to private source data. historically, this has been a very difficult setting. recent advancements in transformer-based plms @xcite suggest that these systems possess a remarkable capacity for learning domain-specific knowledge and effectively answering queries. it is therefore feasible to explore how trained models can be adapted to target domains in the absence of original source data. we propose a novel approach based solely on self-training, called masked domain adaptation for question answering (mdaqa). a bottleneck module is designed first with a masking mechanism. this bottleneck module is then inserted between the qa encoder and the final prediction layer. when the qa model is trained on the source domain, the mask is learned automatically to pass only values of specific kernels of the bottleneck module to the prediction layer, while the remaining kernels are left unused. when the qa model is adapted to the target domain, weights of links to early mentioned kernels are frozen in order to keep previously learned domain knowledge, while the weights related to remaining kernels are adjusted to mitigate the domain variance. since the target domain data are unlabeled, we use the qa model trained on the source domain to make predictions on the target domain data. then we use predictions with high confidence as pseudo labels to further fine-tune our model. 1 we evaluate mdaqa on well-known qa benchmark datasets. experiment outcomes show that mdaqa significantly improves the performance of pretrained qa models on target domains. mdaqa even outperforms previous uda methods, which require access to source data during adaptation. to summarize, the proposed method has the following features: in this section, we briefly review two lines of related work. first, existing uda for qa works. second, the recently proposed source-free uda methods. ## problem definition for extractive qa, given a supporting context @xmath0, p 2 , . . . , p l 1 ) with l 1 tokens and a query @xmath1, q 2 , . . . , q l 2 ) with l 2 tokens, the answer @xmath2, p a s +1 , . . . , p a e ) is a text piece in the original passage p. this goal of extractive qa is to identify the correct answer span (a s , a e ), 0 < a @xmath3, where a s , a e represent the starting and ending indexes of the answer in passage p. source-free uda for qa can then be formally defined as follows: there exists a source domain d s , which possesses labeled data that is only accessible during the initial training of the source model. a target domain d t exists, which has freely accessible but unlabeled data. in the source domain d s , we utilize n labeled samples {x i , y i } n @xmath4, where the input x @xmath5, q i ) and label y @xmath6, a e i ). we then have n unlabeled samples {x j } n @xmath7, adhering to the same qa task as aforementioned. we postulate that the data distribution in d s and d t are not identical. the initial step is to train a model on d s , after which, access to d s ceases. the core of the process, however, lies in the subsequent adaptation of the pre-trained model to d t . this transition is designed to maximize performance on the target domain by proficiently mitigating the domain shift between d s and d t . ## method in this section, we introduce our proposed approach, masked domain adaptation for question answering (mdaqa). our primary motivation is to leverage the domain knowledge acquired from the source domain and adapt it efficiently for the target domain within the context of qa. the ideal model should be capable of discerning and retaining vital information while adapting to the differences between the source and target domains, thus alleviating domain shift. to achieve this, we propose mdaqa, which consists of a feature extraction base model and a specially designed mask module. the mask module plays a pivotal role by controlling the activation and usage of specific features or kernels in the prediction layer, thereby focusing the model's learning process on pertinent domain knowledge. following this, we delve into the architecture of the base model and the specialized mask module. we also detail the training processes implemented for both the source and target domains. ## results in this section, we present our experimental results and analyze them. we first present the experimental performance of mdaqa compared to baselines on benchmark datasets, as well as the ablation study. we then analyze the impact of the number of available target domain samples. finally, we analyze the effect of the threshold α. ## conclusion in this paper, we explore the possibility of transferring knowledge for uda on qa tasks, without access to initial domain data. we proposed a novel self-training-based approach, mdaqa. we specially design an attention mask module to automatically retain key knowledge from the source domain and learn to mitigate domain shift between source and target domains. the module can be easily integrated into existing language models. our comprehensive experiments on well-known benchmark datasets demonstrate that mdaqa outperforms previous methods by a clear margin. it also performs well even when the available target domain data is highly limited, suggesting that mdaqa has wide applicability across a range of scenarios. our current study primarily focuses on extractive qa tasks, aligned to the approach of previous research in the field @xcite @xcite . we chose the roberta model specifically for its exceptional aptitude for such tasks. furthermore, we recognize the potential and need to broaden the scope of our work towards more versatile settings, such as open-domain and generative qa. to this end, our future plans include transitioning from roberta to a t5-based model. known for its ability to handle a wide variety of tasks, the t5 model will provide us the flexibility and breadth required to venture into broader and more challenging realms of qa.