Task Transfer and Domain Adaptation for Zero-Shot Question Answering (2022)

Abstract: Pretrained language models have shown success in various areas of natural language processing, including reading comprehension tasks. However, when applying machine learning methods to new domains, labeled data may not always be available. To address this, we use supervised pretraining on source-domain data to reduce sample complexity on domainspecific downstream tasks. We evaluate zeroshot performance on domain-specific reading comprehension tasks by combining task transfer with domain adaptation to fine-tune a pretrained model with no labelled data from the target task. Our approach outperforms DomainAdaptive Pretraining on downstream domainspecific reading comprehension tasks in 3 out of 4 domains.

## introduction pretrained language models @xcite require substantial quantities of labeled data to learn downstream tasks. for domains that are novel or where labeled data is in short supply, supervised learning methods may not be suitable @xcite @xcite . collecting sufficient quantities of labeled data for each new application can be resource intensive, especially when aiming for both a specific task type and a specific data domain. by traditional transfer learning methods, it is prohibitively difficult to fine-tune a pretrained model on a domain-specific downstream task for which there is no existing training data. in light of this, we would like to use more readily available labeled indomain data from unrelated tasks to domain-adapt our fine-tuned model. in this paper, we consider a problem setting where we have a domain-specific target task (qa) for which we do not have any in-domain training * equal contribution data (squad). however, we assume that we have generic training data for the target task type, and in-domain training data for another task. to address this problem setting, we present task and domain adaptive pretraining (t+dapt), a technique that combines domain adaptation and task adaptation to improve performance in downstream target tasks. we evaluate the effectiveness of t+dapt in zero-shot domain-specific machine reading comprehension (mrc) @xcite @xcite by pretraining on in-domain ner data and fine-tuning for generic domain-agnostic mrc on squadv1 @xcite , combining knowledge from the two different tasks to achieve zero-shot learning on the target task. we test the language model's performance on domain-specific reading comprehension data taken from 4 domains: news, movies, biomedical, and covid-19. in our experiments, roberta-base models trained using our approach perform favorably on domain-specific reading comprehension tasks compared to baseline roberta-base models trained on squad as well as domain adaptive pretraining (dapt). our code is publicly available for reference. 1 we summarize our contributions as follows: it has been shown that pretrained language models can be domain-adapted with further pretraining @xcite on unlabeled indomain data to significantly improve the language model's performance on downstream supervised tasks in-domain. this was originally demonstrated by biobert @xcite . @xcite further explores this method of domain adaptation via unsupervised pretraining, referred to as domain-adaptive pretraining (dapt), and demonstrates its effectiveness across several domains and data availability settings. this procedure has been shown to improve performance on specific domain reading comprehension tasks, in particular in the biomedical domain @xcite . in this paper, as a baseline for comparison, we evaluate the performance of dapt-enhanced language models in their respective domains, both in isolation with squad1.1 fine-tuning and in conjunction with our approach that incorporates the respective domain's ner task. dapt models for two of our domains, news and biomedical, are initialized from pretrained weights as provided by the authors of @xcite . we train our own dapt baselines on the movies and covid-19 domains. @xcite explore methods to reduce catastrophic forgetting during language model fine-tuning. they apply topic modeling on the ms marco dataset @xcite to generate 6 narrow domain-specific data sets, from which we use bioqa and moviesqa as domain-specific reading comprehension benchmarks. ## experiments we aim to achieve zero-shot learning for an unseen domain-specific mrc task by fine-tuning on both a domain transfer task and a generic mrc task. the model is initialized by pretrained roberta weights @xcite , then fine-tuned using our approach with a domain-specific supervised task to augment domain knowledge, and finally trained on squad to learn generic mrc capabilities to achieve zero-shot mrc in the target domain on an unseen domain-specific mrc task without explicitly training on the final task. this method is illustrated in figure 1 . ## analysis specific domains learn from adaptation: our approach shows promising performance gains when used for zero-shot domain-specific question answering, particularly in the biomedical, movies, and covid domains, where the mrc datasets were designed with the evaluation of domainspecific features in mind. performance gains are less apparent in the news domain, where the newsqa dataset was designed primarily to evaluate causal reasoning and inference abilitieswhich correlate strongly with squad and base- roberta retraining procedure moviesqa newsqa bioqa covidqa squad1.1 67.1 57.0 58.0 42.0 dapt + squad1.1 60.7 54.4 57.8 47.2 t+dapt (ours) 68.0 56.0 58.9 42.7 dapt + t+dapt 66.4 54.2 55.1 43.1 when does dapt succeed or fail: in zeroshot qa, dapt performs competitively with the baseline in all domains and outperforms in the covid domain. this builds upon the results of @xcite , which reports superior performance on tasks like relation classification, sentiment analysis, and topic modeling, but does not address reading comprehension tasks, which dapt may not have originally been optimized for. unsupervised language modeling may not provide readily transferable features for reading comprehension, as opposed to ner which identifies key tokens and classifies those tokens into specific entities. these entities are also often answer tokens in reading comprehension, lending to transferable representations between ner and reading comprehension. another possible factor is that roberta was pretrained on the english wikipedia corpus, the same source that the squad questions were drawn from. because of this, it is possible that pretrained roberta already has relevant representations that would provide an intrinsic advantage for squad-style reading comprehension which would be lost due to catastrophic forgetting after retraining on another large language modeling corpus in dapt. in the covid domain, we use the article dataset from @xcite . these articles also make the basis for the covidner and covidqa @xcite datasets, which may explain the large performance improvement from dapt in this domain. these results suggest that the performance of dapt is sensitive to the similarity of its language modeling corpus to the target task dataset. foot_1 ## conclusion we evaluate the performance of our t+dapt approach with domain-specific ner, achieving positive results in a zero-shot reading comprehension setting in four different domain-specific qa datasets. these results indicate that our t+dapt approach robustly improves performance of pretraining language models in zero-shot domain qa across several domains, showing that t+dapt is a promising approach to domain adaptation in lowresource settings for pretrained language models, particularly when directly training on target task data is difficult. in future work, we intend to explore various methods to improve the performance of t+dapt by remedying catastrophic forgetting and maximizing knowledge transfer. for this we hope to emulate the regularization used by @xcite and implement multi-task learning and continual learning methods like adapternet @xcite . in order to improve the transferability of learned features, we will explore different auxiliary tasks such as nli and sentiment analysis in addition to few-shot learning approaches.