Learning to Generalize for Cross-domain QA (2023)

Abstract: There have been growing concerns regarding the out-of-domain generalization ability of natural language processing (NLP) models, particularly in question-answering (QA) tasks. Current synthesized data augmentation methods for QA are hampered by increased training costs. To address this issue, we propose a novel approach that combines prompting methods and linear probing with fine-tuning strategy, which does not entail additional cost. Our method has been theoretically and empirically shown to be effective in enhancing the generalization ability of both generative and discriminative models. Our approach outperforms state-of-the-art baselines, with an average increase in F1 score of 4.5%-7.9%. Furthermore, our method can be easily integrated into any pre-trained models and offers a promising solution to the under-explored cross-domain QA task.

## introduction question answering (qa) models @xcite @xcite ) aim to answer passage-based questions automatically with the help of facts in a given context (sometimes referred to as machine reading comprehension @xcite ). over the last few years, pre-trained models have achieved great progress on a variety of large-scale datasets, e.g., squad @xcite , newsqa @xcite , drop @xcite , cora @xcite , and narrativeqa @xcite . however, existing methods can suffer significant performance degradation when the tuned system is directly applied to outof-domain examples @xcite @xcite @xcite . this paper focuses on a novel cross-domain qa task where we assume models trained on the source domain can be generalized to the target domain, where no labeled or unlabeled data is available. as shown in figure 1 , qa pairs from different domains have intrinsically different feature distributions. for example, in the technology field, the context can frequently contain "e-commerce" and "network". while in the pharmaceutical sector, the context can consist of "covid-19", "vaccine", and "diagnostic" more frequently. cross-domain qa poses significant challenges to real-world scenarios, and it is proved that even large-scale pre-trained models @xcite can still encounter performance degradation under domain generalization. to address these drawbacks, we introduce a novel cross-domain qa setting, focusing on the methods that consistently improve the domain generalization performance without additional computational costs. intuitively, cross-domain qa can benefit from prompting in which instances from different domains can share a unified set of label words. thus, no additional parameters can carry domain-specific information to hinder the ood generalization for an unseen domain. however, using the prompt method solely could increase the risk of overfitting and bring limited benefits, as prompt templates are fixed, which may be learned as spurious features by models. thus, we consider using the linear-probing and then fine-tuning (lp-ft) strategy to reduce the reliance between prompt patterns with labels by freezing pre-trained parameters. in this way, lp-ft can benefit cross-domain qa by preventing pre-trained features from being distorted when tuning on a specific domain @xcite . prompting-based lp-ft method does not introduce new parameters, so the performance decay when training on a source domain and testing on a new target domain can be reduced without entailing additional cost. under the lp-ft framework, we introduce four prompt types to extract invariant features in different domains: question type, sentiment, named entity, and key phrase. these prompts aim to increase the question similarity and benefit the model in generalizing to out-of-domain questions. existing prompting methods have not been applied to natural language processing tasks beyond simple fine-tuning settings. to enable promoting methods to adapt lp-ft, we theoretically prove that lp-ft still holds consistently better robustness for prompting methods (section 3.3). we experiment on three different domain datasets (figure 1 ). results show that our promptbased lp-ft method significantly improves the performance of cross-domain qa models on either the standard hold-out or ood tests, with an average increase in f1 of 4.5%-7.9% compared to baselines. also, our method consistently outperforms the standard fine-tuning strategy on both discriminative and generative models. besides, we provide an in-depth analysis of the ablation study towards the ood robustness that details the efficacy of lp-ft and prompting methods, respectively. to our knowledge, we are the first to present a new zero-shot cross-domain qa task and propose a novel promptbased lp-ft method. all resources are available at https://github.com/freddieniu/prompt-qa . ## related work out-of-domain performance degradation has attracted considerable research interest recently. a line of work @xcite @xcite @xcite aims to improve models' generalization ability on text classification. differently, we investigate the ood generalization problem on the qa task. @xcite and @xcite find that 60-70% of test-time answers of popular open-domain qa benchmark datasets exist in the training set, and it is proved that training set memory plays a vital role in testing. @xcite empirically prove that language models suffer performance degradation when there is no train-test set overlapping. to test the actual generalization ability of qa models, several novel qa datasets have been constructed and released, focusing on evaluating qa models on out-of-domain generalization ability @xcite . @xcite present the first cross-domain qa dataset and observe a performance decay problem regarding the ood test. many existing methods intend to improve the ood performance of qa models through data augmentation. @xcite introduce a synthesizing question-answer pairs method to improve target-domain qa performance. in contrast, we propose a prompt-based method combined with linear probing and then fine-tuning, which is more computationally efficient and does not require target domain annotations. prompt-based methods on pre-trained language models have received considerable research interest. the paradigm "pre-train, prompt, and predict" replaces the "pre-train, fine-tune" procedure for improved few-shot learning @xcite . prompt-based methods have been applied not only in sentence-level few-shot learning tasks, such as named entity recognition @xcite but also in sophisticated learning tasks like natural language understanding @xcite . however, little work applies prompts on the cross-domain qa tasks @xcite . we leverage the fixedformat characteristic of the prompt to extract the invariant features in the changing dataset to enhance the ood generalization of the model. instead of fine-tuning, linear probing is an alternative to tuning a pre-trained model on a downstream task. liu et al. (2019a); tripuraneni et al. (2020) examine the representations produced by several recent pre-trained language models and find that linear models trained on top of frozen contextual representations are competitive with stateof-the-art task-specific fine-tuned models in many cases but fail in tasks where fine-grained language knowledge is required. @xcite theoretically prove that the linear-probing then fine-tuning (lp-ft) approach can enhance the ood generalization ability of pre-trained models. in our work, we are the first to provide theoretical evidence that lp-ft still holds consistently better robustness for compared with the baseline model, in the feature space, the prompt-based lpft model is superior in two respects: the distance between the feature distributions of the two domains is reduced, and the features learned by the model are closer to the intersection of the two domains. the top part demonstrates the linear probing and then fine-tuning process. the "q", "c" and "p" represent "question", "context" and "prompt" in a sample, respectively (in bottom) . prompting methods in nlp. ## method figure 2 illustrates the workflow of the promptbased lp-ft method. we first generate a prompt for each input sample through template engineering and prompt designing ( §3.1). then, the source domain dataset with prompts is used for linear probing and then fine-tuning ( §3.2) a pre-trained model. (top). compared with the baseline model, in the feature space, the prompt-based lpft model is superior in two respects: the distance between the feature distributions of the two domains is reduced, and the features learned by the model are closer to the intersection of the two domains (middle). the feature space demonstrates how the promptbased method and the lp-ft strategy benefit the cross-domain qa, respectively, and also shows the motivation for using prompt-based lp-ft to benefit the cross-domain qa task. ## experimental setup we introduce experiments' datasets, baseline methods, and evaluation metrics in this section. ## results and discussion our method is applied to both domain-independent qa tasks ( §5.1) and cross-domain qa tasks ( §5.2). ## conclusion we introduce a zero-shot cross-domain qa task and present a novel prompt-based lp-ft method by combining prompt with a linear-probing finetuning strategy, providing theoretical proof that the proposed method can enhance the model's indomain performance and out-of-domain generalizability, and empirically showing that the prompt lp-ft method consistently benefits the qa models. experimental results show that (1) current methods still have a lag much behind human-level towards the cross-domain qa generalization; (2) our method brings larger performance gains for generative models than discriminative models; (3) the use of the prompt-based lp-ft in other nlp tasks is worth trying. meanwhile, the emergent ability of llms will definitely decrease the challenge of the current cross-domain qa setting. designing challenging datasets of cross-domain qa towards llms should be paid more attention in the future.