S witch P rompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains (2023)

Abstract: Prompting pre-trained language models leads to promising results across natural language processing tasks but is less effective when applied in low-resource domains, due to the domain gap between the pre-training data and the downstream task. In this work, we bridge this gap with a novel and lightweight prompting methodology called SwitchPrompt for the adaptation of language models trained on datasets from the general domain to diverse low-resource domains. Using domain-specific keywords with a trainable gated prompt, SwitchPrompt offers domain-oriented prompting, that is, effective guidance on the target domains for general-domain language models. Our few-shot experiments on three text classification benchmarks demonstrate the efficacy of the general-domain pre-trained language models when used with SwitchPrompt. They often even outperform their domain-specific counterparts trained with baseline state-of-the-art prompting methods by up to 10.7% performance increase in accuracy. This result indicates that SwitchPrompt effectively reduces the need for domain-specific language model pre-training.

## introduction recent work showed promising results on different natural language processing tasks when prompting pre-trained language models (lms) instead of finetuning them, especially in low-resource settings @xcite . most lms which are publicly available have been trained on general-domain corpora @xcite @xcite , such as wikipedia or the bookscorpus @xcite . applying them to tasks from a special domain results in a domain gap. for some special domains, domain-specific lms exists, e.g., clinical bert @xcite or biobert @xcite . however, pre-training deep language models requires large amounts of text data. 1 while we can assume the availability of large-scale text data in the general domain, this assumption might not hold for lowresource domains, making the creation of domainspecific lms challenging. moreover, training different models for each and every new domain might be inefficient from a computation point of view. 2 even if there are domain-specific texts and computational resources available, domain-specific lms may not be able to get sufficient domain-oriented guidance through traditional prompting techniques because, for instance, domain-specific knowledge might be represented by a large and diverse vocabulary. as a result, both prompting lms from the general domain and from a special domain might be ineffective, especially in low-resource settings. motivated by these challenges, we explore domain-oriented prompts and propose a novel and lightweight method, switchprompt, to effectively retrieve domain-specific knowledge from pre-trained lms. it extends the sequence of softprompting vectors with a sequence of vectors representing domain-specific keywords and introduces gates to allow the model to dynamically switch between a general soft prompt and a domain-specific one based on the input sentence. we hypothesize that this approach helps to effectively retrieve domain-specific knowledge from pre-trained lms. our experiments on benchmark datasets from different domains indicate that switchprompt outperforms different state-of-the-art prompting methods. it improves results in both in-domain and out-of-domain settings, effectively reducing domain gaps among pre-training and downstream task data. we find that it is especially suitable for low-resource settings (both little data and little computational resources) as it neither requires pre-training domain-specific lms nor fine-tuning lms for the downstream task. the code base for switchprompt is available online. 3 2 related work language model prompting. prompting pretrained lms has been shown effective for different nlp tasks @xcite . while discrete prompts are intuitively understandable, their design requires non-trivial human involvement and they may be outperformed by fine-tuning @xcite @xcite . recent studies address this issue by optimizing so-called soft prompts in continuous space. @xcite propose prefix tuning that optimizes prefix activations prepended to the input layer and each layer in the encoder stack. @xcite prepend trainable continuous embeddings to the original sequence of input word embeddings. @xcite propose p-tuning in which an lstm encoder captures the sequential representations of the soft prompts. @xcite use a deep prompting methodology which injects prompts at each layer of the pre-trained lm. in contrast to those prior works, we propose a new soft prompting method that is especially suited for low-resource domains. language models in special domains. most popular pre-trained lms are trained on data from the general domain. tailoring an lm towards a domain can be done via domain-specific pre-training from scratch (i.a., @xcite or adaptation of an existing model to target domain data with continued pre-training (i.a., @xcite @xcite . we refer to the survey of hedderich et al. (2021) for more information on language model adaptation for low-resource domains. in this paper, we take a different approach and investigate to which extend lms that were pre-trained on the general domain can be prompted for domain knowledge in few-shot settings as this requires only a minimal amount of domain-specific data. ## method we now present switchprompt, and give an example of an architecture in which it can be applied. in our architectural setup, the underlying pre-trained language model is fixed (i.e., not fine-tuned). ## experiments in this section, we describe the setup (datasets, training details and baselines) and the results of our experiments. ## analysis in this section, we report the results of our ablation study, give more insights into what our model learned and analyze its training time. we also provide a qualitative error analysis. ablation study. our ablation study in domain-specific keywords. the keywords are an integral part of switchprompt. since we compute keywords automatically (see section 3), we analyze the extracted keywords in more detail. training time analysis. during training time, the underlying lm is frozen in the switchprompt framework. this substantially reduces training time and computational memory, compared to alternative approaches, such as fine-tuning or p-tuning. figure 1 illustrates this. p-tuning v2 is a little bit faster than our approach as it does not need to train the gating parameters. however, the time difference is considerably small (2.2 min for 10 epochs in the all-data setting, i.e., 0.22 min per epoch). qualitative error analysis. finally, we manually conduct a qualitative error analysis on the gard dataset. the results are displayed in ## conclusion in this paper, we proposed a new methodology called switchprompt for effectively prompting pre-trained language models in low-resource domains. integral parts of our method are domainspecific keywords and gates, which allow the language model to dynamically retrieve domainspecific knowledge. experiments on sentence classification datasets from different domains show that our method outperforms various baseline methods in few-shot and all-data settings. in particular, it reduces the performance gap between general-domain and domain-specific language models. future work can investigate the impact on sequence-labeling tasks as well as on mixed-domain datasets.