Prompt Discriminative Language Models for Domain Adaptation (2023)

Abstract: Prompt tuning offers an efficient approach to domain adaptation for pretrained language models, which predominantly focus on masked language modeling or generative objectives. However, the potential of discriminative language models in biomedical tasks remains underexplored.To bridge this gap, we develop BioDLM, a method tailored for biomedical domain adaptation of discriminative language models that incorporates prompt-based continual pretraining and prompt tuning for downstream tasks. BioDLM aims to maximize the potential of discriminative language models in low-resource scenarios by reformulating these tasks as span-level corruption detection, thereby enhancing performance on domain-specific tasks and improving the efficiency of continual pertaining. In this way, BioDLM provides a data-efficient domain adaptation method for discriminative language models, effectively enhancing performance on discriminative tasks within the biomedical domain.

## introduction recent years witnessed the development of biomedical pretrained language models (plms) @xcite . these domain-specific plms contribute to a large number of downstream tasks in the biomedical domain, such as named entity recognition @xcite @xcite , entity linking @xcite , relation extraction @xcite , and question answering @xcite . most existing domain-specific plms rely on tremendous in-domain corpus and computing resources for continual pretraining @xcite @xcite or pretraining from scratch @xcite , which could be infeasible with limited resources. meanwhile, plms figure 1 : a case for prompting discriminative pretrained language models (dlms) on multi-choice biomedical question answering. each option is first concatenated with a predefined hard prompt: "the answer is". they are separately concatenated with the context and question as input. we rank the score from the head of replaced token detection (rtd) in dlms to determine the best option. for general purposes usually fails to achieve comparable performance on biomedical tasks with finetuning compared with in-domain plms at the same model scale @xcite . to combat these issues, exploring a prompt-based domain adaptation method that better leverages existing knowledge learned in pretaining is necessary. recent research demonstrates that prompts or instructions can activate the hidden abilities of plms @xcite @xcite , including cross-domain inference @xcite @xcite . therefore, prompt tuning on general plms can be a data-efficient domain adaptation method as they are proven promising on various downstream tasks @xcite . existing explorations about prompt-based domain adaptation mainly focus on plms with masked language modeling @xcite or generative objectives @xcite . however, we identify that discriminative pretrained language models (dlms) also hold great potential for prompt-based domain adaptation but remains understudied. dlms are pretrained to distinguish between alternatives and proved to be stronger few-short learners than plms with other training objectives @xcite . therefore, dlms are better choices for domain adaptation since many downstream tasks in the biomedical domain focus on discriminative objectives @xcite . however, complex model architecture and training recipes hinder dlms from efficient adaptation to other domains. to shed light on this topic, we develop biodlm (prompt-based biomedical domain adaptation for discriminative language models), which can efficiently take advantage of the state-of-the-art dlms in the general domain. biodlm is a prompt-based biomedical domain adaptation method designed explicitly for dlms, including prompt-based continual pertaining and prompt tuning for downstream tasks. inspired by @xcite , we first formulate discriminative downstream tasks in the biomedical domain, such as multi-choice question answering, as span-level corruption detection. as shown in fig. 1 , this prompt tuning reformulation allows general-domain dlms to be used as zero-shot or few-shot learners in biomedical tasks, which is also supported by our probing experiments in §4.2. we develop an efficient prompt-based continual pretraining method to further enhance the performance of dlms on biomedical tasks. as bajaj et al. (2022) revealed, the selection of corrupted tokens and the corruption methods play a vital role in pretraining dlms and is highly related to the performance on downstream tasks. biodlm selects domain-specific words, defined as different vocabulary between in-domain and general models, as corrupted tokens to lead the continual pretraining focusing on new domain knowledge and improve pretraining efficiency. for corruption, biodlm employs fixed in-domain plms as encoders to corrupt selected tokens instead of co-training encoders and decoders in dlms. biodlm is a flexible domain adaptation method that can be applied to any existing dlms. the contributions of this work are mainly twofold. first, we explore prompt tuning generaldomain dlms on various biomedical downstream tasks, showing prompting dlms has significant potential on these tasks under low-resource scenarios. second, we develop a data-efficient continual pretraining method based on replaced token detec-tion, which employs in-domain plms as generators to corrupt domain-specific words in the biomedical corpus. in summary, biodlm efficiently improves low-resource performance on discriminative tasks in the biomedical domain. prompt tuning for dlms. prompt tuning for dlms is an emerging topic in general and biomedical domains. @xcite presents empirical evidence showing that electra can perform well on downstream tasks without fine-tuning or additional training. @xcite introduces a prompt-based fine-tuning approach that leverages discriminative prompts to guide the model towards learning specific downstream tasks with only a few examples. @xcite proposes a few-shot learning approach with pre-trained token-replaced detection models to transform traditional classification and regression tasks into token-replaced detection problems. @xcite suggests fine-tuning dlms with prompts for task-specific downstream tasks by adding a small number of task-specific parameters as a prompt to guide the model's output. however, these works are limited to a single method electra and do not explore biomedical tasks. we follow the recipe of prompt tuning in @xcite and use it on biomedical discriminative tasks. ## methods we describe preliminaries ( §3.1), prompt-based continual pretraining with rtd ( §3.2), and prompt tuning for discriminative plms ( §3.3). ## experiments this section introduces an experimental evaluation of prompting discriminative plms for biomedical domain adaptation. we describe the experimental setup ( §4.1), main results ( §4.2), and ablation study ( §4.3) on incorporated techniques. ## conclusion we study an efficient way to adapt general-domain dlms to the biomedical domain and propose biodlm. biodlm consists of data-efficient continual pretraining that focuses on domain-specific vocabulary and leverages domain knowledge in the in-domain plms by employing them as rtd generators. we also conduct experiments on a biomedical benchmark with six biomedical datasets, verifying that prompt tuning is an effective way to adapt dlms on biomedical discriminative tasks directly. future works include extending biodlm to more dlms, such as electra @xcite and coco-lm @xcite , and experimenting with biodlm on other discriminative tasks in the biomedical domain.