Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words (2022)

Abstract: Prompt-based fine-tuning for pre-trained models has proven effective for many natural language processing tasks under few-shot settings in general domain. However, tuning with prompt in biomedical domain has not been investigated thoroughly. Biomedical words are often rare in general domain, but quite ubiquitous in biomedical contexts, which dramatically deteriorates the performance of pre-trained models on downstream biomedical applications even after fine-tuning, especially in low-resource scenarios. We propose a simple yet effective approach to helping models learn rare biomedical words during tuning with prompt. Experimental results show that our method can achieve up to 6% improvement in biomedical natural language inference task without any extra parameters or training steps using few-shot vanilla prompt settings.

## introduction pre-trained models have achieved a great success in natural language processing (nlp) and become a new paradigm for various tasks @xcite @xcite . many studies have paid attention to pretrained models in biomedical nlp tasks @xcite @xcite . however, plain pre-trained models sometimes cannot do very well in biomedical nlp tasks. in general, there are two challenges to fully exploit the potential of the pre-trained models for biomedical nlp tasks, i.e., (1) limited data and (2) rare biomedical words. firstly, it is common that the amount of biomedical labeled data is limited due to strict privacy policy constraints @xcite , high cost and professional requirement for data annotation. pre-trained models perform poorly with few samples since abundant training samples are essential to optimize task-related parameters @xcite . secondly, biomedical words are usually low-frequency words but critical to understanding biomedical texts. as an example of natural language inference (nli) task in figure 1 , the model goes wrong during tuning when faced with a rare word "afebrile" foot_0 in the premise, whose meaning is "having no fever". it can be no easy for the pretrained models to predict the correct label if the models haven't seen the rare biomedical words for enough times during pre-training or tuning stage. thus, pre-trained models cannot capture the precise semantics of biomedical texts in the scenario of low-resource tasks. with very few annotated samples available for a new task, it is hard to effectively fine-tune pretrained models with the additional task-specific parameters, which is even more of a challenge to biomedical domain as mentioned above. prompt technique has been introduced to smooth the finetuning process in the few-shot settings by narrowing down the gap between pre-training stage and the downstream task in general domain @xcite , as demonstrated in figure 2 . therefore, it is beneficial to adapt prompt-based tuning to biomedical nlp tasks. although the challenge of rare words is a critical problem for the biomedical pre-trained models, only a handful of works have studied the issue and most of them focus on enriching the representation of rare words through pre-training stage @xcite @xcite . thus, it naturally requires them to involve a secondround pre-training or further training steps with biomedical knowledge to achieve the above goal, which is highly time-consuming and inefficient. alternatively, we emphasize on tuning stage instead of pre-training to resolve these issues. when coming across an unknown word, human may seek the task: ## method in this section, we introduce how we find the rare biomedical words and append paraphrases to the rare biomedical words with the prompt-based tuning of pre-trained models in a model-agnostic plugin manner. ## conclusion rare biomedical words are pervasive in biomedical texts, and understanding domain-specific rare words remains a tough challenge for pre-trained models. in this work, we presented a simple yet effective method to help the pre-trained models grasp the semantics of rare biomedical words. enlightened by human reading behavior, we taught the pre-trained models to understand rare biomedical words by incorporating paraphrases of rare biomedical words. our method can be regarded as a generic plug-in approach for promptbased tuning without additional parameters. experiments showed that our method could substantially improve the pre-trained models under few-shot settings.