Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2022)

Abstract: Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompttuning (KPT), to improve and stabilize prompttuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.

## introduction recent years have witnessed the prominence of pretrained language models (plms) @xcite @xcite @xcite due to their superior performance on a wide range of languagerelated downstream tasks such as text classification @xcite , question answering @xcite , and machine reading comprehension @xcite . to fathom the prin-ciples of such effectiveness of plms, researchers have conducted extensive studies and suggested that plms have obtained rich knowledge during pre-training @xcite . hence, how to stimulate and exploit such knowledge is receiving increasing attention. one conventional approach to achieve that is fine-tuning @xcite , where we add extra classifiers on the top of plms and further train the models under classification objectives. fine-tuning has achieved satisfying results on supervised tasks. however, since the extra classifier requires adequate training instances to tune, it is still challenging to apply fine-tuning in few-shot learning @xcite and zero-shot learning @xcite scenarios. originated from @xcite and lama @xcite , a series of studies using prompts @xcite for model tuning bridge the gap between pre-training objective and down-stream tasks, and demonstrate that such discrete or continuous prompts induce better performances for plms on few-shot and zero-shot tasks. a typical way to use prompts is to wrap the input sentence into a natural language template and let the plm conduct masked language modeling. for instance, to classify the topic of a sentence x: "what's the relation between speed and acceleration?" into the "science" category, we wrap it into a template: "a [mask] question: x". the prediction is made based on the probability that the word "science" is filled in the "[mask]" token. the mapping from label words (e.g., "science" ) to the specific class (e.g., class science) is called the verbalizer @xcite , which bridges a projection between the vocabulary and the label space and has a great influence on the performance of classification @xcite . most existing works use manual verbalizers @xcite , in which the designers manually think up a single word to indicate each class. to ease the human effort of designing the class name, some works propose to learn the label words using discrete search @xcite or gradient descent @xcite . however, the learnedfrom-scratch verbalizer, lack of human prior knowledge, is still considerably inferior to the manual verbalizers (see appendix a for pilot experiments), especially in few-shot setting, and even not applicable in zero-shot setting, which leaves the manual verbalizer a decent choice in many cases. however, manual verbalizers usually determine the predictions based on limited information. for instance, in the above example, the mapping {science}→ science means that only predicting the word "science" for the @xcite token is regarded as correct during inference, regardless of the predictions on other relevant words such as "physics" and "maths", which are also informative. such handcrafted one-one mapping limits the coverage of label words, thus lacking enough information for prediction and introducing bias into the verbalizer. therefore, manual verbalizers are hard to be optimal in text classification, where the semantics of label words are crucial for predictions. the optimization-based expansion, though can be combined with manual verbalizers to yield better performance, only induces a few words or embeddings that are close to the class name in terms of word sense or embedding distance. thus they are difficult to infer words across granularities (e.g. from "science" to "physics"). if we can expand the verbalizer of the above example into {science, physics} → science, the probability of making correct predictions will be considerably enhanced. therefore, to improve the coverage and reduce the bias of the manual verbalizer, we present to incorporate external knowledge into the verbalizers to facilitate prompt-tuning, namely, knowledgeable prompt-tuning (kpt). since our expansion is not based on optimization, it will also be more favorable for zero-shot learning. specifically, kpt contains three steps: construction, refinement, and utilization. (1) firstly, in the construction stage, we use external kbs to generate a set of label words for each label (in § 3.2). note that the expanded label words are not simply synonyms of each other, but cover different granularities and perspectives, thus are more comprehensive and unbiased than the class name. (2) secondly, to cope with the noise in the unsupervised expansion of label words, we propose four refinement methods, namely, frequency refinement, relevance refinement, contextualized calibration, and learnable refinement (in § 3.3), whose effectiveness is studied thoroughly in § 4. (3) finally, we apply either a vanilla average loss function or a weighted average loss function for the utilization of expanded verbalizers, which map the scores on a set of label words to the scores of the labels. we conduct extensive experiments on zero-shot and few-shot text classification tasks. the empirical results show that kpt can reduce the error rate of classification by 16%, 18%, 10%, 7% on average in 0, 1, 5, 10 shot experiments, respectively, which shows the effectiveness of kpt. in addition to the performance boost, kpt also reduces the prediction variances consistently in few-shot experiments and yields more stable performances. ## related work two groups of research are related to kpt: prompttuning, and the verbalizer construction. prompt-tuning. since the emergence of gpt-3 (brown et al., 2020), prompt-tuning has received considerable attention. @xcite demonstrates that with prompt-tuning and incontext learning, the large-scale language models can achieve superior performance in the low-data regime. the following works @xcite argue that small-scale language models @xcite @xcite can also achieve decent performance using prompt-tuning. prompt-tuning has been applied to a large variety of tasks such as text classification @xcite , natural language understanding @xcite , relation extraction @xcite , and knowledge probing @xcite , etc. verbalizer construction. as introduced in § 1, the verbalizer is an important component in prompt-tuning and has a strong influence on the performance of prompt-tuning @xcite . most works use humanwritten verbalizers @xcite , which are highly biased towards personal vocabulary and do not have enough coverage. some other studies @xcite @xcite design automatic verbalizer searching methods for better ver-science, mathematics, biology, research, knowledge, physics, knowledge, innovate, calculation, ..., agrobiology, mscience, euclid, orthogon, einstein, bioacoustics, chemo, axiom, nomenclature what's the relation between speed and acceleration? a [mask] : science construct knowledgeable verbalizer refine knowledgeable verbalizer science, mathematics, biology, research, knowledge, physics, knowledge, innovate, calculation, ..., agrobiology, mscience, euclid, orthogon, einstein, bioacoustics, chemo, axiom, nomenclature physics mathematics science calculation research mlm head original input template and the above part is the construction, refinement and utilization processes of kpt . ## knowledgeable prompt-tuning in this section, we present our methods to incorporate external knowledge into a prompt verbalizer. we first introduce the overall paradigm of prompttuning and then elucidate how to construct, refine and utilize the knowledgeable prompt. ## experiments we evaluate kpt on five text classification datasets to demonstrate the effectiveness of incorporating external knowledge into prompt-tuning. ## analysis ablation studies about our refinement methods have been shown in the previous section. in this section and appendix d, we conduct more in-depth analyses on the proposed methods. ## conclusion in this paper, we propose kpt , which expands the verbalizer in prompt-tuning using the external kb. to better utilize the kb, we propose refinement methods for the knowledgeable verbalizer. the experiments show the potential of kpt in both zero-shot settings and few-shot settings. for future work, there are open questions related to our research for investigation: (1) better approaches for combining kb and prompt-tuning in terms of template construction and verbalizer design. (2) incorporating external knowledge into prompt-tuning for other tasks such as text generation.