Prototypical Verbalizer for Prompt-based Few-shot Tuning (2022)

Abstract: Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tuning wraps the input text into a cloze question. To make predictions, the model maps the output words to labels via a verbalizer, which is either manually designed or automatically built. However, manual verbalizers heavily depend on domain-specific prior knowledge and human efforts, while finding appropriate label words automatically still remains challenging. In this work, we propose the prototypical verbalizer (ProtoVerb) which is built directly from training data. Specifically, ProtoVerb learns prototype vectors as verbalizers by contrastive learning. In this way, the prototypes summarize training instances and are able to enclose rich class-level semantics. We conduct experiments on both topic classification and entity typing tasks, and the results demonstrate that ProtoVerb significantly outperforms current automatic verbalizers, especially when training data is extremely scarce. More surprisingly, ProtoVerb consistently boosts prompt-based tuning even on untuned PLMs, indicating an elegant non-tuning way to utilize PLMs. Our codes are avaliable at https://github.com/thunlp/OpenPrompt .

## introduction the massive-scale pre-trained language models (plms) @xcite have been proven to be backbones for solving a variety of nlp tasks @xcite . to further adapt these plms to downstream tasks such as classification, traditional approaches finetune the language models through an extra classifier @xcite . however, when task-specific data is limited @xcite , training the extra classifier effectively is challenging due to the gap between pre-training tasks (e.g., * corresponding author: z.liu (liuzy@tsinghua.edu.cn) a [mask] news: tokyo olympic daily preview, july 26th. masked language modeling) and fine-tuning tasks (e.g., classification and regression). this gap impedes the fast adaptation of plms to downstream tasks. recently, prompt-based tuning @xcite has risen to be a powerful way for few-shot learning by bridging the gap between the pre-training stage and downstream task stage. in prompt-based tuning, the input texts are wrapped with task-specific templates to re-formalize the original task as a cloze-style task. for example, in topic classification task, we can use template "<@xmath0, @xmath1. the plms are asked to infer the words to fill in @xcite and the words are further mapped to corresponding labels through a verbalizer (e.g. "sports" for label "sports"). verbaliz-ers are of great importance in prompt-based tuning @xcite since they are the bridges between model outputs and the final predictions. how to build effective verbalizers for prompt-based tuning-especially for many-class classification, is a critical issue in prompt-based tuning. typically, most current works adopt three kinds of verbalizers: manual verbalizers, search-based verbalizers, and soft verbalizers. we show them by an example in figure 1 . human-designed manual verbalizers pick some label words (e.g. label names) to depict classes. these verbalizers are powerful across multiple tasks @xcite . despite their success, a major drawback roots in the strong assumption that we own precise understandings of downstream tasks and are able to sum up each class with several words. without task-specific prior knowledge, selecting appropriate label words is non-trivial. further, they also need intensive human labors when facing many classes. to mitigate these issues, search-based verbalizers aim at finding suitable label words from vocabulary with algorithms @xcite @xcite and soft verbalizers use trainable tokens which are optimized during tuning @xcite . however, it is challenging to search or optimize adequately in a large vocabulary or embedding space under a low-data regime, making automatic verbalizers suboptimal compared with manual ones. intuitively, class proxies in verbalizers should encapsulate class-level semantic features, which are expressed implicitly by instances. to obtain these semantic representatives with few data, one promising approach is computing central points of class instances, namely prototypes, as approximation. to this end, we manage to estimate prototype vectors for each class to serve as verbalizer. summarized from instances, prototypes are supposed to establish concepts similar with human-designed labels. in this work, we introduce prototypes into this problem and propose prototypical verbalizer (pro-toverb), which learns class prototypes from training data to build verbalizers automatically. for prototype learning, inspired by the idea of pcl @xcite , protoverb trains the prototype vectors by contrastive learning with the infonce estimator @xcite . specifically, our optimization objective includes two components: the first part is an instance-instance loss to cluster intra-class instances and separate inter-class instances; the second part is an instance-prototype loss which enforces the prototypes to be center points of classes. compared with other verbalizer construction methods, protoverb learns continuous vectors straight from training instances efficiently, which makes it a plug-in-and-play algorithm with high flexibility. to verify the effectiveness of protoverb, we conduct extensive experiments on topic classification and entity typing tasks. we study two different settings where protoverb can work: (1) when manual verbalizers are available, protoverb can play as an extra verbalizer in the inference stage. results show that protoverb consistently improves the classification performance with low cost, and even untuned plms benefit largely. (2) consider a realistic setting where only a limited number of samples are provided with no manual verbalizers, protoverb also produces verbalizers of high quality. experimental results demonstrate that protoverb significantly outperforms existing search-based and soft verbalizers. ## background given a pre-trained language model m, our goal is to tune it for specific downstream tasks. take n way k shot few-shot text classification as an example, the support set for class n d @xmath2, • • • , x n k } contains k sentences. we aim to predict the label @xmath3, where y is the label set with n distinct classes. ## prototypical verbalizer in previous sections, we introduce the general pipeline of prompt-based tuning. as manually defining or automatically searching for appropriate verbalizers can be challenging, here we propose to learn prototypes directly from training instances. inspired by pcl @xcite , the prototypes are trained with contrastive learning. as shown in a [mask] news: stocks fall as oil hits high. a [mask] news: technology as fashion. a [mask] news: arsenal beats everton. a [mask] news: tokyo olympic daily preview, july 26th. sports tech world verbalizer labels template input figure 2 , we first get the hidden states of [mask] tokens to represent instances, then project them to another embedding space for prototype learning. the prototypes are used as verbalizers for prediction. next, we will introduce the learning and inference stages of protoverb in detail. ## experiments we conduct extensive few-shot learning experiments to illustrate the effectiveness of protoverb. in this section, we first introduce the experimental settings in use. then we present and discuss the experiment results. ## analysis in this section, we discuss several analytical topics for further understandings of protoverb. for simplicity, we conduct experiments on ag's news dataset. ## conclusion in this paper, we propose a novel approach for automatic verbalizer construction in prompt-based tuning. the proposed protoverb learns class prototypes from training instances using contrastive learning. we explore the performance of protoverb on few-shot topic classification and entity typing tasks. as a single verbalizer, protoverb outperforms state-of-the-art automatic verbalizers considerably. working together with manual verbalizers, protoverb can also consistently improve promptbased tuning with minor effort. the results validate the effectiveness of protoverb. our analysis further reveals the intrinsic properties of prototypes. for future work, we will focus on extending protoverb for effective non-tuning algorithms of plms and prompt-tuning with soft templates. moreover, we are finding proper ways to combine label words and prototypes for verbalizer construction. @xcite character-level convolutional networks for text classification. in proceedings of nips, volume 28, pages 649-657.