Parameter-free Automatically Prompting: A Latent Pseudo Label Mapping Model for Prompt-based Learning (2022)

Abstract: Prompt-based learning has achieved excellent performance in few-shot learning by mapping the outputs of the pre-trained language model to the labels with the help of a label mapping component. Existing manual label mapping (MLM) methods achieve good results but heavily rely on expensive human knowledge. Automatic label mapping (ALM) methods that learn the mapping functions with extra parameters have shown their potentiality. However, no effective ALM model comparable to MLM methods is developed yet due to the limited data. In this paper, we propose a Latent Pseudo Label Mapping (LPLM) method that optimizes the label mapping without human knowledge and extra parameters. LPLM is built upon a probabilistic latent model and is iteratively self-improved with the EM-style algorithm. The empirical results demonstrate that our LPLM method is superior to the mainstream ALM methods and significantly outperforms the SOTA method in few-shot classification tasks. Moreover, LPLM also shows impressively better performance than the vanilla MLM method which requires extra task-specific prior knowledge.

## introduction with the advent of the powerful pre-trained language model (plm) such as gpt-3 @xcite , prompt-based learning becomes blooming in recent years because it can effectively bridge the gap between pre-training tasks and downstream tasks @xcite . several corresponding studies show better performance than the traditional fine-tuning methods in few-shot learning tasks @xcite , where few-shot learning is generally considered as n-way k-shot learning. each task consists of n classes with k instances per class. the pipeline of vanilla prompt-based learning for classification tasks is composed of a prompting phase and a prediction phase. in the prompting phase, the input sentence is reconstructed into a cloze-type sentence with a pre-defined template containing @xmath0. in the prediction phase, the plm tries to fill @xmath1. one challenge in this pipeline is mapping the fill-in word with the expected class label. to address this challenge, label mapping is introduced into the pipeline, assigning a class representative to each class to associate the plm output with the expected class label. typically, each class representative is one or a set of human-chosen word(s) that should be highly relevant to each class. based on the word-occurrence probability, the model can calculate the occurrence probability of the class representatives. finally, the label of the input sentence is inferred from the class representative with the highest occurrence probability. as label mapping plays an important role in prompt-based learning @xcite , design-ing more effective label mapping becomes significant. the recent label mapping methods can be divided into two branches: manual label mapping (mlm) and automatic label mapping (alm). mlm method manually selects the class representative for each class, and it is proven to be very powerful in many tasks @xcite . however, this method is time-consuming and laborintensive. humans need to thoroughly understand the downstream tasks first and select the appropriate words for each class representative. additionally, relying heavily on human subjectivity can cause different problems. thus, alm methods are proposed to eliminate the effects of human involvement. they can be further divided into soft label mapping, search-based label mapping, and prototypical label mapping. soft label mapping @xcite replaces the concrete tokens with trainable tokens to generate class representatives. search-based label mapping @xcite aims to find a subset of candidate words from the vocabulary using algorithms or additional networks with additional parameters. prototypical label mapping @xcite projects @xmath2. however, these alm methods involve extra parameters which require a big amount of instances to train, where in practice, the available data is scarce in few-shot scenarios, making it difficult to train the model to find optimal class representatives. consequently, inexpressive class representatives may degrade the final classification performance. to address the above problem, we propose a latent pseudo label mapping (lplm) model. lplm first uses a majority voter that shares the same parameters with plm to automatically generate class representatives. as the class representatives produced by this method may be noisy, we build a latent variable model and introduce an em-style algorithm to gradually reduce the noise and enhance the expressiveness of each class representative, as shown in figure 1 foot_0 . the novel prompt-based learning pipeline with lplm has the advantage that no additional parameters or human knowledge are required to generate class representatives. in the e-step, the distributions of latent variables are updated with a majority voter and the class representatives are re-selected based on the parameters optimized in the previous m-step. in the m-step, predictions are made according to the generated class representatives from the e-step, and the parameters of the plm are updated based on the prediction results. the two steps perform alternately to obtain increasingly optimal distributions for class representatives and to improve the overall performance. moreover, we introduce two strategies for selecting the keywords in each class representative, which are discussed in section 3.3. to verify the effectiveness of lplm, we conduct a series of experiments on widely-used classification datasets. since our method can be applied to any classification task, we deliberately choose three different types of datasets, a sentiment classification dataset sst-2 (2-way), a well-known text classification dataset ag's news (4-way), and a popular entity typing dataset few-nerd (66-way). the experimental results show that lplm significantly outperforms the other three alms and even outperforms mlm. the contributions of this paper can be summarized as follows: ## latent pseudo label mapping in order to remedy the shortcomings of the methods mentioned in section 2.2, we introduce lplm to effectively exploit the knowledge provided by the limited instances. it is built upon a latent variable model and optimized with the em-style algorithm. in this section, we first define the necessary notations in few-shot classification tasks and introduce the overall architecture of lplm. then, we explain the detailed mechanism of the e-step and the m-step with the mathematical derivation and the learning process of the em-style algorithm. ## analysis in this section, we further analyze the details of lplm. for convenience, the following experiments are performed on ag's news dataset. ## conclusion in this paper, we propose a novel automatic label mapping model that excludes the reliance on human knowledge in manual label mapping methods by automatically generating a set of keywords as each class representative. to increase the distinction of the class representatives, we further introduce an em-style algorithm to optimize the distribution of latent variables, namely the class representatives, to discover better class representatives and improve the overall classification performance.