Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification (2022)

Abstract: Prompt-based learning (i.e., prompting) is an emerging paradigm for exploiting knowledge learned by a pretrained language model. In this paper, we propose Automatic Multi-Label Prompting (AMuLaP), a simple yet effective method to automatically select label mappings for few-shot text classification with prompting. Our method exploits one-to-many label mappings and a statistics-based algorithm to select label mappings given a prompt template. Our experiments demonstrate that AMuLaP achieves competitive performance on the GLUE benchmark without human effort or external resources.

## introduction since the release of @xcite , several studies have focused on exploiting pretrained language models with only a few training examples @xcite @xcite . these works demonstrate the potential of using natural language prompts to encourage the model to recall similar patterns in its training corpus and thus make accurate predictions. this setting of few-shot learning is closer to how humans learn to solve a task, often without many examples as in a traditional deep learning paradigm. the use of prompts can strengthen the explicit connection between input and output, helping the model exploit the knowledge learned from pretraining in a better way. furthermore, recent works @xcite show that prompts can also help the model generalize better in fine-tuning. prompt-based learning (i.e., prompting) aims to use a template to convert the original input into a prompt-based input with some unfilled masked tokens, and then use the pretrained language model to fill these masked tokens, and finally the tokens filled into these slots are mapped to the corresponding labels as the final output. in prompting, the design of prompts often plays an important role. many attempts have been made in this emerging direction of prompt engineering @xcite . meanwhile, finding a good mapping from the original task labels to tokens (i.e., label engineering) is also critical to few-shot performance, as found in @xcite ; @xcite . however, manually assigning the label mapping requires human expertise with trial and error. one may argue that the same effort can be used to label more supervised data for a conventional deep learning pipeline. thus, an efficient automatic label mapping method is desirable. in this paper, we aim to design a method that can automatically find a good label mapping to save human effort from label engineering. we propose automatic multi-label prompting (amu-lap), a simple yet effective method to tackle the label selection problem for few-shot classification. amulap is a parameter-free statistical technique that can identify the label patterns from a few-shot training set given a prompt template. amulap exploits multiple labels to suppress the noise and inherently extend the training set for prompt-based fine-tuning. compared with a hand-crafted label mapping and previous works on automatic label mapping @xcite , amulap achieves competitive performance despite being simpler and does not require access to the weights of the backbone model, or finetune an external pretrained language model for searching label mapping. we conduct extensive experiments and demonstrate the effectiveness of our method under multiple settings. moreover, we attempt to scale amulap with different sizes of the training set and find amulap to work surprisingly well even with one or two shots. we further analyze why does amulap work and discuss the pros and cons of prompting as a new paradigm. instead of manually designing prompts (i.e., prompt engineering), some recent studies also explore automatic prompt generation. petal @xcite augments pattern exploiting training (pet, @xcite with automatically identified label words; @xcite uses re-ranking to find the best label words by fine-tuning a roberta model on the candidates searched by roberta, and using an external generation model for data augmentation of prompt templates; autoprompt @xcite uses a gradient-based search to determine both prompts and label words. however, these methods require parameter updates with gradient descent, which is infeasible without access to the model weights (e.g., gpt-3). pet and its variants also require a large unlabeled set and need to be fine-tuned multiple times. autoprompt uses discretization techniques to approximately map a continuous vector back to tokens in the vocabulary (i.e., "vocablization"). these searched prompts and labels are often uninterpretable by humans. different from these prior studies, our proposed amulap is a simple and interpretable method for few-shot prompting that can work well with and without access to model weights. concurrently to our work, @xcite propose a method that exploits an external knowledge base to find label mapping. t0 @xcite constructs a dataset of different nlp tasks by manually writing prompt templates and shows that a large language model with multitask training can generalize to unseen tasks. ## prompting for few-shot classification we follow the setup in lm-bff @xcite for few-shot text classification. given a pretrained language model l, a task d and its defined label space y, we have n training examples per class for the training set d train . as pointed out in @xcite , using the full development set may be misleading to claim a few-shot setting. thus, we use a few-shot development set with the same size as the training set (i.e., |d train | = |d dev |), to be consistent with @xcite and constitute a "true few-shot" setting @xcite . for an input example x (a single sentence or a sentence pair), we first use a task-specific template t to convert it to x ′ , a token sequence with a [mask] token. we then map the original label space to a set of selected words from the vocabulary, denoted as m : @xmath0. some examples of t and m are shown in alternately, one can further fine-tune l with supervised pairs {x ′ , m (y)} to achieve even better performance. ## analysis 6.1 case study as shown in ## discussion why does amulap work? @xcite argues that one single label sometimes cannot represent all examples in a class, and thus multiple labels are needed. however, we find this explanation insufficient for understanding the mechanism behind the improved performance with multiple labels. under a few-shot setting, the limited number of training examples n and complex training procedure of the backbone model l can often bring noise to both automatic label selection and inference. one example is the @xmath2, as shown in as we analyze, increasing k harms the overall quality of selected labels and thus overrides the benefit of a larger k. in general, we do not observe a clear law for choosing the best k for amulap. as mentioned before, k can influence both the overall quality of labels (in both ways) and the training procedure (for fine-tuning). thus, for the optimal performance, we find it essential to search k with a development set. limitations and future directions in this paper, we only focus on the selection of the label mapping with a fixed prompt template. there is more to explore when considering the prompt template at the same time. similar to our paper, previous works @xcite separately search for a prompt template t and the label mapping m. however, these two variables are closely related and greedily search for the best template t then the best mapping under t may be suboptimal. jointly searching for t and m could be a promising direction for future research. more broadly, we would like to point out some limitation and contradictions within current fewshot prompting techniques. there is a natural contradiction between performance and access to the model weights. brown et al. (2020) highlights few-shot prompting as a way to mitigate their decision to not release the model weights. however, as shown in our in addition, complicated prompting techniques are not practically useful for real-world scenarios. for most techniques, the required effort for finding good templates and label mappings, and sometimes training models outweighs the cost of simply labeling more training examples. as shown in figure 2 , 64 examples per class are enough to bring the performance of standard fine-tuning to the same level of prompting. although recent works on automatic selection of prompts and label mappings are making meaningful contribution to the practicability of few-shot learning, we believe more work should be done to simplify the learning procedure and eliminate human effort while achieving good performance.