Pre-trained Language Models Can be Fully Zero-Shot Learners (2023)

Abstract: How can we extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text entailment, similar text retrieval, paraphrasing, and multiple-choice question answering. Experimental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 15.6% on the GLUE benchmark. Our source code is available at https://anonymous.4open.science/r/NPPrompt .

## introduction natural language understanding (nlu) has been important in many applications such as intelligent dialog assistants, online search, and social media analysis. recent advancement of nlu has been driven by emergent pre-trained language models (plms) including bert @xcite , gpt @xcite @xcite , bart @xcite , and t5 @xcite . prior studies show that plms obtain substantial knowledge during pretraining on raw text corpus @xcite . by fine-tuning on taskspecific labeled data, plms exploit such knowl-edge and gain impressive accuracy on a wide range of nlp tasks, such as text classification @xcite , question answering @xcite , machine reading comprehension @xcite , etc. however, fine-tuning approaches are expensive. it requires labeled datasets, which are rarely available for many tasks. significant computational efforts are needed to update plms' parameters for multiple tasks. in addition, fine-tuning results in one distinct model for each task to maintain. how can we generalize a pre-trained model to many nlp tasks, without labeled or additional unlabeled data? existing few-shot and zero-shot approaches propose to construct prompts to elicit desired predictions from plms @xcite . the main idea of prompting plms is to convert an input utterance to one with masked templates. for example, in text classification an input can be "the warriors won the nba championship 2022" and it is instead converted to "the warriors won the nba championship 2022. this topic is about [mask]". a plm (e.g. bert) takes the converted text and produces predictions for the masked token, along with the probability. ideally, a plm will generate a higher probability for the word "sports" than "politics" on the [mask] token. although these prompting-based methods are effective, they require unlabeled data for training or huge human efforts to construct prompts and to choose designated tokens to represent class labels @xcite . in addition, these manually constructed verbalizers, i.e. mapping from words (e.g. "basketball") to class labels (e.g. sports), do not extend to new emerging categories after plms are deployed. in this paper, we investigate the fully zero-shot learning problem for nlu where only the target label names are available but not the extra raw text. we propose nonparametric prompting plm (npprompt), a novel method to generate predic-tions for semantic labels without any fine-tuning. npprompt uses plm's own embeddings to automatically find relevant words to labels (e.g. "basketball" and "nba" for sports), therefore it does not need humans to construct verbalizers. our key idea is to search for the top k nearest neighbors to a label name in the embedding manifold and then generate and aggregate plm's predicted logits from masked prompts. in the above case, both predicted values for "basketball" and "nba" contribute to the final prediction for the sports category. in this way, npprompt can be easily generalized to any new categories as long as the category names are semantically meaningful. the contributions of this paper are as follows. a) we develop npprompt, a novel method for fully zero-shot learning with plms. b) we conduct extensive experiments on diverse language understanding tasks including text classification, text entailment, similar text retrieval, paraphrasing, and multiple-choice question answering. experimental results show that npprompt outperforms the previous zero-shot methods by absolute 12.8% in accuracy on text classification and 15.6% on the glue benchmark. surprisingly, npprompt is on a par with the best prior method that trained with manual verbalizers, an additional knowledge base, and extra unlabeled data. ## related work prompting the success of @xcite has attracted much attention to prompting engineering, a new way to leverage pre-trained language models. (brown et al., 2020) concatenate a few input and output pairs and feed them to the large-scale gpt-3 language model, which is an intuitive in-context learning paradigm, allowing the model to generate answers for additional cases autoregressively. recent works @xcite show that small-scale pretrained language models such as bert @xcite , roberta @xcite and al-bert @xcite can also achieve decent performance using prompt-tuning. prompting has been applied to a large variety of tasks such as text classification @xcite , natural language understanding @xcite , knowledge probing @xcite , and relation extraction @xcite . typically, a piece of prompt contains a template and a verbalizer. the language model predicts a probability distribution over vocabulary given the template and the verbalizer transforms it into a prediction over class labels. in this work, we focus on designing the verbalizers automatically. verbalizer design the verbalizer plays a crucial role in prompting as it connects model outputs and labels, significantly influencing performance. @xcite design human written verbalizers for prompting, however, they are highly biased towards personal vocabulary with inadequate coverage. apart from manually designed verbalizers, some recent studies explore automatic verbalizer construction. auto-l @xcite uses re-ranking to find the label words set by finetuning the model on the candidates searched by roberta; autoprompt @xcite applies gradient-based search to create both prompts and label words automatically with a few trigger examples. but these approaches need to update parameters with gradient descent, which turns out to be infeasible without access to the model weights (e.g., gpt-3). kpt @xcite incorporates external knowledge into the verbalizer in which the unlabeled dataset is needed to refine the label words and thus is not applicable to scenarios where only label names are known. in contrast, our approach npprompt directly finds, without any gradient update, relevant words to label names with plm's initial word embedding only. zero-shot text classification general zero-shot text classification typically focuses on classifying texts into categories that were not seen during the training process. transferring knowledge from seen classes to unseen ones requires accurate and discriminative descriptions of all classes @xcite or joint embeddings of categories and documents @xcite . however, these methods rely on supervised data for the known label set, making them unsuitable for scenarios where no labeled pairs for any category are available. simptc @xcite improves zero-shot classification by clustering input texts and employing class-related prompts. lotclass @xcite proposes a model that utilizes label names with self-training for zero-shot classification. nonetheless, both simptc and lot-class still require an unlabeled corpus or knowledge base to extract topic-related words and perform self-training. in contrast, npprompt achieves comparable or even superior performance without the need for any unlabeled dataset or knowledge base. ## background: prompt-based tuning for plms we first provide standard paradigms, prompt-based tuning, that perform well in few-shot scenarios, before introducing our approach for the zero-shot case. take n way text classification as an example. we aim to predict the label @xmath0, where y is the label set with n distinct classes. prompt-based tuning tunes plm using customized prompts @xcite . the regular prompt-based tuning converts a specific task to a cloze-style mask language modeling problem. for each input example x (single sentence or sentence pair), we first apply a task template t on it, converting original input x to x prompt . for instance, we concatenate the template "t (•) = this topic is about [mask]" with the original input "the warriors won the nba championship 2022" and wrap it into: this topic is about @xcite the verbalizer f in vanilla prompt engineering maps a set of selected words v from the vocabulary to the original label space y, i.e., f : @xmath1. inversely, we use m(y j ) to denote the label words in v that are mapped into a specific label y j , ∪ y @xmath2. then we calculate the probability of label y j : ## proposed method: npprompt we inherit plm with verbalizers framework but keep plm's parameters frozen @xcite . the key idea of npprompt is using plm's word embeddings to automatically construct verbalizers -mapping from words to labels -in a fully zeroshot way. it does not need any additional raw text corpus for fine-tuning. npprompt consists of two steps to compute predictions for any labels in a nonparametric form (figure 1 ). 1) we search for all label words closely related to each class y j in plm's token embedding manifold. 2) then we use the plm to predict values for [mask], filter them using each class's set of label words, and aggregate the properly weighed outputs to produce the final prediction. in the following, we describe npprompt for text classification but it generalizes to other language understanding tasks. ## discussion our proposed method, npprompt, demonstrates exceptional performance in zero-shot text classification tasks. we attribute this success to two key factors. firstly, by utilizing the initial word embedding from pre-trained language models (plms), we are able to identify cognates of the label words. for instance, in ## conclusion in this paper, we propose npprompt, a novel and effective method for fully zero-shot learning with pre-trained language models. we use initial word embedding of plm to automatically find related words for category names, which enables us to construct the verbalizers without manual design or unlabeled corpus. experimental results show that npprompt outperforms the previous zero-shot methods by large margins.