Exploring Lottery Prompts for Pre-trained Language Models (2023)

Abstract: Consistently scaling pre-trained language models (PLMs) imposes substantial burdens on model adaptation, necessitating more efficient alternatives to conventional fine-tuning. Given the advantage of prompting in the zero-shot setting and the observed performance fluctuation among different prompts, we explore the instance-level prompt and their generalizability.By searching through the prompt space, we first validate the assumption that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM, and such prompt can be obtained at a low cost thanks to the inherent ability of PLMs.Meanwhile, it is shown that some strong lottery prompts have high performance over the whole training set, and they are equipped with distinguishable linguistic features. Lastly, we attempt to generalize the searched strong lottery prompts to unseen data with prompt ensembling method. Experiments are conducted on various types of NLP classification tasks and demonstrate that the proposed method can achieve comparable results with other gradient-free and optimization-free baselines.

## introduction since pre-trained language models (plms) became the de-facto standard in modern nlp researches @xcite @xcite , the pretraining-finetuning paradigm has been prevailing until recent years when models keep scaling @xcite @xcite and become too expensive to be optimized. to this end, researchers are actively seeking more effective strategies that require little or even no optimization to harness plms. among these exploratory studies of advanced model adaptation, prompting (brown et al., * equal contributions † corresponding authors 2020; @xcite @xcite is gaining popularity in the community, which uses additional context (prompts) to wrap input instances and trigger desired output tokens. note that in this paper, the term "prompt" technically refers to the template that wraps the original input. in classification tasks, these tokens are further mapped to particular labels by a verbalizer. such a paradigm is verified to be effective in a variety of downstream tasks, even when annotations are insufficient. particularly, empirical evidence shows that coincidental prompts could achieve extraordinary performance in the zero-shot setting, i.e., no training examples are presented. for example, simple manual prompt can achieve an f1 score of over 60% on 46-class entity typing dataset @xcite and reaches 73% accuracy on dbpedia with 14 classes @xcite in the zero-shot setting. despite the promising performance of prompting, it is often accompanied by drastic fluctuations among different prompts @xcite . given the observed sensitivity and context-dependent nature of the prompting method, it is intuitive to assign distinct prompts to each instance to trigger the desired output. intrigued by this intuition, we explore a bold hypothesis: is it possible to find at least one instancelevel prompt that induces correct output for every data point (lottery prompt) in classification tasks without any optimization? we empirically show that after building an automatic searching procedure with reasonable searching space on 13 representative classification datasets of up to 66 classes, the existence of such lottery prompts can be validated ( § 2) . that is, the combination of just a few discrete tokens can make a plm output correct labels for almost any classification data. this finding updates our recognition of the limit of prompted knowledge in plms and demonstrates a promising upper bound of the plms' inference capability. with the hypothesis verified, we conduct further analysis on the internal mechanisms and properties of lottery prompts to explore how the lottery prompts relate to model capability and how lottery prompts generalize to unseen data without any optimization. in summary, we validate the existence of lottery prompts and conduct an in-depth analysis of the properties of lottery prompts. we also show that by directly ensembling the strong prompts, prominent performance can be achieved on test data without any optimization. our study points to the great potential of plms and is hoped to inspire future works in more efficient ways in searching and ensembling lottery prompts as an optimization-free adaptation of plms. ## the existence of lottery prompts for every data point considering the extraordinary performance observed on zero-shot classification and the large variance brought by the prompt selection, we make an assumption as follows: given a pre-trained language model and a classification dataset, for each instance, at least one lottery prompt exists that can induce the desired label from the plm, without the need to update the plm parameters. to validate the assumption, we conduct experiments that attempt to find the lottery prompt for every data point on 13 classification tasks. note that for different instances, the prompt may be different, and our goal is to verify the existence of such prompts in this experiment. ## empirical analysis since we have verified the existence lottery prompts, in this section, we conduct further analysis on search cost and the searched lottery prompts. ## explore the generalizability of strong prompts in § 2 and § 3, we have empirically verified that conditioned on a pre-trained model and a classification task, it is possible to find a lottery prompt for almost every data point, and that there are a handful of strong prompts that perform non-trivially on the training set. in this section, we first describe the ensembling method and then present the generalization results. ## conclusion in this work, we explore the existence of lottery prompts for every single instance and the adaptation of them for various classification tasks in an optimization-free manner. we propose a large prompt space composed of common words as the search space to verify the assumption. we also identify the searched strong prompts and the relation between model capacity and search cost and demonstrate the effectiveness of ensembling the strong prompts on the test set. our proposed optimizationfree method achieves satisfactory results on various nlp tasks under few-shot settings. above all, this work illuminates the fact that the great potential of plms can be successfully harnessed and prompted by plain textual prompts mined from plm vocabulary without parameter optimization and thus points to the need for future efforts in more efficient ways in mining and utilizing lottery prompts.