What Makes Pre-trained Language Models Better Zero-shot Learners? (2023)

Abstract: Current methods for prompt learning in zero-shot scenarios widely rely on a development set with sufficient human-annotated data to select the best-performing prompt template a posteriori. This is not ideal because in a real-world zero-shot scenario of practical relevance, no labelled data is available. Thus, we propose a simple yet effective method for screening reasonable prompt templates in zero-shot text classification: Perplexity Selection (Perplection). We hypothesize that language discrepancy can be used to measure the efficacy of prompt templates, and thereby develop a substantiated perplexity-based scheme allowing for forecasting the performance of prompt templates in advance. Experiments show that our method leads to improved prediction performance in a realistic zero-shot setting, eliminating the need for any labelled examples.

## introduction prompt learning has been demonstrated to be a successful remedy for challenges associated with pre-training and fine-tuning paradigm, especially in zero/few-shot scenarios @xcite @xcite . research has repeatedly shown that various transformer-based language models can benefit from prompt learning. for example, decoder-only models, such as those in the gpt family @xcite , can better generalise to unseen cases by prefixing inputs with a few training examples (in natural language). this is known as in-context learning @xcite @xcite . encoder-decoder models, such as t5 @xcite or bart @xcite , can leverage prompt learning to train versatile models for multiple tasks (khashabi et al., + work was done during internship at sensetime research * corresponding author 2020; @xcite . bidirectional encoderonly models, such as those in the bert family @xcite , can also manifest impressive zero-shot capacity when given proper prompts. these prompts often take the form of pre-training tasks, such as next sentence prediction @xcite or masked language modeling (mlm) @xcite @xcite -also known as clozestyle prompt learning. despite its success in encoder-only models, cloze-style prompt learning is sensitive to the specific involved templates. multiple studies have shown that the design and choice of prompt templates greatly affect the effectiveness of zero-shot learning @xcite @xcite . ideally, they are supposed to be as close as possible to the language used in downstream task. for example, in a sentiment analysis task, a suitable template may be "[very/not] pleased." that carries emotional information. however, other templates can also be used here like "[very/not] good.". as shown in current prompt learning methods still rely on a development set of human-annotated data for post-hoc template selection @xcite @xcite : all candidate templates are evaluated using the development set and the best-performing one is chosen. this requires human annotators and does not align well with realistic zero-shot learning scenarios in which no human-annotated data is available. to address this problem, we propose a truly annotationfree perplexity-based template selection method for zero-shot prompt learning: perplexity selection (perplection). experiments show that perplection is highly likely to select the most effective template accommodating true zero-shot scenarios. in this paper, we first describe cloze-style prompt learning and corresponding terminologies in section 2. then, in section 3, we present our hypothesis that underpins the work. based on this hypothesis, in section 4 we detail perplection that uses perplexity to select templates a priori without the need of any annotated examples. section 5 describes a pilot study and in section 6, we present realistic experiments that show that perplection leads to performance on par with other zero-shot prompt methods that utilise a development set. finally, section 7 discusses the underlying rationales and the potential impact of the work in a large language models (llm) era. to the best of our knowledge, we spearhead the performance screening of prompt templates for a realistic zero-shot text classification without using any human-annotated data. * ## preliminaries in this section, we describe basic concepts and terminologies associated with prompt learning. ## hypotheses this section proposes two hypotheses that underpin our work, and describes the way they interpret observations in the literature. ## method as discussed in section 3, a heuristic approach can be employed to select the most effective templates in zero-shot text classification. one way to do this is to utilise language discrepancy to "forecast" the performance of different prompt templates. specifically, the prompt template that results in the lowest language discrepancy when prefixed to a given input text can be considered the most effective. however, how can the language discrepancy between downstream text and pre-training corpora be measured? in this study, we propose using perplexity @xcite as an approximation of language discrepancy. perplexity is one of the most common metrics for evaluating language models, and is defined as the exponential average negative log-likelihood of a sequence: ## pilot study the aim of the pilot study described in this section was to qualitatively validate the hypotheses proposed in section 3, and to examine the utility of perplexity as a metric for screening prompt templates (another study that examines the utility of perplexity is presented in appendix d). to this end, we manually curated four prompt templates as shown in ## experiments in this section, we demonstrate the proposed perplection approach in a more realistic and useful experimental setting to verify whether we can use language discrepancy to forecast the efficacy of ## discussion what contributes better zero-shot learners? this work empirically reveals that the large language discrepancy between the pre-training corpora and the downstream data may hinder the zeroshot generalization. on top of that, we develop a perplexity-based scheme that leverages cloze-style prompt templates to bridge language discrepancy and thus, fully releases the potential of pre-trained language models. the significance of this work lies in its pioneering study of a feasible objective for optimising realistic zero-shot prompting templates. the idea may be applied to various variations (e.g., continuous prompts) beyond the discrete prompts currently being studied. why realistic zero-shot matters? in this work, we constantly emphasise a realistic zero-shot scenarios (no labelled data), as opposed to the existing zero-shot setting in the field of nlp @xcite or multi-modality @xcite , where a development set is available for template selection or hyper-parameter tuning. realistic zero-shot can be quite appealing for industrial scenarios and thus, this research opens up a new avenue for research in the field of zero-shot learning, probably inspiring follow-up studies in broader tasks for advancing the zero-shot learning in industrial applications (especially in many low-resource scenarios). potential impact in the llm era. in light of the advancements in large language models (llm) based on the decoder-only architecture @xcite , searching for effective instructions or incontext demonstration examples @xcite has become an essential challenge. perplection can be seamlessly applied to decoderonly models for searching effective instructions/incontext examples for various natural language gen-eration (nlg) tasks. we make our code available for replication and further extension to nlg tasks by the community. ## conclusion we developed perplexity selection prompt (perplection) a method that enables real-world zeroshot text classification without the use of any human-annotated data. a pilot study demonstrated that perplexity can be an effective measure of the efficacy of templates. experimental results show that, for datasets in both english and chinese, our method can boost zero-shot performance of clozestyle prompt learning in binary sentiment analysis as well as multi-class classification, without using a development set. further in-depth analysis supports the observation that perplection can "foresee" the efficacy of prompt templates.