Beyond prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations (2022)

Abstract: Recent work has demonstrated that pre-trained language models (PLMs) are zero-shot learners. However, most existing zero-shot methods involve heavy human engineering or complicated self-training pipelines, hindering their application to new situations. In this work, we show that zero-shot text classification can be improved simply by clustering texts in the embedding spaces of PLMs. Specifically, we fit the unlabeled texts with a Bayesian Gaussian Mixture Model after initializing cluster positions and shapes using class names. Despite its simplicity, this approach achieves superior or comparable performance on both topic and sentiment classification datasets and outperforms prior works significantly on unbalanced datasets. We further explore the applicability of our clustering approach by evaluating it on 14 datasets with more diverse topics, text lengths, and numbers of classes. Our approach achieves an average of 20% absolute improvement over prompt-based zero-shot learning. Finally, we compare different PLM embedding spaces and find that texts are well-clustered by topics even if the PLM is not explicitly pre-trained to generate meaningful sentence embeddings. This work indicates that PLM embeddings can categorize texts without task-specific fine-tuning, thus providing a new way to analyze and utilize their knowledge and zero-shot learning ability.

## introduction recent developments in large pre-trained language models (plms) @xcite @xcite open up the possibility of classifying texts without massive in-task data annotation. such a zero-shot setting is receiving increasing attention as it is a good way to evaluate the generalizability of knowledge in plms. currently, most existing methods either utilize keywords for self-training @xcite @xcite or reformulate the classification task into a cloze task using prompts @xcite @xcite . keyword-based methods usually train multiple modules sequentially @xcite , while prompting methods depend heavily on human engineering @xcite or external knowledge @xcite . such task-specific training or engineering is inefficient and usually does not generalize well to new applications. in this work, we show that we can better elicit the zero-shot text classification abilities of plms simply by clustering texts in their embedding spaces. we draw inspiration from recent findings (aharoni and goldberg, 2020) that texts in the same domain (e.g., legal or medical texts) tend to be clustered together in the plm embedding spaces. this indicates that plms already have the knowledge to distinguish texts with different meanings. following this idea, we propose simptc: a simple probabilistic text classification framework building upon state-of-the-art sentence embeddings sim-cse @xcite . given an unlabeled dataset and the corresponding class names, simptc models the texts in each class with a gaussian distribution and fits the text embeddings with a bayesian gaussian mixture model (bgmm). to initialize the clusters, we first use the class names to generate class-related anchor sentences. then the initial cluster assignment of a text is determined according to its similarity to the class anchors in the embedding space. despite the simplicity of simptc, it achieves state-of-the-art performance while avoiding many previously mentioned drawbacks of existing methods: 1) without self-training of the plm, simptc achieves superior or comparable performance on both topic and sentiment classification datasets; 2) unlike prompt-based methods, simptc works well without human engineering or access to external knowledge; 3) simptc outperforms previous methods when the dataset is unbalanced. finally, once we obtain the sentence embeddings, we no longer use the plm, and simptc clusters the embeddings in a fixed dimensional space. thus, one can easily apply simptc to new and large datasets. to explore the applications and limitations of simptc, we compare it with prompt-based zeroshot learning @xcite on 14 datasets with more diverse topics, text lengths, and numbers of classes. simptc gives consistently better performance with a 20% absolute improvement in macro-f1 score on average. we find that simptc handles domain-specific rare class names and large class numbers better, while both the prompt-based method and simptc suffer when the class names are abstract concepts, e.g., subjective v.s. objective. finally, we analyze the embedding spaces of different plms using simptc. surprisingly, although roberta large @xcite is not explicitly pre-trained to generate meaningful sentence embeddings, texts of the same topic are clustered with state-of-the-art zero-shot accuracy. a larger plm like t5 @xcite is able to achieve better zero-shot results, even matching the fully supervised performance of bert @xcite on some datasets. on the other hand, sim-cse embeddings separate topics better, and texts of sub-topics can form sub-clusters. on some datasets, we can even observe a linear semantic structure. to conclude, the strong performance of such a simple clustering-based algorithm suggests that the zero-shot learning ability of plms is still underexplored. with simptc, we provide a new starting point to utilize and analyze the implicit knowledge and zero-shot learning ability of plms. ## related work in this section, we review three types of zero-shot text classification approaches. zero-shot text classification aims at classifying texts without any annotated data. this is also referred to as weaklysupervised text classification as it can use various weak supervision signals, such as the names or descriptions of the classes, to make predictions. ## simptc as illustrated in figure 1 , simptc formalizes a zeroshot text classification task into a clustering problem and solves it in three steps: encode, match, and update. we start by modeling each class with a gaussian cluster in the embedding space. next, the encode step and match step provide a coarse initialization of the cluster means and covariances using the class names. finally, starting from the initialization, we fit the unlabeled data with a bgmm. we elaborate on the three steps of simptc below. ## conclusion in this work, we show that a simple clusteringbased approach, simptc, can achieve state-of-theart zero-shot text classification performance on a wide range of tasks. with extensive experiments, we identify the keys to cluster texts in the plm embedding spaces and also the limitations of simptc. further analysis of different plms shows that plms can categorize texts in their embedding spaces without being trained to derive semantically meaningful sentence embeddings, and larger plms tend to have more informative embeddings. we hope our exploration into the embedding spaces of plms can provide insights into understanding and developing new methods to elicit the zero-/fewshot learning ability of plms.