PESCO : Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classification (2023)

Abstract: We present PESCO, a novel contrastive learning framework that substantially improves the performance of zero-shot text classification. We formulate text classification as a neural text retrieval problem where each document is treated as a query, and the system learns the mapping from each query to the relevant class labels by (1) adding prompts to enhance label retrieval, and (2) using retrieved labels to enrich the training set in a self-training loop of contrastive learning. PESCO achieves state-of-the-art performance on four benchmark text classification datasets. On DBpedia, we achieve 98.5% accuracy without any labeled data, which is close to the fully-supervised result. Extensive experiments and analyses show all the components of PESCO are necessary for improving the performance of zero-shot text classification.

## introduction text classification is the task of assigning relevant category labels to each input document. it is an important problem in machine learning research with a wide spectrum of applications, including sentiment analysis @xcite @xcite , question answering @xcite , and intent classification @xcite , etc. recently, deep neural networks have obtained remarkable improvements in text classification, including cnns @xcite , rnns @xcite , transformers @xcite , and more, thanks to the successful modeling of contextualized representations. despite the remarkable progress, training wellperforming neural classifiers still requires a large amount of human-labeled documents, which is costly and time-consuming, especially for new application domains. this stimulates the recent trend of exploring self-supervised pre-training neural models on text classification tasks. in particu-lar, pre-trained language models (ptlms) @xcite @xcite clearly stand out from other methods owing to the pre-training on large-scale unlabeled data. nevertheless, how to adapt ptlms to downstream tasks with less supervision remains an open question for the research community, inviting new ideas to explore. prompt-based learning @xcite @xcite @xcite has been actively studied to better adapt ptlms to downstream tasks with the goal of reducing human annotation effort. for example, pet @xcite ) is a prompt-based method for few-shot text classification. it formulates the task as a cloze test, where a ptlm is used to predict the output label(s) by completing a prompt concatenated right after an input document. for example, the sentiment of a product review is highly likely to be positive if a ptlm fills the word "good" into the following input: this example shows that prompt-based learning could unleash the potential power of a ptlm by constructing the input format of a downstream task in a way that closely resembles the ptlm pretraining objective, which is masked language modeling (mlm) in this case. motivated by the recent success of prompt-based learning, we propose pesco, a novel self-training framework for zero-shot classification that uses prompts to enhance performance. the self-training consists of two iterative steps, pseudo-label prediction and model update. to make label descriptions more informative, we first put label descriptions into some predefined prompts and call the enhanced descriptions label-prompts. as depicted in figure 1 , to predict the pseudo-label of a document, pesco formulates text classification as a neural matching task. a pre-trained text encoder maps both docu-ments and label-prompts into a shared embedding space. a label whose embedding is closest to the document is predicted as the pseudo-label. to effectively update the text encoder with pseudo-labels, we propose the prompt-enhanced label-aware cloze test (plct), a contrastive learning framework for self-training. the text encoder is trained to match a document and the text relevant to its pseudo-label. the relevant texts include pseudo-label prompts and the key sentences from the documents assigned to the same pseudolabel. the key sentence of each document is the sentence most related to its pseudo-label. in our experiments, we show that the iterative self-training consistently improves the classification performance compared to the same model without self-training and that our proposed approach substantially outperforms other strong zero-shot classification baselines. on some datasets, the zeroshot results are even on par with a fully supervised baseline. on the dbpedia dataset, in particular, pesco achieves 98.5% accuracy without any labeled data. in summary, the contributions of this paper are twofold: ## zero-shot classification as matching in our zero-shot setting, there are n unlabeled documents @xmath0, x 2 , • • • , x n } and a set of label descriptions @xmath1, c 2 , • • • , c l }, where l denotes the number of classes. we aim to learn a scoring function g(x, c) so that relevant document and label description pairs can have higher scores. a label whose label description has the highest score is selected as model prediction: inspired by the recent success of pre-trained sentence encoder @xcite which has shown impressive performance on matching relevant texts, we explore using pretrained encoders as g(x, c j ). specifically, as illustrated in figure 1 , we formulate zero-shot text classification as a neural text matching problem. both document and label descriptions are encoded into dense vectors by a shared encoder. the matching score can be obtained by measuring cosine similarity between dense vectors. however, label descriptions are usually a few words rather than a sentence with full semantics, which makes ptlms unable to fully understand the meaning of the labels. to tackle this, query reformulation @xcite ) is a commonly used technique in retrieval to enhance the semantics of a query. this technique can be further incorporated with promptbased learning @xcite , which has shown that adding prompts to a text helps ptlms understand classification tasks. we use a prompt function p(•) to convert a label description c into a prompt by placing label descriptions into pre-defined templates. we design t templates for each dataset, and the scoring function is: ## pesco pesco is a simple but effective self-training framework for zero-shot text classification. algorithm 1 gives an overview of pesco. in our iterative selftraining loop, we first use a pre-trained sentence encoder f θ to generate pseudo-labels (i.e. predicted labels) by the matching process described in section 3. we then use the pseudo-labels to update f θ by prompt-enhanced label-aware cloze test (plct), which leverages pseudo-labels to construct positive training pairs. we continue the selftraining process by iteratively generating pseudolabels and updating the model using the plct objective function. ## conclusion this paper presents a novel approach to zero-shot text classification, which significantly improves the sota results on four benchmark datasets by formulating the classification task as a prompt-enhanced retrieval problem and by combining the strengths of pre-trained language models and contrastive learning over pseudo-labeled data in a self-training loop. our experiments in comparison with representative baselines and ablation analysis show evidence for the effectiveness of the proposed approach.