Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM (2024)

Abstract: The remarkable performance of large language models (LLMs) in zero-shot language understanding has garnered significant attention.However, employing LLMs for large-scale inference or domain-specific fine-tuning requires immense computational resources due to their substantial model size. To overcome these limitations, we introduce a novel method, namely GenCo, which leverages the strong generative power of LLMs to assist in training a smaller and more adaptable language model. In our method, an LLM plays an important role in the self-training loop of a smaller model in two important ways. Firstly, we utilize an LLM to generate multiple augmented texts for each input instance to enhance its semantic meaning for better understanding. Secondly, we additionally generate high-quality training instances conditioned on predicted labels, ensuring the generated texts are relevant to the labels. In this way, GenCo not only corrects the errors of predicted labels during self-training but also eliminates the need for extensive unlabeled texts. In our experiments, GenCo outperforms previous state-of-the-art methods when only limited ( <5\% of original) in-domain text data is available. Notably, our approach surpasses Alpaca-7B with human instructions, highlighting the significance of self-training.

## introduction zero-shot text classification poses a challenge in predicting class labels for text instances without requiring labeled instances for supervised training. effective solutions to this problem is crucial for many real-world applications, as it diminishes the labor-intensive process of manual labeling. with the remarkable advancements of large language models (llms) @xcite in recent years, exploiting the generative capabilities of such models to tackle zero-shot text classification problems has emerged as a critical research question. recent research in zero-shot text classification primarily falls into two distinct groups. the first approach applies llm (with billions of parameters) in label prediction with the help of human instructions or prompts @xcite . however, even a relatively smaller llm such as alpaca-7b @xcite necessitate considerable computational power and time for large-scale inference and model fine-tuning. without domain-specific fine-tuning, llms struggle to discern between classes characterized by unclear decision boundaries. the second approach to zero-shot classification involves the self-training of smaller language models, often comparable in size to bert @xcite @xcite . in these methods, the models predict "pseudo labels" for unlabeled instances, and then use these instances alongside their assigned pseudo labels as supervised data for model finetuning. this process is iterated for the model to incrementally adapt to the target domain. however, these techniques hinge on accessing a substantial volume of unlabeled texts from the intended domain, sometimes reaching the magnitude of millions as indicated in in this paper, we introduce a novel approach called generation-driven contrastive self-training (genco). this approach adeptly combines the language understanding ability of llms with the adaptability and efficiency of smaller models. drawing inspiration from @xcite , we treat zero-shot classification as a sentence alignment task and employ contrastive self-training with smaller models. we provide a theoretical analysis of how self-training can bolster classification generalization. crucially, we sidestep the dependency on extensive unlabeled texts by capitalizing on the generative strengths of llms. our approach exploits the llm generation power in two ways. firstly, to enhance pseudo label prediction, we employ an llm to generate multiple variations or extensions of an input text. this augmentation strategy enriches the available information for the classifier, enabling it to make better predictions based on a more comprehensive understanding of the input. secondly, we employ the llm to craft new training instances conditioned on the pseudo labels, ensuring the generated content is closely aligned with its assigned pseudo label. this tackles the prevalent issue of mislabeling in self-training. in summary, this paper makes three key contributions: ## preliminary: zero-shot text classification as sentence alignment given a set of n unlabeled documents @xmath0, x 2 , • • • , x n } and a set of l category descriptions @xmath1, c 2 , • • • , c l }, the goal is to learn a scoring function g(x, c i ) that takes document x and label description c i as input and produces a similarity score as the measure of how well the document and the label match to each other. in the zero-shot setting, text classification can be formulated as a sentence alignment problem @xcite , where both the input sentence and the label descriptions are encoded using a pre-trained sentence encoder like simcse @xcite . the similarity scores between the sentence and label embeddings are used to predict related labels. the performance can be further improved by converting a short label description into a full sentence via prompts @xcite . for example, the label "sports" can be converted to "this is an article about sports." subsequently, we represent the label prompt for a label description c i as p i . the scoring function can be implemented as follows: given an input text at inference time, the predicted label is the one with the highest similarity score: ## conclusion in conclusion, our proposed approach, genco, effectively addresses the difficulties and limitations of using llms directly for zero-shot text classification. by leveraging the generative power of an llm in a self-training loop of a smaller, sentence encoder classifier with contrastive learning, genco outperform state-of-the-art methods on four benchmark datasets. our approach is particularly effective when limited in-domain text data are available. the success of our approach highlights the potential benefits of incorporating the generative power of llm into iterative self-training processes for smaller zero-shot classifiers. we hope that our work will inspire further research in this direction, ultimately leading to more efficient and effective nlp models.