R e G en: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval (2023)

Abstract: With the development of large language models (LLMs), zero-shot learning has attracted much attention for various NLP tasks. Different from prior works that generate training data with billion-scale natural language generation (NLG) models, we propose a retrieval-enhanced framework to create training data from a general-domain unlabeled corpus. To realize this, we first conduct contrastive pretraining to learn an unsupervised dense retriever for extracting the most relevant documents using class-descriptive verbalizers. We then further pro- pose two simple strategies, namely Verbalizer Augmentation with Demonstrations and Self- consistency Guided Filtering to improve the topic coverage of the dataset while removing noisy examples. Experiments on nine datasets demonstrate that ReGen achieves 4.3% gain over the strongest baselines and saves around 70% of the time when compared with baselines using large NLG models. Besides, REGEN can be naturally integrated with recently proposed large language models to boost performance.

## introduction text classification serves a fundamental task in natural language processing (nlp) with a broad spectrum of applications. recently, large pretrained language models (plms) @xcite have achieved strong performance on text classification with a large amount of task-specific training data. however, in real world scenarios, collecting labeled data can be challenging due to the cost of time, money, and domain expertise. to reduce the burden of human annotation, we study automatic dataset generation for text classification under the zero-shot setting, where no task-specific or cross-task data is available. such a setting is different from previous works that use a large collection of labels from auxiliary tasks for zero-shot text classification @xcite @xcite , and is particularly challenging since we need to adapt the language understanding abilities of plms to target classification tasks with minimal supervision. prior works on zero-shot synthetic dataset generation mainly fall into two categories: (1) generative methods leverage a billion-scale nlg model to generate class-conditioned texts for plm finetuning @xcite . while these methods work well on easy tasks (e.g. binary classification), they can be fragile on challenging tasks with more classes, as the generated text can be less discriminative. besides, the gigantic size of the nlg model will also cause the inefficiency issue. (2) mining-based methods design rule-based regular expressions to extract text from the background corpus as synthesized training data @xcite , but these rules are often too simple to capture the complex semantics of text. as a result, the mined dataset contains many incorrectly-labeled data, and the fine-tuned plm can easily overfit noisy labels. we design a new framework regen foot_1 to solve zero-shot text classification. the setting of regen is close to the mining-based technique (van de @xcite , where a set of class-specific verbalizers and a collection of general-domain unlabeled corpus are available. motivated by the limitation of hard matching with regular expressions which hardly preserves the meaning of verbalizers, we propose to leverage dense retrieval (dr) @xcite @xcite @xcite , which calculates semantic relevance in a continuous representation space, for dataset curation. with such a soft matching mechanism, dr is able to better encode the category-specific semantics and thus fetch the relevant documents from the corpus. to integrate dr with the target classification task, we employ two plms: one retrieval model (r θ ) to extract the most relevant documents from the unlabeled corpus for synthetic dataset curation, and one classification model (c ϕ ) to be fine-tuned on the generated synthetic dataset to perform the downstream task. before performing text retrieval, we first conduct contrastive learning on the unlabeled corpus to further pretrain the retrieval model r θ for producing better sequence embeddings. then, with the retrieval model, we use the verbalizers from each class as queries to retrieve relevant documents from the unlabeled corpus, which will be used as the training data for target tasks. simply fine-tuning the classifier on the above training data may yield limited performance, as the verbalizers are often too generic to cover all the category-related topics (e.g., the word 'sports' alone does not cover concrete types of sports). thus, the retrieved data may contain noisy and irrelevant documents. to enhance the quality of the synthetic dataset, we conduct multi-step retrieval with two additional strategies to strengthen our framework: (1) we augment the verbalizer with the retrieved documents from the previous round as additional information @xcite to enrich its representation, which allows for extracting more relevant documents for the downstream task. (2) we exploit self-consistency to filter the potentially incorrect examples when the pseudo labels produced by the retrieval model (r θ ) and the classifier (c ϕ ) disagree with each other. we note that regen does not use annotated labels from any other tasks, making it applicable to the true zeroshot learning. besides, regen only requires two bert base scale plms, which is efficient compared with methods using large nlg models. our contribution can be summarized as follows: ## related work zero-shot text classification (zstc) aims to categorize the text document without using taskspecific labeled data. with pretrained language models, a plenty of works attempted to convert the classification task into other formats such as masked language modeling @xcite , question answering @xcite @xcite or entailment @xcite for zero-shot learning. these works are orthogonal to regen as we do not directly perform inference and do not leverage human annotations from additional tasks. more relevant to us, there are some recent studies that perform zstc via generating a task-specific dataset using nlg models, which is then used to fine-tune a classifier for the target task such as text classification @xcite , sentence similarity calculation (schick and schütze, 2021b), commonsense reasoning @xcite , and instruction-based tuning @xcite . unfortunately, the generation step is time-consuming and the quality of the generated text can be less satisfactory in capturing fine-grained semantics. the most relevant work to us is (van de @xcite , which also extracts documents from the unlabeled corpus to form the training set. but it simply uses regular expressions to mine documents and cannot fully capture the contextual information of verbalizers. instead, we leverage dense retrieval for concept understanding and obtain the most relevant documents, which is combined with verbalizer augmentation to improve retrieval quality. on the other hand, retrieval-augmented language models have been used in language modeling @xcite , openqa @xcite , information extraction @xcite and knowledge-intensive tasks @xcite , where tokens or documents are retrieved based on contextual representations and are used as additional inputs to support target tasks. while such a paradigm has also been explored for zero-shot learning, it is mainly used for zero-shot prompt-based inference @xcite . instead, we empirically demonstrate the efficacy of retrieval-enhanced learning for zero-shot dataset curation with an unsupervised dense retrieval model. ## method in this section, we present regen (our framework) and introduce the major components.