Z ero G en: Efficient Zero-shot Learning via Dataset Generation (2022)

Abstract: There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs). In this paper, we study a flexible and efficient zero-short learning method, ZeroGen.Given a zero-shot task, we first generate a dataset from scratch using PLMs in an unsupervised manner. Then, we train a tiny task model (e.g., LSTM) under the supervision of the synthesized dataset. This approach allows highly efficient inference as the final task model only has orders of magnitude fewer parameters comparing to PLMs (e.g., GPT2-XL).Apart from being annotation-free and efficient, we argue that ZeroGen can also provide useful insights from the perspective of data-free model-agnostic knowledge distillation, and unreferenced text generation evaluation. Experiments and analysis on different NLP tasks, namely, text classification, question answering, and natural language inference, show the effectiveness of ZeroGen.

## introduction while generating training data with language model is not new to natural language processing @xcite @xcite , it has garnered enormous interests recently due to the superior generative capacity of large-scale pre-trained language models (plms). training examples created in such a manner have been found effective in various scenarios via the data augmentation procedure @xcite @xcite . in this paper, we study an extreme scenario of such an approach, zerogen. given a downstream task, we first generate its training data from scratch using a powerful plm, whose generation is steered by carefully designed task-specific prompts. ## preliminary: prompt-based zero-shot learning we start with preliminary knowledge about prompt-based zero-shot learning framework (named prompting). giving a pre-trained language model (plm) p and a text classification (tc) task @xmath0, y), prompting first instantiates a prompt t (•) with each input x @xmath1. for instance, we show an example on sentiment analysis task in figure 1 (a), where x i is "a deep and meaningful film." and t (x i ) is "a deep and meaningful film. the sentiment of the movie review is ". furthermore, prompting defines a verbalizer m(•) that maps each label/class y i to a word/words in p's vocabulary. for instance, "positive" and "negative" represents the two classes. in this way, prompting models the probability of class y @xmath2: during the whole process, the pre-trained weights of p are frozen and no training is required. the vast linguistic @xcite @xcite and factual @xcite knowledge encoded in plms' parameters is the key towards prompting's success. however, prompting fails to fully exert the capacity of plms and heavily relies on gigantic plms during inference. this motivates us to explore a more flexible and efficient way of conducting zero-shot learning with plms. ## zerogen in this work, we take the dataset generation method to the extreme and study zerogen, a flexible and efficient zero-shot learning framework via dataset generation. zerogen framework comprises three sequential stages as shown in figure 1(b): 1. the goal of the first stage is to make use of the generative power of plms to synthesize a dataset to solve the downstream task. with carefully designed prompts and a powerful plm, the generated dataset is believed to incorporate rich task-specific knowledge. pseudo dataset generation for a singlesentence classification task d, we aim to generate a pseudo dataset d @xmath3, y g ) with the help of a left-to-right plm p. we first sample a class label y g from a uniform distribution: since the parameters of p is frozen and the generation x g for each y g is deterministic, we can adopt different sampling algorithms (e.g., top-k sampling @xcite and nucleus sampling @xcite ) to increase the diversity of generated dataset. we then pair the generated x g with y g to construct a pseudo training data. we show an example about generating a pseudo sentiment classification dataset in figure 1(b) . the prompt t (y g ) for a positive label y g is "the movie review in positive sentiment is: "". with the sampling strategies, this prompt steers plms to generate multiple sentence ending with another quotation mark, e.g., "a deep and meaningful movie."" or "good film!!!"". for sentence-pair classification tasks, we need to generate two sequences that bear certain relationships (e.g., premise and hypothesis in nli, context and question in qa). we decompose the generation into two steps: (i) we first generate and/or sample a conditional context c g (e.g., c g represents premise in nli and context in qa). the context c g is then concatenated with a sampled label y g and transformed into a prompt t (c g , y g ). pseudo-supervised training with the pseudo dataset d g , we train a tiny task model tam to conduct the given task. this procedure is highly flexible, meaning that we can use any model architecture, loss function, and training strategy. in this work, we primarily focus on the overall framework, thus we leave the tuning of these components for future work. under the zero-shot learning setting, it should be noted that we have no access to the standard validation set. therefore, we use a portion (e.g., 10%) of the pseudo dataset as the validation set for model selection. zero-shot evaluation finally, we conduct inference on the trained tam model. as tam is order-of-magnitude smaller than plm, it is able to perform extremely efficient inference. ## conclusion and future directions in this paper, we study an extreme instance of dataset generation via plms for zero-shot learning. without any human annotations, we show that an small lstm can surpass the zeroshot performance of its plm counterparts (e.g., gpt2-xl), and even outperform the same model trained with human annotations. despite the demonstrated effectiveness, we discuss several issues we observed when developing zerogen and reveal a substantial room of improvement in future research. despite positive results on tc tasks, we find the stability regarding prompt choice of zerogen is still far from satisfactory on nli tasks. future work could include multi-task prompt-based pre-training methods @xcite . furthermore, we observe noisy examples in synthetic dataset on difficult tasks such as nli and qa, this situation progressively deteriorates when incorporating more diverse decoding strategy (e.g., nucleus sampling). better decoding strategies are needed to ensure the label correctness while preserving the dataset diversity @xcite . besides, methods that learn from noisy labels can be integrated into the training of the tiny task model @xcite . we hope this paper can provide contributions for further exploiting dataset-generation-based zero-shot learning with large pre-trained language models.