P ro G en: Progressive Zero-shot Dataset Generation via In-context Feedback (2022)

Abstract: Recently, dataset-generation-based zero-shot learning has shown promising results by training a task-specific model with a dataset synthesized from large pre-trained language models (PLMs). The final task-specific model often achieves compatible or even better performance than PLMs under the zero-shot setting, with orders of magnitude fewer parameters.However, synthetic datasets have their drawbacks. They have long being suffering from the low-quality issue (e.g., low informativeness, redundancy). This explains why the massive synthetic data does not lead to better performance – a scenario we would expect in the human-labeled data. To improve the quality in dataset synthesis, we propose a progressive zero-shot dataset generation framework, ProGen, which leverages the feedback from the task-specific model to guide the generation of new training data via in-context examples.Extensive experiments on five text classification datasets demonstrate the effectiveness of the proposed approach. We also show ProGen achieves on-par or superior performance with only 1% synthetic dataset size, when comparing to baseline methods without in-context feedback.

## introduction dataset generation with pre-trained language models (plms) has attracted enormous interest recently due to the superior generative capacity of plms. given task-specific supervision, recent work @xcite @xcite manages to fine-tune the plms to synthesize highquality datasets for downstream applications. nevertheless, obtaining task supervision from human experts can be expensive or even unrealistic. recent attempts @xcite @xcite turn their eyes to the unsupervised dataset generation. ## background in this section, we briefly review the baseline approaches of zero-shot dataset generation and how the synthesized dataset can be used for zero-shot learning on downstream tasks. zero-shot dataset generation take text classification task as an example, vanilla zero-shot dataset generation methods @xcite aims to generate a synthetic dataset @xmath0, y)} with the help of a plm p. they first sample a class label y from a uniform distribution: since the parameters of p is frozen and the generation x for each y is deterministic, different sampling algorithms (e.g., top-k sampling @xcite and nucleus sampling @xcite ) can be adopted to increase the diversity of generated dataset. a synthetic dataset is constructed after pairing the generated x with y. ## progen we now describe our framework for progressive zero-shot dataset generation via in-context feedback (progen), as shown in figure 2 . we follow zerogen @xcite to build the backbone of our framework. concretely, we first train a task-specific model (tam) with partially generated dataset. then, assuming no access to human annotations, we estimate the influence of each sample via the noise-robust influence function. finally, with those identified most influential samples, we explore the use of in-context learning to shift the generation distribution towards that of influential samples, so that the system generates more related samples. the whole framework progressively constructs the synthetic dataset and enhances the performance of the final task-specific model. ## conclusions this work proposes progen for zero-shot dataset generation, which progressively improves the dataset quality by leveraging feedback from a taskspecific model trained on the current dataset. by evaluating zero-shot performance with the trained model, we show progen can generate a much smaller (e.g., 1%) synthetic high-quality dataset that achieves comparable or superior performance to baseline method. we also provide a variety of analyses, including formats of in-context examples, other measurements on synthetic datasets, and the influence of prompt selection.