Generate, Annotate, and Learn: NLP with Synthetic Text (2022)

Abstract: This paper studies the use of language models as a source of synthetic unlabeled text for NLP. We formulate a general framework called “generate, annotate, and learn (GAL)” to take advantage of synthetic text within knowledge distillation, self-training, and few-shot learning applications. To generate high-quality task-specific text, we either fine-tune LMs on inputs from the task of interest, or prompt large LMs with few examples. We use the best available classifier to annotate synthetic text with soft pseudo labels for knowledge distillation and self-training, and use LMs to obtain hard labels for few-shot learning. We train new supervised models on the combination of labeled and pseudo-labeled data, which results in significant gains across several applications. We investigate key components of GAL and present theoretical and empirical arguments against the use of class-conditional LMs to generate synthetic labeled text instead of unlabeled text. GAL achieves new state-of-the-art knowledge distillation results for 6-layer transformers on the GLUE leaderboard.

## introduction there is an abundance of unlabeled data in the real world, but task-specific unlabeled data within the scope of a given machine learning problem can be challenging to find. for instance, one cannot easily find in-domain unlabeled text conforming to the input distribution of a specific natural language processing (nlp) task from the glue benchmark @xcite . some nlp tasks require an input comprising a pair of sentences with a particular relationship between them. moreover, classification datasets typically represent a tailored distribution of data and only include a limited number of class labels. if task-specific unlabeled data were available, one could adopt self-training @xcite to automatically annotate unlabeled data with pseudo labels to improve accuracy and robustness of classifiers @xcite . in addition, one can use knowledge distillation @xcite on fresh task-specific unlabeled data to more effectively compress deep neural networks and ensembles @xcite . in the absence of task-specific unlabeled data, one could retrieve unlabeled examples from a large and diverse open-domain dataset @xcite . however, such a retrieval-based approach may not scale to problems with complex input schemes, for example, sentence pairs with certain relations. recent work @xcite has considered the use of language models (lms) like gpt-2 @xcite as a means of data augmentation, showing the effectiveness of this approach for commonsense reasoning and classification tasks. existing approaches often consider class-conditional generation, where the synthetic data is produced by con ditioning on a specified class label. however, it is unclear whether class-conditional generation is best suited for nlp tasks. furthermore, existing pipelines often make synthetic data generation complicated as one needs to detect and discard low-quality synthetic labeled data or optionally re-label data @xcite . for instance, @xcite observe that it is difficult for sentences generated by labelconditioned gpt-2 to retain the semantics/pragmatics of the conditioning label, leading to poor performance on downstream tasks. we unify and simplify existing work on lms as a data source for nlp and develop a general framework called ''generate, annotate, and learn (gal )''. the generality of gal allows us to use lm-generated synthetic data within novel applications such as knowledge distillation (kd) and few-shot learning. gal builds on recent advances in text generation @xcite and uses powerful lms to synthesize task-specific unlabeled text by fine-tuning or conditioning a large lm on in-distribution examples. we use state-of-the-art classifiers to annotate generated text with soft pseudo labels when possible. we then combine labeled data and pseudo-labeled data to train more effective supervised models, resulting in significant gains on a range of nlp tasks like kd and few-shot learning. we present a justification for gal based on the empirical and vicinal risk minimization frameworks @xcite . we also investigate key components of gal. we find that even if class-conditional lms are available for text generation, it is more effective to discard the conditioning labels and let the teacher models produce pseudo labels. this observation is supported by our theoretical and empirical results. accordingly, in contrast to prior work @xcite , we advocate for the use of simple unconditional lms for text synthesis. further, we avoid any form of data filtering. not surprisingly, we find that the diversity of synthetic text matters. that said, simple unconditional generation given random seeds provides sufficient diversity, and crafting diverse lm prompts is not needed. in summary: it further improves prompt-based few-shot learning, providing an average improvement of 1.3% on four 4-shot learning nlp tasks, outperforming gpt-3-6b. self-training is one of the oldest approaches for semi-supervised learning @xcite @xcite @xcite @xcite . @xcite and @xcite have theoretically analyzed selftraining for simple decision lists. recent theoretical work analyzes self-training for linear models, often under the assumption that the data distribution is (nearly) gaussian @xcite @xcite @xcite . @xcite prove that, under ''expansion'' and ''class separation'' assumptions, self-training can lead to more accurate neural network classifiers. we present a theoretical framing of gal in terms of empirical and vicinal risk minimization @xcite . knowledge distillation (kd) @xcite uses a procedure similar to self-training to distill knowledge of an expressive teacher model into a smaller student model. in contrast, self-distillation @xcite @xcite uses teacher and student models of equal size, hoping to iteratively refine class labels. previous work uses unlabeled data @xcite and adversarial training @xcite to improve kd. we demonstrate that synthetic data generated by unconditional generative models can improve kd on nlp, outperforming strong kd baselines, which often add more complexity and additional hyperparameters (e.g., @xcite @xcite . ## generate, annotate, and learn (gal) given a labeled dataset @xmath0, y i )} n @xmath1, we first train an unconditional domain-specific generative model g(x) on l @xmath2, and then use it to synthesize unlabeled data. such synthetic unlabeled data is used within self-training and kd even in the absence of in-domain unlabeled data. we restrict our attention to basic kd and selftraining methods, even though gal can be combined with more sophisticated semi-supervised techniques, too. the effectiveness of gal depends on the fidelity and diversity of synthetic examples. if we had access to the oracle generative process, we would be able to obtain the best kd and ssl results, as if we had access to real task-specific unlabeled data. our preliminary experiments suggest that large language models are particularly effective within the gal framework. hence, as shown in figure 1 , to build the best domainspecific language model, we adopt a large language model pretrained on lots of open-domain text, and fine-tune it on a given dataset's inputs, that is, l x , ignoring class labels. both our theory and ablations confirm that ignoring class labels is a good idea (c.f., section 4 and 5). transferring the knowledge of large language models is particularly beneficial when a small input dataset l x of text is available @xcite . to improve computational efficiency of gal, we do not generate unlabeled data on the fly, but generate as many unconditional samples as possible and store them in a synthetic unlabeled we use open-domain data once for self-supervised pretraining (e.g., bert) and once for training a large lm (e.g., gpt-2). bert is fine-tuned on labeled data to yield a classifier for the task of interest. gpt-2 is fine-tuned on the same data without labels to obtain an unconditional task-specific lm, which is used to generate lots of synthetic in-domain unlabeled data for self-training and kd. dataset u . we use soft pseudo labels within selftraining and kd, as we empirically found it is more effective than using hard labels on synthetic data. ## an empirical risk minimization perspective in supervised learning, one seeks to learn a mapping f that, given an input x, predicts a reasonable output y. to define the supervised learning problem formally, one assumes that input-output pairs are drawn from a joint distribution p , namely, (x, y) ∼ p (x, y), and a loss function h(y, f (x)) is used to assess the quality of a mapping f . this loss is used to define a notion of expected risk: in almost all practical applications p (x, y) is unknown. hence, a labeled dataset of examples @xmath3 this objective function is known as empirical risk, and learning f through minimizing r(f ) is known as the empirical risk minimization principle @xcite . to compensate for the finite sample size in (4), one typically combines r(f ) with a regularizer to improve generalization. beyond empirical risk minimization. empirical risk minimization (4) is motivated as a way to approximate p (x, y) through a set of dirac delta functions on labeled examples: p δ (x, y) = i δ(@xmath4, @xmath5. however, this approximation is far from perfect, hence one uses a heldout validation set for early stopping and hyperparameter tuning. the goal is to increase the support of each labeled data point and improve the quality and robustness of the risk function. recent work on mixup regularization @xcite proposes an effective way to construct another vicinity distribution by interpolating between two data points and their labels. despite their simplicity, these smoothing techniques tend to improve matters. ## experiments in this section, we assess the effectiveness of gal on kd, self-training, and few-shot learning. ## limitations this work demonstrates that one can leverage synthetic in-domain data generated by powerful pre-trained generative models. for simplicity, we do not employ any filtering avenue to retain diverse but high-quality data points. however, previous work has shown that advanced filtering approaches can further improve the performance @xcite @xcite . given that the improvements in the selftraining are not sizeable, we believe it is worth imposing filtering methods on the synthetic data to mitigate the side effects caused by the noisy data points. although we examine the effectiveness of gal on various classification tasks, we still focus on the sentence-level tasks. because of the superior performance on sentence-level tasks, there has been a surge of interest shift to document-level tasks, such as document-level machine translation @xcite @xcite , document summarization @xcite , and so forth. as these tasks suffer from data scarcity, one can leverage gal to synthesize more data points. however, previous work has shown that gpt-2 has difficulty generating coherent text requiring long-range dependency @xcite . thus, such a limitation may hinder the application of gal to document-level tasks. in addition, the label space of the studied tasks is not as complex as the structured prediction tasks, such as machine translation, dialog system, question answering, and so on. however, we believe one can smoothly adapt gal to these tasks as well. let us consider machine translation (mt) as a canonical structured prediction task. prior work has shown that one can use (real) monolingual data, in either source or the target language, through data augmentation @xcite or knowledge distillation @xcite to improve the structured prediction tasks. this suggests a promising avenue for future research on using synthetically generate monolingual data to improve mt for specialized domains where even monolingual data is scarce. furthermore, @xcite suggest that one can leverage a retrieval-based approach to obtain monolingual sentences from the generic data stores. this retrieved monolingual data is then employed to improve the translation quality in a domain adaptation setting. this suggests that a gal-based approach to synthetically generate monolingual text is a promising method to improve mt for specialized domains-an interesting direction for future research. ## conclusion we present generate, annotate, and learn (gal): a framework for self-training and knowledge distillation with generated unlabeled data. we motivate gal from an expected risk minimization perspective and demonstrate both theoretically and empirically that the use of unconditional generative models for synthetic data generation is more effective than class-conditional generative models previously used in the literature. gal leverages advances in large pretrained language models to help supervised learning and can have implications for learning from limited labeled data. gal significantly helps improve knowledge distillation and prompt-based few-shot learning. in addition, a concurrent work @xcite has shown that using generated images can enhance the robustness of images classifiers. we will explore this direction on nlp tasks in the future. finally, we hope that gal will stimulate new research on the evaluation and development of large language models.