G ENERATE , A NNOTATE , AND L EARN : G ENERATIVE M ODELS A DVANCE S ELF -T RAINING AND K NOWLEDGE D ISTILLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Semi-Supervised Learning (SSL) has seen success in many application domains, but this success often relies on the availability of task-speciﬁc unlabeled data.
Knowledge distillation (KD) has enabled compressing deep networks, achieving the best results when distilling knowledge on fresh task-speciﬁc unlabeled data.
However, task-speciﬁc unlabeled data can be challenging to ﬁnd, especially for NLP.
We present a simple framework called “generate, annotate, and learn (GAL)” that uses unconditional language models to synthesize in-domain unlabeled data, helping advance SSL and KD on NLP and tabular tasks.
To obtain strong taskspeciﬁc generative models, we either ﬁne-tune a large language model (LLM) on inputs from speciﬁc tasks, or prompt a LLM with a few input examples to generate more unlabeled examples.
Then, we use existing classiﬁers to annotate generated unlabeled examples with pseudo labels, which are used as additional training data or as additional prompts.
GAL improves prompt-based few-shot learning on several NLP tasks.
It also yields a new state-of-the-art for 6-layer transformers on the GLUE leaderboard.
Finally, self-training with GAL offers large gains on four tabular tasks from the UCI repository.
I NTRODUCTION
Unlabeled data is abundant in the real world, but task-speciﬁc unlabeled data within the scope of a given machine learning problem can be challenging to ﬁnd.
For instance, one cannot easily ﬁnd in-domain unlabeled data conforming to the input distribution of a speciﬁc Natural Language Processing (NLP) task from the GLUE benchmark (Wang et al., 2019b).
Some NLP tasks require an input comprising a pair of sentences with a particular relationship between them.
Moreover, classiﬁcation datasets typically represent a tailored distribution of text and only include a limited number of class labels.
If task-speciﬁc unlabeled data were available, one could adopt self-training (Yarowsky, 1995) to automatically annotate unlabeled data with pseudo labels to improve accuracy and robustness of classiﬁers (Xie et al., 2020; Carmon et al., 2019b).
In addition, one can use knowledge distillation (Hinton et al., 2015) on fresh task-speciﬁc unlabeled data to more effectively compress deep neural networks and ensembles (Buciluˇa et al., 2006; Chen et al., 2020c).
When task-speciﬁc unlabeled examples do not exist, one can try to retrieve them from a large and diverse open-domain dataset.
For instance, Du et al.
(2020) have used nearest neighbor retrieval to harvest in-domain unlabeled text from the internet, leading to a successful application of selftraining and knowledge distillation to certain NLP tasks.
While retrieval can indeed help to ﬁnd in-domain data for problems with simple inputs, it is not practical for problems with complex input schemes, e.g., sentence pairs with certain relations and tabular data.
Accordingly, self-training and retrieval-based methods have not been widely adopted for NLP tasks, e.g., on the GLUE benchmark.
This paper presents a deceptively simple and general framework called “generate, annotate, and learn (GAL)” to help advance semi-supervised learning and knowledge distillation on various applications that do not come with unlabeled data.
We advocate for the use of language models to synthesize unlabeled tasks-speciﬁc data, in lieu of real unlabeled data.
We build on recent advances in text generation (Radford et al., 2019; Gao et al., 2021), and use powerful generative models to synthesize unlabeled text and tables.
Then, we use state-of-the-art classiﬁers to annotate generated unlabeled data with pseudo labels.
Finally, we combine labeled data with pseudo labeled data to train more effective classiﬁers or for the purpose of knowledge distillation (KD)
We motivate GAL by making connections to empirical and vicinal risk minimization (Vapnik, 1992;
Chapelle et al., 2001), and demonstrate its utility by presenting empirical results on a wide range of applications.
Our key contributions include:
• We propose a simple way to advance SSL, KD, and few-shot learning on NLP by using language models to synthesize large amounts of task-speciﬁc unlabeled data.
• We link GAL to empirical and vicinal risk minimization, helping explain why GAL works and why synthetic samples from class-conditional language models are not as effective.
• We systematically dissect GAL and study the key components leading to its success.
• GAL establishes a new SoTA for a single 6-layer transformer on the GLUE test set.
• GAL improves prompt-based few-shot learning, providing an average improvement of 1.3% on four 4-shot learning NLP tasks.
• GAL advance self-training for tabular tasks, outperforming XGBoost on 2 out of 4 tasks.
R ELATED W ORK
There has been a surge of interest in improving accuracy and label efﬁciency of classiﬁers via:
Self-Supervised pretraining on open-domain unlabeled data in a task-agnostic way (Peters et al., 2018;
Devlin et al., 2019;
Chen et al., 2020b), 2.
Self-Training using domain-speciﬁc unlabeled data in a task-speciﬁc way (Rosenberg et al., 2005; McClosky et al., 2006; Xie et al., 2020).
While self-supervised learning can be applied to a broad distribution of unlabeled data, self-training requires unlabeled data that at least can be annotated using the same set of class labels available for the downstream task (Oliver et al., 2018).
For instance, if one is interested in training a classiﬁer to distinguish images of cats and dogs, self-training with images of aircraft is likely not helpful, but it is conceivable that self-supervised learning with images of aircraft can still help.
A growing body of recent work suggests that perhaps self-supervised pretraining and self-training are compatible and can be combined to achieve the best semi-supervised learning performance (Chen et al., 2020c; Du et al., 2020).
We corroborate the existing evidence by showing gains from generative self-training.
Semi-supervised learning (SSL) has a long and rich history in machine learning (Cooper & Freeman, 1970;
McLachlan & Ganesalingam, 1982; Riloff, 1996; Chapelle et al., 2009; Van Engelen &
Hoos, 2020).
One of the oldest family of SSL algorithms is self-training , a.k.a.
self-learning or selflabeling (Scudder, 1965; Fralick, 1967; Agrawala, 1970; Yarowsky, 1995).
Self-training encourages knowledge transfer between a teacher and a student model in such a way that the student can outperform the teacher.
Speciﬁcally, one leverages the teacher’s knowledge to annotate unlabeled data with so-called pseudo labels , and the student learns from a mixture of pseudo- and human-labeled data.
Self-training has recently seen renewed interest across vision and NLP applications (Yalniz et al., 2019; Xie et al., 2020; Zoph et al., 2020; Du et al., 2020).
Recent work aims to combine self-training and consistency regularization to develop powerful SSL algorithms.
The key idea is to ensure that the predictions of a classiﬁer on unlabeled examples are robust to strong augmentations (Berthelot et al., 2019a; Sohn et al., 2020; Xie et al., 2019).
We build on prior work and investigate the use of synthetic data within the broad family of self-training methods.
Recent theoretical work analyzes self-training for linear models, often under the assumption that the data distribution is (nearly) Gaussian (Carmon et al., 2019a; Raghunathan et al., 2020; Chen et al., 2020d; Kumar et al., 2020a; Oymak & Gulcu, 2020).
Wei et al.
(2021) prove that, under “expansion” and “class separation” assumptions, self-training can lead to more accurate neural network classiﬁers.
We present a theoretical framing of GAL in terms of empirical and vicinal risk minimization (Vapnik, 1992; Chapelle et al., 2001).
An important family of related work uses generative models for SSL by learning features that are useful for both generation and discrimination ( e.g., Chen et al., 2020a;
Odena, 2016; Dai et al., 2017).
For instance, Kingma et al.
(2014) approach SSL by viewing missing class labels as a set of latent variables and use variational inference to impute missing labels as well as other factors of variation.
By contrast, our work does not learn features using generative models and keeps the generative and discriminative processes separate.
This offers more ﬂexibility and allows GAL to use self-supervised pretraining methods that are not fully generative.
Our work is closely related to recent work on the uses of generative models for data augmentation (Norouzi et al., 2020; Yang et al., 2020).
Unlike Norouzi et al.
(2020), we do not use instance- based generative models.
Yang et al.
(2020) propose a complex scheme, including data relabeling, data ﬁltering, and two-stage training, to utilize synthetic data.
By contrast, we show that a simple mixture of the original data and synthetic data can provide sizable gains.
Furthermore, we show a more broad use of generative models on KD and few-shot learning, in addition to tabular tasks.
Knowledge Distillation (KD) (Buciluˇa et al., 2006; Hinton et al., 2015) uses a procedure similar to self-training to distill knowledge of an expressive teacher model into a smaller student model.
In contrast, self-distillation (Furlanello et al., 2018; Zhang et al., 2019a; Mobahi et al., 2020) uses teacher and student models of equal size, hoping to iteratively reﬁne class labels.
Previous work uses unlabeled data (Buciluˇa et al., 2006) and adversarial training (Wang et al., 2018) to improve
KD.
We demonstrate that synthetic data generated by unconditional generative models can improve
KD on NLP, outperforming strong KD baselines, which often add more complexity and additional hyper-parameters ( e.g., Sun et al., 2019a; Jiao et al., 2019; Xu et al., 2020; Rashid et al., 2021).
Advanced generative models are able to generate realistic images and text (Karras et al., 2017;
Brock et al., 2019; Karras et al., 2019; Radford et al., 2019; Brown et al., 2020).
The quality of synthetic samples has improved to the extent that deep fake detection has become an important research topic itself (Zellers et al., 2019; Dolhansky et al., 2019).
Recent work has aimed to utilize classconditional generative models to help improve supervised learning (Antoniou et al., 2017; Bowles et al., 2018;
Zhang et al., 2019b; Kumar et al., 2020b; Gao et al., 2020).
However, Ravuri & Vinyals (2019) have shown that images generated by state-of-the-art class-conditional generative models fall short of improving ImageNet classiﬁcation accuracy, despite strong sample quality scores (Salimans et al., 2016; Heusel et al., 2017).
Similarly, Kumar et al.
(2020b)
ﬁnd that it is difﬁcult for sentences generated by label-conditioned GPT-2 (Radford et al., 2019) to retain the semantics or pragmatics of a speciﬁed category, which leads to poor performance on downstream tasks.
We discuss why class-conditional generative models are hardly effective for supervised learning, and instead, focus on unconditional generative models.
B ACKGROUND ON S ELF -T RAINING
Given a labeled dataset L = { ( x i , y i ) } Ni =1 and an unlabeled dataset U
= { x j } Mj =1 , we summarize the general family of SSL algorithms known as self-training as:
First, an initial model denoted f 1 is trained using supervised learning on the labeled dataset L . 2.
Then, at iteration t , one adopts f t as the teacher model to annotate the unlabeled dataset U using pseudo labels .
Optionally, one uses a selection method to pick a subset S t ⊆ { ( x j , f t ( x j )) } Mj
= of pseudo labeled examples.
A student model f t +1 is trained to optimize a classiﬁcation loss on the combination of L and S t :
[Equation]
where H ( q, p ) =
q (cid:62) log p is the softmax cross entropy loss, and y is assumed to be a one-hot vector (original labels) or a vector of class probabilities ( soft pseudo labels).
Self-training iterations are repeated T times or until performance plateaus.
Many different variants of the basic self-training algorithm discussed above exist in the literature.
These variants differ in the type of pseudo labels used, the selection strategy to ﬁlter pseudo labeled examples, the speed at which f t is replaced with f t +1 , the choice of data augmentation strategy in the teacher and student models, and the weighting of the two datasets in the objective (Berthelot et al.
, 2019b;a; Xie et al., 2020; Sohn et al., 2020; Du et al., 2020).
An important design choice is the type of pseudo labels used.
One can simply use soft class probabilities predicted by a teacher f t (Du et al., 2020), sharpened class probabilities (Berthelot et al., 2019b), or hard labels (a one-hot vector that is zero except at argmax f t ( x ) )
(Lee et al., 2013).
Another important consideration is the selection strategy to retain a subset of pseudo-labeled examples.
FixMatch (Sohn et al., 2020) uses a hyper-parameter τ to select examples on which the teacher model has a certain level of conﬁdence, i.e.,
[Equation]
NoisyStudent (Xie et al., 2020) also uses a form of conﬁdence ﬁltering but ensures that the class labels in the selected subset are balanced.
In principle, any method for out-of-distribution detection (Hendrycks & Gimpel, 2016) can be adopted for ﬁltering pseudo-labeled examples.
We adopt the simplest variant of self-training and limit hyper-parameter tuning to a bare minimum.
[Figure]
G ENERATE , A NNOTATE , AND L EARN (GAL)
Given a labeled dataset L = { ( x i , y i ) } Ni =1 , we ﬁrst train an unconditional domain-speciﬁc generative model g ( x ) on L x = { x i } Ni =1 , and then use it to synthesize unlabeled data.
Such synthetic unlabeled data is used within self-training and KD even in the absence of in-domain unlabeled data.
We restrict our attention to basic KD and self-training methods, even though GAL can be combined with more sophisticated semi-supervised techniques too.
The objective function of GAL for self-training during iteration t , provided a teacher model f t , is expressed as:
[Equation]
We use soft pseudo labels within self-training and KD.
To improve computational efﬁciency of GAL, we do not generate unlabeled data on the ﬂy, but generate as many unconditional samples as possible and store them in a synthetic unlabeled dataset U .
We simply set λ = 0 .
5 , unless stated otherwise.
Not surprisingly, the effectiveness of GAL depends on the ﬁdelity and diversity of synthetic examples.
If we had access to the oracle generative process, we were able to obtain the best KD and SSL results, as if we had access to real task-speciﬁc unlabeled data.
Our preliminary experiments suggest that large language models are particularly effective within the GAL framework.
Hence, as shown in
Figure 1, to build the best domain-speciﬁc language model, we adopt a large large language model pretrained on lots of open-domain text, and ﬁne-tune it on a given dataset’s inputs, i.e., L x , ignoring class labels .
Both our theory and ablations conﬁrm that ignoring class labels is a good idea.
Transferring the knowledge of large language models is particularly beneﬁcial a small input dataset L x of text is available (Hernandez et al., 2021).
In what follows, we ﬁrst discuss practical considerations around building domain-speciﬁc language models, and then link GAL to empirical and vicinal risk minimization, to motivate the approach.
4. D OMAIN -S PECIFIC G ENERATIVE M ODELS OF T EXT AND T ABLES
Text.
Many NLP tasks have a relatively small labeled dataset (Wang et al., 2019b;a).
While selfsupervised pretraining, followed by supervised ﬁne-tuning (Devlin et al., 2019; Liu et al., 2019b;
Clark et al., 2020; Lewis et al., 2019) has become the prominent approach to NLP, previous work has also investigated different data augmentation methods to increase the size of the training datasets.
In summary, existing approaches to data augmentation for NLP include lexicon replacement, sentence retrieval, and round-trip machine translation (Wang & Yang, 2015; Yu et al., 2018; Kobayashi, 2018;
Wu et al., 2019; Lichtarge et al., 2019; Wei & Zou, 2019; Alberti et al., 2019; Du et al., 2020;
Shen et al., 2020).
By contrast, we propose the use of unconditional autoregressive LMs for data augmentation.
This is simple, ﬂexible, and powerful.
We take a pretrained GPT-2 language model (Radford et al., 2019) and ﬁne-tune it separately on each dataset of interest after removing class labels.
We ﬁnd that training from scratch on these datasets is hopeless, but the larger the pretrained GPT-2 variant, the better the validation perplexity scores are.
For tasks modeling a relationship between multiple sentences, we concatenate a separator token “[SEP]” between consecutive sentences.
Once a ﬁne-tuned GPT-2 model is obtained, we generate task-speciﬁc synthetic data up to 40 × larger than the original training sets.
For some samples of generated text for GLUE see Table C.1 to C.6.
We believe using bigger LMs and larger synthetic datasets will improve our results, but we are constrained by compute resources.
Tabular data.
Features from tabular tasks are often in a well-structured format, i.e., each data point comprises a ﬁxed number of attributes as in Table D.2.
This property impedes the acquisition of task-speciﬁc unlabeled data, i.e., most augmentation techniques such as round-trip translation and retrieval are hardly applicable.
To enable generative modeling on tabular data, we convert each data point ( i.e., row from the table) into a sentence by concatenating all of its attributes.
This reformatting enables the use of GPT-2 ﬁne-tuning similar to text, and surprisingly GPT-2 samples are very effective.
4. A N E MPIRICAL R ISK M INIMIZATION P ERSPECTIVE
In supervised learning, one seeks to learn a mapping f that given an input x , predicts a reasonable output y .
To deﬁne the supervised learning problem formally, one assumes that input-output pairs are drawn from a joint distribution P , i.e., ( x , y ) ∼ P ( x , y ) , and a loss function H ( y, f ( x )) is used to assess the quality of a mapping f .
This loss is used to deﬁne a notion of expected risk :
[Equation] [Equation]
In almost all practical applications P ( x , y ) is unknown.
Hence, a labeled dataset of examples L
= { ( x i , y i ) }
Ni =1 is used to approximate R ( f ) as
[Equation]
This objective function is known as empirical risk , and learning f through minimizing (cid:98) R ( f ) is known as the empirical risk minimization principle (Vapnik, 1992).
To compensate for the ﬁnite sample size in (5), one typically combines (cid:98) R ( f ) with a regularizer to improve generalization.
Beyond empirical risk minimization.
Empirical risk minimization (5) is motivated as a way to approximate P ( x , y ) through a set of Dirac delta functions on labeled examples: P δ ( x , y )
= (cid:80) i
δ
( x = x i , y = y i ) /N .
However, this approximation is far from perfect, hence one uses a heldout validation set for early stopping and hyper parameter tuning.
Vicinal risk minimization (Chapelle et al., 2001) approximates expected risk as E P ν ( x ,y ) H ( y, f ( x )) , using a vicinity distribution , e.g., ν (˜ x , ˜ y
| x , y ) =
N (˜ x − x , σ 2 ) δ (˜ y =
y ) to approximate P ( x , y ) as
[Equation]
The goal is to increase the support of each labeled data point and improve the quality and robustness of the risk function.
Recent work on mixup regularization (Zhang et al., 2018) proposes an effective way to construct another vicinity distribution by interpolating between two data points and their labels.
Albeit its simplicity, these smoothing techniques tend to improve matters.
Generative models for risk minimization.
One can factorize the joint distribution of input-output pairs as P ( x , y ) =
P ( x ) P ( y | x ) .
Accordingly, if one is able to learn a reasonable unconditional generative model of x denoted g ( x ) , then one can draw a pair ( x , y ) by ﬁrst drawing x ∼ g ( x ) and then using the current instance of f t to draw y ∼ f t ( x ) .
Then, one can use f t and g to approximate expected risk as
[Equation]
The quality of this approximation highly depends on the quality of f t and g .
If f t is far from an optimal classiﬁer f ∗ or g ( x ) is far from P ( x ) , (7) yields a poor approximation.
The expected risk in (7) smoothens the risk landscape in complex ways beyond simple Gaussian smoothing and interpolation.
This smoothing is applicable to any continuous, discrete, or structured domain as long as expressive generative models of P ( x ) are available.
That said, for almost all reasonable loss functions H ( e.g., softmax cross entropy and squared error), (7) is minimized when f t +1 = f t , which is not ideal, especially when f t is far from f ∗ .
On the other hand, empirical risk (5) anchors the problem in real labeled examples that are provided as ground truth.
GAL aims to combine the beneﬁts of (5) and (7) via:
[Equation]
In this formulation, if f t represents the minimizer of empirical risk (5), then f t +1 =
f t is the minimizer of (8) too.
However, one does not seek the global minimizer of empirical risk, but rather the best performance on heldout data.
If f t is obtained by stochastic gradient descent on any risk function, but early stopped according to empirical risk on a heldout set, then using such f t in (8) to deﬁne R t ( f t +1 ) promotes the selection of a mapping f t +1 that minimizes empirical risk while staying close to the best performing mapping so far ( i.e., f t ).
This formulation motivates self-training and GAL as regularizers in the functional space and explains why they can conceivably work.
How about class-conditional generative models?
One can also factorize the joint distribution
P ( x , y ) as P ( y ) P ( x | y ) and accordingly utilize a class-conditional generative model g ( x | y ) to derive the following expected risk formulation:
[Equation]
In this setting pseudo labeling is not needed as synthetic data is already labeled.
One can show that the optimal classiﬁer f ∗ g that minimizes (9) for cross entropy loss is given by,
[Equation]
that is turning the class-conditional generative model into a classiﬁer by using the Bayes rule yields the optimal solution.
Provided that the accuracy of generative classiﬁers on natural image and text classiﬁcation is far behind their discriminate counterparts ( e.g., Ravuri & Vinyals, 2019), we think substituting (9) into (8) is not a good idea.
Essentially, by substituting (9) into the classiﬁcation objective, one is regularizing f to remain close to f ∗ g , which is not an effective strategy if f ∗ g is not competitive.
This argument corroborates the evidence from our ablation studies and recent work showing that using class-conditional generative models to augment supervised learning does not provide big gains (Ravuri & Vinyals, 2019).
That said, one can still use class-conditional generative models to synthesize high-ﬁdelity samples.
As long as these samples are treated as unlabeled examples and annotated using a classiﬁer, e.g., f t , we believe this is a reasonable approach falling under GAL.
Our argument above only applies to the scenario that class-conditional generative models are used to synthesize labeled examples.
E XPERIMENTS
We asses the effectiveness of GAL on KD, self-training and few-shot learning for NLP.
We also present self-training results on tabular tasks.
Appendix B shows the applicability of GAL to two image classiﬁcation tasks as a proof of concept, but more advanced techniques such as Mixup (Zhang et al., 2018) are needed to bridge the gap with the state-of-the-art.
5. S TATE - OF - THE - ART R ESULTS ON K NOWLEDGE D ISTILLATION ON GLUE
We use the GLUE benchmark (Wang et al., 2019b) for our KD experiments; see Appendix D for benchmark details and Appendix E for the details of synthetic text generation.
Our synthetic unlabeled dataset U includes 40 × as many examples as the original dataset for each task in the GLUE benchmark.
The goal of knowledge distillation (KD) (Bucilu ˇ a et al., 2006; Hinton et al., 2015) is to distill the knowledge of a powerful teacher model into a compact student model with as little loss in performance as possible.
This can help with model compression (Jiao et al., 2019; Sun et al., 2019a) and multi-task learning (Liu et al., 2019a;
Clark et al., 2019).
It is known that KD on fresh data, unseen during training, performs better (Buciluˇa et al., 2006; Chen et al., 2020c) than KD on original training data.
Accordingly, we investigate the effectiveness of knowledge distillation using generated unlabeled data through GAL. [Table]
[Table] [Table] We use the HuggingFace implementation (Wolf et al., 2020) for KD experiments and adopt a standard experimental setup consistent with previous work (Sun et al., 2019a; Xu et al., 2020).
Following
Rashid et al.
(2021), ﬁne-tuned RoBERTa-large (24-layer transformer) represents the teacher and
a
DistilRoBERTa (6-layer transformer)
(Sanh et al., 2019) is used as the student.
We train the student model on U and L , where U is annotated by an ensemble of 10 models, achieving an average score of 87 .
We then mix U and L with a ratio of 1:4, which is equivalent to λ = 0 .
This ratio works best on the dev set.
Table 1 shows the results of individual 6-layer transformers on the GLUE test set.
All of the baselines use an identical student architecture.
GAL achieves the best entry on the GLUE leaderboard, marking a new state-of-the-art for KD on NLP.
It outperforms strong KD baselines such as DistilRoBERTa (Sanh et al., 2019), BERT-PKD (Sun et al., 2019a), BERT-Theseus (Xu et al., 2020), tinyBERT (Jiao et al., 2019) and tinyRoBERTa (Rashid et al., 2021).
It also outperforms our own
DistilRoBERTa+KD baseline, which learns from soft labels produced by an identical RoBERTalarge ensemble on the original labeled dataset.
While the use of soft labels outperform the vanilla ﬁne-tuned DistilRoBERTa model, it signiﬁcantly underperforms our KD+GAL baseline.
We also compare with round-trip translation (RT), a strong data-augmentation baseline ( e.g., Yu et al., 2018;
Shleifer, 2019).
We mirror the experimental setup of GAL and generate 40 × unlabeled data using
German as the bridge language (English → German → English).
The translations are generated via the best model in WMT19 (Ng et al., 2019). Although DistilRoBERTa+RT is better than vanilla DistilRoBERTa and KD variants, it still signiﬁcantly underperforms our approach. 5. NLP S ELF -T RAINING E XPERIMENTS ON GLUE
We ﬁne-tune pretrained RoBERTa models provided by fairseq (Ott et al., 2019) on each GLUE task.
Fine-tuned RoBERTa serves as the ﬁrst teacher model for self-training.
Each student model is initialized with the original pretrained RoBERTa and ﬁne-tuned with exactly the same hyperparameters as suggested by fairseq (Ott et al., 2019).
We combine the labeled dataset L and the synthetic dataset U with a ratio of 1:1, by oversampling labeled data.
This corresponds to λ = 0 .in
Eq.
Table 2 shows that GAL provides an average improvement of +1.3% over RoBERTa-base.
We see consistent improvements with more GAL iterations, but performance saturates after three iterations.
We further compare our approach with a self-distillation (Furlanello et al., 2018) baseline, in which the teacher and student models use the same architecture and transfer knowledge via the original labeled training set.
Although self-distillation provides a slight improvement, the gains from GAL are more signiﬁcant.
We delve deeper and combine GAL self-training with RoBERTa-large and report test results for both single model and ensemble model in Table 3.
We observe consistent gains coming from GAL on RoBERTa-large.
Our results underperform the latest and biggest LMs from the GLUE leaderboard [Table] but we are optimistic that GAL can be effectively combined with enormous LMs to provide additional gains. [Table]
[Table]
We assess the effectiveness of GAL self-training on four tabular tasks, namely connect-4 (Burton & Kelly, 2006), Drug Review (Gr ¨ aßer et al., 2018), Drybean (Koklu & Ozkan, 2020) and Spambase (Dua & Graff, 2017). The details of these tasks can be found in Appendix D. We follow the same protocol as GLUE tasks and generate 40 × unlabeled data from a ﬁne-tuned GPT-2-large.
Table 4 shows that GAL achieves sizable gains on these tasks even though neither RoBERTa nor GPTare optimized for tabular tasks. XGBoost (Chen & Guestrin, 2016), a strong supervised baseline for tabular tasks underperforms RoBERTa+GAL on connect-4 and Drug Review. It is worth noting that since inputs of Drug Review contain free form text, we convert them into numeric values through the max-pooled representation of the last hidden states of RoBERTa base. XGBoost outperforms
RoBERTa on Drybean and Spambase, but it is important to note that these two datasets include ﬂoating point attributes, which we simply tokenize using BPE (Sennrich et al., 2016) into subwords. Nevertheless, GAL is capable of bridging the gap between transformers and XGBoost. We believe GAL can be successfully combined with XGBoost, but we leave this to future work, since the XGBoost library does not easily accommodate soft labels.
[Table] [Table]
P ROMPT - BASED F EW - SHOT E XPERIMENTS
GPT3 (Brown et al., 2020) has introduced an optimization-free paradigm for few-shot learning for
NLP.
Without updating the parameters, large LMs can correctly predict the labels of the inputs by conditioning on a prompt, which consists of an instruction, a few labeled instances and a new unlabeled input.
We apply GAL to prompt-based few-shot learning.
Speciﬁcally, we present k labeled examples as a prompt to GPT-J (Wang & Komatsuzaki, 2021), an open-sourced re-implementation of GPT-3-6B [Table] and generate m synthetic examples, followed by the corresponding labels.
Note that to mitigate noisy outputs, the generation of each synthetic example only conditions on the original k labeled examples.
Finally, we concatenate the original k examples and m synthetic examples
[Table]
and conduct a ( k + m ) -shot learning experiment with GPT-J.
Brown et al. (2020) studied a total of 51 few-shot learning tasks.
Studying all of these tasks is prohibitively expensive. Thus, we ﬁlter tasks by following these two steps. First, since generating m synthetic examples for each test instance is computationally expensive, we exclude tasks that have more than 5k test examples. Second, we ﬁlter tasks on which GPT-3-6B achieves a score lower than 65% (please refer to Table H.1 in Brown et al.
(2020) for more details). After applying the ﬁltering steps, we use four datasets: SST-2 (Wang et al., 2019b), PIQA (Bisk et al., 2020), COPA and BoolQ (Wang et al., 2019a) as the testbed.
We notice that in order to generate valid synthetic data, GPT-J requires to see at least 4 labeled examples.
In addition, at most 16 examples of BoolQ can be fed into GPT-J without truncation.
Thus, we set k and m to 4 and 12 respectively.
As seen in Table 5, GAL leads to an average improvement of 1.2% over 4-shot learning, and reduces the gap between 4-shot and 16-shot learning.
We noticed that the quality of some generated examples is low.
We believe the performance of few-shot learning can be further improved with high-quality instances.
One solution is to generate many synthetic examples, and select a high-quality subset.
Since each test instance conditions on distinct labeled instances, one has to generate different synthetic instances for each test example from GPT-J, which causes expensive computation.
Due to such computational constraints, we leave the investigation of data selection strategies to the future work.
5. A BLATING C OMPONENTS OF GAL ON GLUE
[Table]
We conduct an in-depth study of different components of GAL on GLUE datasets.
Unless stated otherwise, we use a RoBERTabase model with a combination of the original training data and 40 × synthetic data for each experiment.
GPT-2 model size.
Radford et al.
(2019) present a few variants of the GPT-2 model including GPT-2 , GPT-2-medium , GPT-2large , and GPT-2-XL .
Larger GPT-2 models yield better perplexity scores and higher generation quality.
We utilize these models except GPT-2-XL within the GAL framework to study the impact of the generative model’s quality on downstream task’s performance.
Table 6 shows that regardless of the GPT-2 model sizes, GAL consistently surpasses the vanilla RoBERTa base. Moreover, SST-2 and RTE datasets are not sensitive to the capacity of the GPT-2 model, but higher quality synthetic text improves the results on MRPC and CoLA datasets.
We leave investigation of GPT-2-XL and even larger LMs such as GPT-3 (Brown et al., 2020) to future work.
Class-conditional synthetic data generation.
Previous work (Kumar et al., 2020b; Ravuri &
Vinyals, 2019) suggests that it is challenging to utilize synthetic data from class-conditional gener- ative models to boost the accuracy of text and image classiﬁers.
Our theory in Section 4.2 points to the potential drawback of class-conditional synthetic data.
We empirically study this phenomenon, by ﬁne-tuning GPT-2 in a class-conditional manner.
Table 7 shows that not only class-conditional
LMs underperform unconditional LMs in our GAL framework, but also they are much worse than the baseline.
[Table]
C ONCLUSION
We present Generate, Annotate, and Learn (GAL): a framework for self-training and knowledge distillation with generated unlabeled data.
We motivate GAL from an expected risk minimization perspective and demonstrate both theoretically and empirically that the use of unconditional generative models for synthetic data generation is more effective than class-conditional generative models, previously used in the literature.
GAL leverages advances in large pretrained language models to help supervised learning and can have implications for learning from limited labeled data.
GAL works surprisingly well on NLP and tabular tasks, and helps improve knowledge distillation and prompt-based few-shot learning.
We hope that GAL will stimulate new research on the evaluation and development of large language models.
A P SEUDO - CODE OF A LGORITHMS
B GAL ON I MAGE C LASSIFICATION T ASKS
As a proof of concept, in addition to NLP and tabular tasks, we assess the effectiveness of GAL on CIFAR-10 (Krizhevsky & Hinton, 2009) and Fashion MNIST (Xiao et al., 2017) as well.
We adopt the NCSN model of Song & Ermon (2019) as the task-speciﬁc generative model.
We use the CIFAR-10 model provided by the authors and train a model on Fashion MNIST using the same conﬁguration as CIFAR-10.
We select the model checkpoint resulting in the best FID score vs. training set (Heusel et al., 2017) based on 1000 samples.
We then use the NCSN models to generate up to 10 × synthetic unlabeled data, i.e., 500K for CIFAR-10 and 600K for Fashion MNIST.
See
Appendix C for representative samples.
[Table]
We adopt FixMatch (Sohn et al., 2020) to conduct semi-supervised learning on vision tasks, since FixMatch has shown promising results on CIFAR-10.
Speciﬁcally, we train a classiﬁer on mini-batches of intertwined labeled and unlabeled data (synthetic).
In each iteration, we obtain pseudo-labels for the unlabeled data, but ﬁlter unlabeled examples based on classiﬁer’s conﬁdence, i.e., examples are kept on which the largest class probability exceeds τ .
Weak augmentation is used to deﬁne pseudo labels, but strong augmentations are used to obtain student model’s predictions.
We randomly sample from the strong augmentations list deﬁned in RandAugment (Cubuk et al., 2020).
We only apply strong augmentations to the synthetic samples and not the original labeled data to ensure a fair comparison with the baseline.
We conduct experiments on three different convolutional neural network architectures: VGG19 (Simonyan & Zisserman, 2014), WideResnet28-10 (Zagoruyko & Komodakis, 2016), and
ResNet110 (He et al., 2016).
For the full list of hyperparameters and other implementation details, please refer to Appendix H. Each classiﬁer is trained for 200 epochs and 3 synthetic datasets of size ( 1 × , 5 × , 10 × ) of the training dataset are used.
Table B.1 shows that GAL achieves an average error reduction of 0 .
78% over the baseline on CIFAR10 across the 3 architectures tested.
Further, it appears that the larger the synthetic dataset size, the better the performance of GAL is.
We note that the reported results are the average of 3 independent runs.
Similarly on Fashion MNIST, we witness consistent gains across all architectures.
Fashion
MNIST results are included in Appendix I. Our image classiﬁcation experiments conﬁrm that even when the generative model of GAL is not pretrained on open domain data and solely trained on the dataset at hand, GAL can offer signiﬁcant improvements.
Table B.2 presents GAL results on Fashion MNIST dataset.
Similar to CIFAR-10, we observe a performance improvement across the three architectures.
[Table]
C G ENERATED U NLABELED E XAMPLES A NNOTATED WITH P SEUDO L ABELS
[Figure] [Figure]
pullover
[Figure]
When did the third Digimon series begin?
[SEP] Unlike the two seasons before it and most of the seasons that followed, Digimon Tamers takes a darker and more realistic approach to its story featuring Digimon who do not reincarnate after their deaths and more complex character development in the original Japanese.
( not entailment )
KNN:1:What is the name of the third season?
[SEP]
In addition to the ﬁrst two seasons, the third season is the season that introduced new characters such as Captain Malice, a supervillain who became the antagonist in season two; and the villains known as the Heartbreakers, who introduced a group of crime ﬁghters.
( not entailment ) 2: When did the ”Walking Dead” series end?
[SEP] In 2013, AMC announced that it would develop a ”superhero series”, which would follow the storylines and characters from the ”Walking Dead” series in order to bring the popular AMC original series to a new and younger audience.
( not entailment ) 3: What is the main objective of the ﬁrst season of the X-Files?
[SEP]
The ﬁrst season was notable in that the characters were introduced and developed within the space of a single season, as was the format of the show itself.
( not entailment )
What did Arsenal consider the yellow and blue colors to be after losing a FA Cup ﬁnal wearing red and white?
[SEP] Arsenal then competed in three consecutive FA Cup ﬁnals between 1978 and 1980 wearing their ”lucky” yellow and blue strip, which remained the club’s away strip until the release of a green and navy away kit in 1982–83.
( entailment
) KNN:1:Who was the most important player for Arsenal Football Club in the 1950s?
[SEP]
Wenger continued
to use Arsenal’s famous red shirts and red kits throughout the 1950s
and
1960s, and the red strip became the club’s most recognised and recognizable symbol.
( not entailment ) 2: When were the ﬁrst two teams to play for the trophy in the Premier League?
[SEP]
[Table] [Table]
EE
[Table]
Like the United States
[Table]
U.N. ofﬁcials are also dismayed that Aristide killed a conference called by Prime Minister Robert Malval in Port-au-Prince in hopes of bringing all the feuding parties together.
[SEP] Aristide had Prime Minister Robert Malval murdered in Port-au-Prince.
( not entailment )
[Table]
French President Nicolas Sarkozy was ad-
[Table]
by U.S. President George W. Bush about a possible trip to Iraq on Thursday
[Table]
is a member of the United States
[Table]
Only a week after it had no comment on upping the storage capacity of its Hotmail e-mail service
[Table]
Microsoft early Thursday announced it was boosting the allowance to
250MB
to follow similar moves by rivals such as Google, Yahoo
[Table]
and Lycos.
[SEP] Microsoft’s
Hotmail has raised
its storage capacity to
250MB
[Table]
( entailment )
[Table]
1:The company, known as Microsoft Ofﬁce, said it plans to sell all of the copies of its popular Ofﬁce suite at a loss in the wake of the launch of Microsoft Windows 7, saying it will also make $25 million in advertising costs, a move likely to hurt its long-standing position among consumers and business leaders.
[SEP] Microsoft Ofﬁce is a popular ofﬁce suite.
( entailment ) 2: The company’s
A BMI of 25 or above is considered overweight ; 30 or above is considered obese .
[SEP] A BMI between 18.5 and 24.9 is considered normal , over 25 is considered overweight and 30 or greater is deﬁned as obese .
( paraphrase ) KNN:1:
The report said that the average woman in her twenties who takes oral contraceptives daily can expect a loss of around 40 per cent of her bone density between the ages of 20 and 45 .
[SEP] The study said the average woman in her twenties who used the pill every day , or every day for up to ﬁve years , can expect a loss of about 40 per cent of her bone density between the ages of 20 and 45 .
[Table]
2: The report found that 17 percent of U.S. adults between ages 18 and 64 have a body mass index at or above the ” normal ” 20 .
[SEP] For people of that age , 17.1 percent of adults have a body mass index at or above the ” normal ” 20 , while 12.6 percent have a body mass index of 30 or above .
( not paraphrase ) 3:
The survey shows the proportion of women between 20 and 44 who were obese was 6.3 percent , up from 5.7 percent in 2001 .
[SEP]
The proportion of women betweenand 44 who were obese increased to 6.3 percent from 5.7 percent in 2001 .
( paraphrase ) Shares of Genentech , a much larger company with several products on the market , rose more than 2 percent .
[SEP] Shares of Xoma fell 16 percent in early trade , while shares of Genentech , a much larger company with several products on the market , were up 2 percent .
( not paraphrase ) KNN:1:Shares in Aventura fell as much as 5 percent , while shares in Medi-Cal climbed 2.
percent .
[SEP] Shares in Aventura were up 2.5 percent , while shares in Medi-Cal rose 2.5 percent .
( paraphrase )
[Table] [Table]
Conceptually cream skimming has two basic dimensions - product and geography.
[SEP] Product and geography are what make cream skimming work
[Table]
KNN:1:There are two main types of analysis and they are the case study and the case report.
[SEP] The case study is the most popular method used to analyze a subject.
( neutral ) 2:
A third approach to capturing and using this type of experience is to engage the program management and ﬁnance systems of the organization.
[SEP] There are two strategies to capturing and using experience.
( contradiction ) 3:
The ﬁrst is to see the basic elements of a business model in action.
[SEP] Basic elements of business models are the most important for the success of any company.
( neutral )
I don’t mean to be glib about your concerns, but if I were you, I might be more concerned about the near-term rate implications of this $1.
[SEP] I am concerned more about your issues than the near-term rate implications.
( contradiction ) KNN:1:I’m not here to tell you of my own experiences, but they are important to others who might have similar concerns.
[SEP] If you were to have similar concerns, I’d like to encourage you to tell them to me.
( neutral ) 2: I don’t mean to sound judgmental, but as a person, I think that’s an issue you’re probably pretty much on your own if you think about it.
[SEP] You’re probably right if you think about it.
( neutral ) 3:
But I don’t mean to take your word for it.
[SEP] I know you are correct, but I want to make sure it’s clear that I do not agree.
( contradiction )
are more deeply thought through than in most ‘ right-thinking ’ ﬁlms ( positive ) KNN:1:isfar more sophisticated , insightful and thought-provoking than his previous ﬁlms .
( positive ) 2: is more sophisticated than its more obvious and less-than-dazzling counterparts ( positive ) 3: is about as well-thought as the idea of a bad hair day , ( negative ) contains no wit , only labored gags ( negative ) KNN:1:lacks insight , and lacks empathy ( negative ) 2: has little humor or intelligence ( negative ) 3: lacks all wit and humanity ( negative )
D D ATASETS
[Table]
[Table]
E G ENERATING S YNTHETIC T EXT FOR GLUE
To generate domain-speciﬁc synthetic data, we ﬁne-tune GPT-2-large on the training set of each downstream task, excluding labels.
For tasks with multiple input sentences, we concatenate input sentences into a long sequences and separate sentences by special [SEP] tokens.
We generate new domain-speciﬁc data by using top-k random sampling similar to Radford et al.
We do not feed any prompt to the LM, but a special [BOS] token to initiate the generation chain.
A generation episode is terminated when a special [EOS] token is produced.
We generate diverse sentences by varying the random seed.
After collecting enough synthetic data, we only retain unique sentences.
For tasks with α input sentences, we discard generated samples that violate this constraint (approximately 10% of samples were rejected).
Quality of synthetic dataset.
An effective generative model of text should learn the word preferences and genre associated with a given corpus, but still produce novel sentences.
In order to study the characteristics of our synthetic datasets, Table E.1 reports the number of unique n-grams in the training and synthetic datasets, as well as the number of unique n-grams shared between them.
The high degree of overlap on uni-grams suggests that the ﬁne-tuned GPT-2-large is somewhat domainspeciﬁc.
Meanwhile, the large number of unique n-grams in the synthetic dataset suggests that many novel word combinations are generated, which is helpful for GAL.
[Table]
F I MPORTANCE OF P SEUDO - LABELS
We have argued and demonstrated that using class-conditional generative models to generate labeled synthetic examples is less effective than GAL in section 4 and section 5.
To further verify this argument, we sample 100 instances from the synthetic RTE dataset generated by a class-conditional
LM.
Then we annotate these examples using a human annotator and the RoBERTa classiﬁer.
Finally, we compute the Accuracy, F1, Precision and Recall scores between human labels and RoBERTa labels, and between human labels and conditioning labels (i.e., labels that the class-conditional
LM conditions on.).
Table F.1 shows that class-conditional LM has difﬁculty generating sentences retaining the semantics or pragmatics of a speciﬁed category, which also corroborates our theoretical analysis in section 4.
On the other hand, RoBERTa is able to produce higher quality labels that correlate better with human annotations.
[Table]
G GPT-2 FOR CLASSIFICATION
We have conducted additional experiments, where we ﬁne-tune GPT-2 as a classiﬁer.
We have considered two variants of the GPT-2 model.
The ﬁrst varant is the original GPT-2 model (GPT2original) pre-trained on open-domain text.
The second variant is the GPT-2 model that was ﬁnetuned on the inputs of each task separately (GPT-2-ﬁnetuned).
This model was used to generate task-speciﬁc (synthetic) unlabeled data.
Finally, we also consider self-training with GAL on top of GPT2-original.
Speciﬁcally, we use the GPT-2-ﬁnetuned model to synthesize 40x in-domain unlabeled data.
Then we apply self-training to GPT-2-original, where the data is a combination of the original labeled data and pseudo-labeled synthetic data.
Table G.1 suggests that the gains of
GAL come from the pseudo-labeled synthetic data, i
[Table]
e., both synthetic unlabeled data and teacher’s knowledge.
Without the generation of synthetic unlabeled data, the domain-speciﬁc knowledge embedded in GPT-2-ﬁnetuned model cannot be utilized.
As such, GPT-2-ﬁnetuned model is inferior to the GPT2-original model.
[Table]
H T RAINING D ETAILS
We use the fairseq codebase (Ott et al., 2019) for implementing both NLP and tabular experiments.
Training details are summarized in Table H.1 and Table H.2.
We use the HuggingFace codebase (Wolf et al., 2020) for KD experiments.
All NLP models are trained for 5 epochs with a learning rate of 2e-5 and a batch size of 32.
All experiments are run on a single Nvidia V100 GPU.
For the CV tasks, we ﬁrst use the ofﬁcial implementation of NCSN (Song & Ermon, 2019) to generate the synthetic images for CIFAR-10 and Fashion MNIST.
We use the pretrained checkpoints provided by the authors for the generation of synthetic CIFAR-10 images and we train a new generative model for Fashion MNIST from scratch with the same hyperparameters of the
CIFAR-10 network.
After generating the synthetic images, we apply GAL using a FixMatch-like setup (Sohn et al., 2020), using the hyperparameters listed in Table H.4.
We follow Cubuk et al.
[Table]
[Table]
[Table]
(2020) for strong augmentations.
Finally, the backbone of the classiﬁers is from this codebase: https://github.com/bearpaw/pytorch-classification .
[Table]
[Figure]
[Table]
[Figure]
[Table]
I A DDITIONAL D ETAILS
In Tables I.1, and I.2, we present some descriptive statistics of our CIFAR-10 synthetic image dataset to complement the samples shown in Figure C.1 and to help shed some light on the nature of the images generated by the NCSN network.
[Table] [Table]