Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
Abstract
Pretrained language models (PLMs) have demonstrated remarkable performancein various natural language processing tasks:
Unidirectional PLMs ( e.g. , GPT) arewell known for their superior text generation capabilities; bidirectional PLMs ( e.g. ,
BERT) have been the prominent choice for natural language understanding (NLU)tasks.
While both types of models have achieved promising few-shot learningperformance, their potential for zero-shot learning has been underexplored.
In thispaper, we present a simple approach that uses both types of PLMs for fully zero-shotlearning of NLU tasks without requiring any task-specific data: A unidirectional
PLM generates class-conditioned texts guided by prompts, which are used asthe training data for fine-tuning a bidirectional PLM.
With quality training dataselected based on the generation probability and regularization techniques (labelsmoothing and temporal ensembling) applied to the fine-tuning stage for bettergeneralization and stability, our approach demonstrates strong performance acrossseven classification tasks of the GLUE benchmark ( e.g. , 72 . 3 / 73 . 8 on MNLI-m/mmand 92 . 8 on SST-2), significantly outperforming zero-shot prompting methods andachieving even comparable results to strong few-shot approaches using 32 trainingsamples per class 1 .
Introduction
Pretrained language models (PLMs) [5, 8, 11, 19, 33, 39] have achieved human-level performanceon natural language understanding (NLU) tasks [63, 64] when fine-tuned on a large amount oftask-specific training data.
However, such a supervised fine-tuning paradigm is drastically differentfrom how humans perform these tasks: We barely need to see many task-specific training samples toperform well.
Recently, many studies have revealed the intriguing few-shot learning potential of PLMs: Byconverting task descriptions to natural language prompts and injecting them into PLMs, prompt-basedapproaches [5, 13, 53, 54, 56] leverage task-specific information for better training data efficiencyand have achieved remarkable few-shot results.
When prompt-based methods are applied to the zero-shot setting, however, the PLMs’ predictionsare much less accurate.
For example, GPT-3’s zero-shot performance is much degraded relative toits few-shot performance [5], especially on challenging tasks like natural language inference (NLI).
Without any task-specific samples, it is indeed challenging for PLMs to effectively interpret theprompts that come in different formats and are unseen in the pretraining data.
To familiarize PLMswith various prompts for zero-shot generalization to unseen tasks, a recent study proposes instructiontuning [67], which fine-tunes PLMs on a large collection of different tasks described by instructions.
Despite its strong performance, its success is grounded in the large number of cross-task annotated1 Code is shared in the supplementary material.
datasets ( e.g. , train on many non-NLI tasks and transfer to NLI tasks) and the gigantic model size( e.g. , hundreds of billions of parameters), posing great challenges for training and using such modelsin most real-world applications.
In this work, we study zero-shot learning of PLMs on NLU tasks without any task-specific or crosstask data.
Motivated by the strong text generation power of recent PLMs [5, 23, 29, 50], we propose
SuperGen, a Super vision Gen eration approach, wherein training data are created via a unidirectional
PLM ( i.e. , the generator) which generates class-conditioned texts guided by label-descriptive prompts.
A bidirectional PLM ( i.e. , the classifier) is then fine-tuned on the generated texts to perform thecorresponding task.
Both PLMs can be of moderate size to fit in typical research hardware ( e.g. , a
GPT-2-sized [49] generator and a RoBERTa Large -sized [33] classifier).
With supervision automaticallycreated by the generator, SuperGen eliminates the need for task-specific annotations and providesthe classifier PLM with a larger amount of training data than in few-shot scenarios.
SuperGen iscompatible with any PLM as the classifier and any fine-tuning method.
Across seven classification tasks of the GLUE benchmark [63], SuperGen significantly outperformsthe prompt-based zero-shot method and even achieves an overall better result in both averageperformance and stability than strong few-shot approaches that use 32 annotated samples per class.
We identify several key factors to the strong performance of SuperGen through ablation studies: (1)selecting quality training data based on their generated probability, and (2) using label smoothing andtemporal ensembling to regularize fine-tuning on generated data.
Related Work
2.1 Few-Shot and Zero-Shot Learning with PLMs
Instead of using a large amount of annotated training data for fine-tuning PLMs on downstreamtasks, few-shot learning studies how to better leverage only a small amount of task-specific trainingdata, a more realistic scenario in many applications.
The most strict few-shot learning setting doesnot assume access to any unlabeled data or large validation sets for hyperparameter tuning [46],where prompt-based methods [5, 13, 32, 34, 53–56, 60, 79] are prominently deployed to inject taskdescriptions into PLMs and make effective use of their language modeling capability for improvedtraining data efficiency in low-data regimes.
More broadly, semi-supervised learning additionallyleverages unlabeled task-specific data, where data augmentation [7, 70], regularization [41] andbootstrapping [54] methods are commonly used.
Zero-shot learning, on the other hand, is a much more challenging setting with absolutely no accessto any task-specific data.
When prompt-based methods are directly used to obtain predictions from
PLMs without any training, their zero-shot performance can be much worse [5, 13]—difficult NLUtasks can be barely formulated as prompts that resemble the format of pretraining data, posing greatchallenges for PLMs to accurately interpret and leverage the prompts without given any trainingsamples.
The current mainstream of zero-shot learning is based on transfer learning: By convertinga set of tasks with abundant annotations into instruction templates [40, 52, 67, 71], entailmentpairs [75, 76] or question-answer formats [48, 81] and fine-tuning PLMs on them, the PLMs acquirethe cross-task transfer ability [74] to execute unseen tasks when they are formulated in a similarformat.
Our work proposes a different approach from these studies: We use a unidirectional PLM togenerate training data for fine-tuning another PLM on the target task.
This not only removes the needfor a large amount of cross-task annotations, but also eliminates the task difference in training andinference.
Moreover, different from previous studies [1, 73] that rely on labeled data to fine-tune thegenerative PLM, we directly use prompts to guide data generation without fine-tuning.
2.2 Controlled Text Generation with PLMs
Controlled text generation [22] aims to steer the generated texts of language models towards desiredcontents, styles or domains.
Through fine-tuning PLMs on attribute-specific data, high-level control( e.g. , generating certain topics or sentiments [83]), fine-grained control ( e.g. , generating specific wordsor phrases [6]) or both [24] can be achieved.
Adapting PLMs to generate texts of specific attributes canalso be realized at inference time without any further training of the PLMs [10, 25, 26, 31, 45, 65, 72].
Different text attributes can also be represented during pretraining time as control codes [23] whichlater can serve as explicit guidance for generating domain/attribute-specific texts.
Along another line of controlling text generation, the idea of using prompts as guidance has emergedrecently—Since natural language generation is largely based on contexts, providing certain prompts
[Table]
[Table] [Table]
[Figure]
as the beginning of a sequence can effectively steer the subsequent texts to be generated.
The promptscan be either in natural language [55] or as learnable parameters [30].
In this work, we also guide textgeneration via prompts, but for the novel purpose of creating training data for NLU tasks.
The ideaof using generated texts as training data has been explored in topic classification with bag-of-wordsor LSTM-based language models [37, 38] which do not have enough capacity to generate qualitytraining data for challenging NLU tasks.
Compared to annotated task-specific data, the generatedtexts may contain noise and have domain difference from the downstream task.
We will introduceseveral important strategies for effective fine-tuning of PLMs on generated data.
Method
3.1 Preliminaries
Problem Formulation.
We consider solving a classification problem 2 where we are only given thelabel space Y and a mapping M : Y
→ W that converts each label y ∈ Y into a label-descriptive 101 prompt ( i.e. , a short phrase) w y ∈ W .
We assume access to a unidirectional PLM G θ as the generatorand a bidirectional PLM C ϕ which will be fine-tuned as the classifier 3 .
We also assume the pretrainingcorpus D ( e.g. , Wikipedia) is available.
Text Generation with Unidirectional PLMs.
A unidirectional PLM G θ is pretrained to maximize the generation probability of each token in a sequence x = [ x 1 , x 2 , . . .
, x n ] conditioned on previous tokens:
[Equation]
Here, p θ ( · ) is usually parameterized using token embeddings e and contextualized embeddings h 105 given by a Transformer [62] encoder.
After pretraining, G θ can be directly used to generate new texts by recursively sampling tokens fromits output probability distribution.
Typically, a temperature hyperparameter τ > 0 is introducedduring sampling [20] to adjust the sharpness of the probability distribution:
[Equation]
where τ → 0 approximates greedily picking the most probable next token; τ → ∞ induces a uniform 110 distribution.
Additionally, sampled tokens can be confined to the top- k most probable ones to avoid regression tasks this the texts continuous label space
However there classification
SuperGen low-quality tokens.
In this work, we find such top- k sampling with temperature is sufficient toproduce coherent and meaningful texts as training data for NLU tasks.
Exploring more sophisticatedsampling strategies [21] is left for future work.
3.2 Training Data Generation
[Table]
When given a label-descriptive prompt suchas “Write a negative review:”, humans areable to produce texts pertaining to the cor-
[Table]
responding class.
We aim to leverage thestrong text generation power of a unidirec-
[Table]
tional PLM G θ for the same purpose of cre-
[Table]
ating class-conditioned training data.
Wenote that G θ is directly used for generation
[Table]
without any parameter updates.
The prompts
[Table]
used for different NLU tasks in GLUE are
[Table]
summarized in Table 1.
Generating Single Sequences.
For singlesequence NLU tasks such as sentiment classification ( e.g. , SST-2), we simply use a prompt w y corresponding to label y as the beginning of the sequence and let G θ generate the remaining sequence:
[Equation]
where G θ ( w y ) denotes using w y as the in- 127 put to G θ and recursively sampling tokens 128 from the distribution in Eq.
(1) until a fullsequence is generated; x g denotes the generated sequence ( i.e. , excluding the prompt),which will be paired with y to form one training sample ( x g , y ) .
For syntactic tasks like linguistic acceptability classification ( e.g. , CoLA) which requires generating both linguistically acceptable and unacceptable sequences, we start the sequence with random stop words and use varying sampling temperaturesfor generating different sequences.
A smaller temperature ( e.g. , τ = 0 .
1 in Equation (1)) sharpens thesampling probability distribution towards the most probable tokens, thus the resulting sequence willmore likely to be linguistically acceptable.
Using a larger temperature ( e.g. , τ = 10 in Equation (1))flattens the sampling probability distribution to be more uniform, and the generated tokens will benearly random, which can create linguistically incorrect sequences.
Generating Sequence Pairs.
Sequence-pair classification tasks require generating two sequences of specific relationships ( e.g. , entailment, contradiction).
We sample 4 the first sequence x s from the pretraining corpus D , concatenate the prompt w y with x s , and let G θ generate the second sequence x g :
[Equation]
The sequence pair training sample will then be formed as ( x s , x g , y ) .
Rewarding and Penalizing Repetitions for Sequence Pair Generation.
A common issue in textgeneration is degenerate repetition [21, 23, 49, 68] where generated texts can get stuck in repetitionloops.
To address this issue, one approach is to discourage repetition by reducing the logits of tokensthat are already in the sequence before performing sampling [23].
In sequence pair generation,however, it is sometimes desirable to encourage the second sequence to repeat some words in thefirst sentence ( e.g. , for generating an entailment or a paraphrase).
Therefore, we propose a simplemodification of Eq.
(1) that rewards/penalizes repetition based on whether the token has appeared in x s / x g :
[Equation]
exp( e
[Equation] [Equation]
(cid:80) |
[Equation]
| j =1 exp( e ⊤ j h i /ω ) , where ω =   τα x i ∈ x s ∧ x i / ∈ x g τβ x i ∈ x g τ else , (2) and α > 0 , β > 0 are hyperparameters.
By setting α < 1 and β > 1 , we can promote tokens inx s that have not appeared in x g to have a higher chance of being generated, and discourage thegeneration of repetitive tokens in x g to mitigate degenerate repetition.
The parameters used fordifferent tasks are listed in Appendix C Table 8.
3.3 Effective Fine-Tuning on Generated Texts
With the generated training data, one can fine-tune a bidirectional PLM C ϕ as the classifier to performthe NLU task.
However, training C ϕ via standard supervised training on all generated texts is likelyto yield suboptimal performance on downstream tasks because (1) the generated texts may containnoise as G θ may not always produce texts pertaining to the desired class, especially for challengingsequence pair tasks with subtle semantic relationships; and (2) the generated texts can be consideredas originated from the domain of G θ ’s pretraining data, with a potentially different distribution fromthe downstream task; straightforward application of supervised training will result in overfittingto the pretraining domain and diminishing generalization ability, a common challenge in transferlearning [61, 82].
To address these challenges, we next introduce several simple and importantstrategies for more effective and stable fine-tuning on generated texts.
Selecting Quality Training Data.
We aim to select generated texts x g that are most likely to pertain to the desired label
y
( i.e. , with the highest p ( x g | y ) ).
The true probability p ( x g | y ) is unknown and we estimate it via the generation probability given by G θ conditioned on the prompt w y :
[Equation]
Since the above measure is biased towards shorter sequences, we instead use the geometric meanof the above conditional generation probability (or equivalently, the average log probability) of alltokens in x g as the ranking score, following [77]:
[Equation]
To construct a training set consisting of N samples per class, we will generate more samples ( e.g. ,10 N ), and select training data based on the score r in Eq.
For all tasks except CoLA, thetop-
N ones of each class are selected; for CoLA, the top- N ones are used as the training sampleas linguistically acceptable sequences, and the bottom- N ones are as linguistically unacceptablesequences.
Regularization for Better Generalization and Stability.
Even with the above training dataselection procedure, the resulting training set may still contain noise and there exists domain differencefrom the downstream tasks.
We apply two regularization techniques, label smoothing [59] andtemporal ensembling [27] for better fine-tuning stability and generalization.
Given a training sample ( x g , y ) , label smoothing trains the classifier C ϕ to minimize the standardcross-entropy loss between the label and the classifier’s prediction p ϕ
( x g ) , except that the label is aweighted average of the one-hot vector and a uniform distribution over all labels:
[Equation]
where q j = 1 ( j = y )(1 − ϵ ) + ϵ/ |Y|
and ϵ is the smoothing weight.
By forcing the classifier to be 181 less confident on training data, label smoothing improves robustness to label noise [35] and preventsoverfitting to the training set [42], thus improving generalization to different domains.
The motivation for temporal ensembling is that neural networks usually first pick up easy and generalpatterns in the data before learning more sophisticated and dataset-specific features [78], and thus the earlier states of the network offer better generalizability to different domains.
We therefore record thepredictions p ϕ =
p ϕ ( x g ) of C ϕ on each training sample ( x g , y ) at different training steps, and usethe accumulated moving-average predictions ¯ z to regularize the latest model training.
This also helpssuppress the fluctuation in model predictions due to data noise, offering better noise-robustness [43].
We update ensembled predictions ¯ z once every B batches:
[Equation]
where ˆ z has a zero initialization; γ is the momentum parameter; t is the number of updates ¯ z hasreceived; the division (1 − γ t ) is for bias correction [27].
We also use the ensembled prediction ¯ z asa reliable signal to filter out noisy training samples: Only those samples on which ¯ z strongly agreeswith the label y ( i.e. , ¯ z y > δ where δ > 0 is a threshold parameter) will be used for training.
We regularize model training by extending Eq.
(4) to add a KL divergence regularization term fromthe model prediction to the ensembled prediction weighed by λ :
[Equation]
We follow [27] to slowly ramp-up λ during training.
3.4 Overall Algorithm
[Table]
// Class y train set init.
for i ∈ [1 , 2 , . . .
, M ] do x g ← G θ ( w y )
T y ← T y (cid:83) { ( x g , y ) } end end
T ← {} //
Selected train set.
for y ∈ Y do Sort T y in descending order by Eq.
T ← T (cid:83) T y
[: N ] end ˆ z ←// Ensembled prediction init.
T ∗ ← T //
Filtered train set.
for i ∈ [1 , 2 , . . .
, T ] do Fine-tune C ϕ via Eq.
(6) on a minibatch of T ∗ if i % B = 0 then Update ˆ z , ¯ z via Eq.
T ∗ ← { ( x g , y ) | ¯ z y > δ, ( x g , y ) ∈ T } end endreturn C ∗ ϕ =
C ϕ
We summarize SuperGen for singlesequence NLU tasks in Algorithm 1.
Solving sequence-pair problems follows thesame algorithm except the pretraining corpus D is needed for sampling the first se- 203 quence x s .
Experimental Setup
Downstream Tasks and Metrics.
Weuse all the tasks included in GLUE [63]except STS-B which is a regression task.
Please refer to Appendix A for more details about GLUE tasks.
We follow theevaluation protocol of [13]: We use F1score as the metric for QQP and MRPC,
Matthews correlation for CoLA, and accuracy for the rest of the tasks.
The originaldevelopment sets of these tasks are usedfor testing.
For all reported results, we include the average and standard deviationover 5 different random seeds.
Models.
Unless specified otherwise, weuse CTRL ( 1 . 63 B parameters)
[23] as thegenerator G θ and COCO-LM Large ( 367 M 221 parameters)
[39] as the classifier C ϕ .
Wealso show the results using similar-sized
PLMs (GPT-2 [49]/RoBERTa [33]) as thegenerator/classifier in Appendix D.
Fine-Tuning Settings and Hyperparameters.
We note that SuperGen is compatible with any fine-tuning method; while using more sophisticated methods may grantfurther performance improvement, we use the basic prompt-based fine-tuning with manual templatesapproach for simplicity and clarity.
For all tasks, we use the same templates and label words as in[13].
Under the zero-shot learning setting, it is not possible to tune hyperparameters due to the lack
[Table]
[Table]
of validation sets
Therefore, we keep all fine-tuning hyperparameters ( e.g. , learning rate, batch size,training epochs, number of generated training samples, label smoothing and temporal ensemblinghyperparameters) same across all tasks.
See Appendix C Table 9 for details.
Compared Methods and Ablations.
We include the results of zero-shot prompting, standardfew-shot fine-tuning and the four few-shot prompt-based fine-tuning methods proposed in [13].
Wealso conduct ablation studies by removing the following three techniques from SuperGen one at atime: (1) not using Eq.
(3) for training data selection but randomly selecting the same amount oftraining data (- data selection); (2) not using label smoothing (- label smooth) but using one-hotlabels; and (3) not using temporal ensembling ( i.e. , using Eq.
(4) instead of Eq.
(6) as the trainingobjective) (- temporal ensemble).
Lastly, we report the fully supervised fine-tuning results trained onthe entire training sets.
Evaluation
5.1 Main Results
We present the results of SuperGen, its ablations and compared methods in Table 2.
Overall, SuperGensignificantly outperforms zero-shot prompting and achieves an overall better result than all few-shotmethods.
Notably, SuperGen results in much smaller variance over different random seeds thanfew-shot approaches on most tasks—with access to more training data, fine-tuning of PLMs becomesmuch more stable.
The ablation results demonstrate that all three strategies ( i.e. , quality trainingdata selection, label smoothing and temporal ensembling) play important roles in improving andstabilizing the final performance, especially on challenging tasks like MNLI.
5.2 Using Different Prompts
One important factor of SuperGen is the choice of label-descriptive prompts as they directly influencethe quality of generated training samples.
To study the impact of different prompt choices on the final model performance, we create different groups of prompts other than the original ones.
Wereplace the prompt for one label used in Table 1 with a synonymous one and keep other promptsunchanged when forming a different prompt group (Please refer to Appendix B Table 7 for details).
We also experiment with mixing the generated data by different prompt groups (mixed).
The resultsare shown in Table 3.
Overall, the model performance under different prompts is quite close, excepton RTE whose test set is very small, potentially resulting in the higher variance.
In this work, wemanually choose simple prompts that make intuitive sense, and we leave the automatic searching ofoptimal prompts as future work.
5.3 Results with Different Amount of Generated Data
[Figure]
With training data automatically created by the generator, we can havea virtually infinite amount of training samples.
We show the resultsof using different amount of generated data (after quality data selection) for fine-tuning the classifier
C ϕ in Fig.
2 on MNLI-m and SST-2.
When the number of training datais small ( e.g. , 100 ), the fine-tuningvariance is high, resulting in the similar instability issue with few-shotsettings.
With more generated dataused, both average performance andtraining stability improve, yieldingcomparable results (with smaller variance) to fine-tuning using few-shot task-specific data.
However,when too many generated data ( e.g. , 10 , 000 ) are used, the classifier’s performance slightly drops,probably due to increased label noise—recall that the training data are selected based on the rankingscore in Eq.
(3), so using more data results in the inclusion of more lower-ranking texts in the trainingset and reduced data quality.
One way to address this issue is to use a fixed selection ratio and increasethe total number of generated texts to obtain a larger number of high-quality training data.
However,this comes at a greater computation cost in the generation step.
An important future direction is thusto develop better data selection strategies.
5.4 Using Generators for Knowledge Distillation
[Table] Apart from using unidirectional PLMs G θ for training data generation, one could also directly applythem to unlabeled data formulated as prompts to obtain zero-shot predictions ( i.e. , prompting [5, 13]),which can then be used as soft labels to train theclassifier C ϕ . In Table 4, we show (1) the zeroshot prediction accuracy of CTRL (the best out ofthree different prompts, details in Appendix F) and(2) the classifier performance trained from CTRL’spredictions on the entire unlabeled training set assoft labels ( i.e. , knowledge distillation). Similarto the observations in previous studies [5, 67, 80],the zero-shot predictions of unidirectional PLMs are quite inaccurate and directly using them assoft labels to train classifiers does not yield good results.
We hypothesize that the advantages ofusing unidirectional PLMs for training data generation over using them for zero-shot predictions aretwofold: (1) Better flexibility in prompt formats. When unidirectional PLMs are used for zero-shotpredictions, the prompts have to be designed so that the label word is the last token in the sequence tobe predicted, as unidirectional PLMs cannot attend to subsequent tokens. Such constraints may resultin the prompt being dissimilar to the pretraining data distribution and worsen the prediction qualityof the PLMs. On the contrary, using unidirectional PLMs for generation is not subject to any promptformat constraints.
(2) More direct uses of PLMs’ language modeling ability. Using unidirectional PLMs for training data generation directly leverages the PLMs’ output token probability.
Applying PLMs for zero-shot prediction, however, requires an additional step to convert token predictions to label predictions ( i.e. , the verbalizer [54]), and such a mapping process usually necessitates manualcuration and can hardly be optimal [13] especially without abundant task-specific data.
Using SuperGen in Few-Shot Settings
[Table]
We present a simple extension of SuperGen to few-shot settings and show thatthe generated data of SuperGen may further improve the few-shot performance.
When few-shot samples are available, wefirst fine-tune the classifier on the fewshot training set (standard prompt-basedfine-tuning without regularization), andthen continue fine-tuning the classifieron the generated data by SuperGen asdescribed in Section 3.3. This allows the classifier to effectively leverage the knowledge fromthe few-shot training set to filter out noisy samples in the generated data, as temporal ensemblingregularizes the classifier to remember the predictions learned previously and only keeps samples onwhich the model predictions agree with the label. As shown in Table 5, such a simple approach thatapplies SuperGen to few-shot settings improves both zero-shot SuperGen and few-shot prompt-basedfine-tuning. We note that few-shot samples are not used in the training data generation stage, andwe expect the results to be even better if they are leveraged to generate training data closer to thetask-specific distribution.
Possible ways to use few-shot samples for generation include using themas demonstrations [5], for creating augmentations [28] and for tuning the generators. We leave theexplorations of generating higher quality data by leveraging few-shot samples for future work.
Discussions and Conclusions
Ethical Considerations.
While PLMs have demonstrated remarkable text generation and understanding capability, they can come with potential risks or harms [2, 3, 5] such as generatingmisinformation [44] or amplifying harmful biases [47].
The focus of our work is on utilizing existing PLMs to generate training data for NLU tasks instead of developing new PLMs or generationmethods.
Therefore, our method can be used in company with any bias reduction and correctiontechniques [15, 36] to mitigate the risks of PLMs.
Limitations.
One inherent limitation with zero-shot learning is the lack of access to task-specificsamples for hyperparameter tuning, whereas the performance of neural networks is usually heavilydependent on the choice of hyperparameters even when the training algorithm and training set arefixed [46].
Also, without access to any labeled data, the generated training data quality may not behigh enough to achieve good performance on challenging tasks, especially when the task distributionis significantly different from the pretraining data distribution ( e.g. , the “linguistically incorrect” labelof CoLA requires generating sequences with grammar mistakes – a different distribution from the oneused to train PLMs).
A promising direction to address the above limitations is extending SuperGen tofew-shot settings and leveraging a small amount of labeled data for generating better quality data andfor hyperparameter tuning.
Conclusions.
We propose SuperGen, an automatic supervision generation approach for zero-shotlearning of NLU tasks.
By providing label-descriptive prompts as guidance to a unidirectional
PLM, training data can be automatically created for fine-tuning a bidirectional PLM.
Our frameworkdiffers from previous transfer-learning-based zero-shot methods in that SuperGen does not rely oncross-task annotations and eliminates the task difference in training and inference.
We show thatseveral strategies are important for effective and stable fine-tuning on generated data, includingquality training data selection, label smoothing and temporal ensembling.
SuperGen achieves strongperformance on seven classification tasks of the GLUE benchmark, even yielding comparable orbetter results than sophisticated few-shot learning methods and offering better stability.
There is largeroom for future work, including but not limited to: Extension to few-shot learning settings, exploringlarger generator models, better fine-tuning techniques to leverage generated data and better strategiesfor selecting quality training data.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope
[Yes
b ) Did you describe the limitations of your work
[Yes]
(c) Did you discuss any potential negative societal impacts of your work
[Yes]
d) Have you read the ethics review guidelines and ensured that your paper conforms tothem
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]
A GLUE Tasks
We provide the details of the seven classification tasks included in the GLUE benchmark.
MNLI: Multi-genre Natural Language Inference [69] aims to predict whether a given premisesentence entails, contradicts or neutral with respect to a given hypothesis sentence. QQP: Quora Question Pairs [57] aims to determine whether a pair of questions asked are semanticallyequivalent. QNLI: Question Natural
Language Inference
aims to
predict
whether a given sentence
contains
the
answer to
a given question sentence.
SST-2: Stanford Sentiment
Treebank
[58] aims to determine if a movie review has positive or negativesentiment.
CoLA:
Corpus of Linguistic Acceptability [66] aims to determine
whether a given
sentence is
linguistically
acceptable or not
RTE: Recognizing Textual Entailment [4, 9, 16, 18] aims to predict whether a given premise sentenceentails a given hypothesis sentence or not.
MRPC:
Microsoft Research
Paraphrase
Corpus [
] aims to predict whether two sentences aresemantically
equivalent
or
not
B Details of Prompts Used for Different Tasks
[Table]
We present more details about the prompts used for different tasks in Table 6 which is an extendedversion of Table 1.
For SST-2, we fix the beginning of the generated sequence x g to be “The/this film/movie” to makesure the generated texts are related to movie reviews.
For CoLA, we start the generated sequence x g with a random stop word.
[Table]
For QNLI and QQP, the first sequence is always a question, and we require the sampled sequencex s to end with a question mark and begin with one of the following words: “how”, “what”, “why”,“who”, “which”, “where”, “when”, “whom”, “whose”.
For QNLI, the generated sequence x g for the “entailment” label is the one that immediately followsthe sampled sequence x s ; the generated sequence x g for the “not entailment” label is randomlysampled from the paragraph following x g excluding the first sequence that immediately follows x g .
We also show the different prompt groups used in the experiments of Section 5.2 in Table 7.
C Hyperparameters and Reproducibility
generating
[Table]
Hyperparameters for Generating
[Table]
Training Data.
Table 8 lists the hyperparameters used in the trainingdata generation stage.
For sequencepair tasks, we use greedy samplingfor better reproducibility.
For labelsthat require generating entailment,paraphrase, or equivalent sequencepairs, we set α ≤ 1 to encourage 642 word overlapping between the second sequence and the first sequence;otherwise, we set α = β > 1 to discourage word repetition.
To construct a training set consistingof N samples per class, we will generate M samples per class, and selecttraining data based on the score r in
Eq.
For all tasks except CoLAand the “neutral” label of MNLI, thetop- N ones of each class are selected;for CoLA, the top- N ones are usedas the training sample as linguistically acceptable sequences, and thebottom- N ones are as linguisticallyunacceptable sequences; for the “neutral” label of MNLI, we find it betterto randomly select N samples from
Fine-tuning-related hyperparameters ( e.g. , learning rate, batch size) follow the default values (when the validation set is not available) in Appendix A of [ regularization related hyperparameters the default
[Table]
[Table]
the total M samples instead of using the ranking score, probably because a neutral hypothesis withrespect to the premise has a wide range of possibilities ( i.e. , any hypothesis that is not entailed byor contradicts with the premise will be neutral), and random selection improves the diversity ingenerated hypotheses of the neutral label.
Hyperparameters for Fine-Tuning.
Table 9 lists the hyperparameters used in the fine-tuning stage.
We keep them same across all tasks except CoLA which uses δ = 0 because half of the training datafor CoLA are intentionally made to be of low quality ( i.e. , as linguistically unacceptable sequences)and there is no need to filter them out.
We follow [27] to slowly ramp-up λ in Equation (6) during thefirst 10 ensembles: λ ( t ) =
λ max exp( − 5(1 − t/ 10) 2 ) where t is the number of prediction ensembles 669 performed.
Computation Environment.
{+All experiments are conducted on NVIDIA GeForce RTX 
+}
GPUs.
SuperGen can be run on typical research hardware ( e.g. , with > 10 GB GPU memory).
Thegenerator PLM G θ does not need to be trained so a relatively large generator can be used ( e.g. , a1 .
63 B-parameter CTRL model).
D Using Different PLMs
The final performance is also relevant to the choice of PLMs as the generator/classifier.
Apartfrom the default PLM choice, we report the results of using GPT-2 XLarge ( 1 . 54 B parameters)
[49]as the generator and RoBERTa Large ( 356 M parameters)
[33] as the classifier in Table 10 with everything else unchanged.
When using GPT-2, we change the prompt used for SST-2 to “The film isbad/terrible/awful.”
for the negative label and “The film is good/great/excellent.”
for the positivelabel, since the original prompts used for SST-2 in Table 1 are a part of the control codes of CTRL andcannot be effectively leveraged by GPT-2.
Overall, both CTRL and GPT-2 are able to generate qualitytraining data for good fine-tuned classifier performance; CTRL consistently yields better resultsthan GPT-2 regardless of the choice of the classifier PLM, probably because CTRL is pretrainedwith control codes which provide explicit guidance for generating texts of certain domains andattributes.
We also observe that the generated text quality is strongly correlated to the generator’smodel size—using a smaller version of GPT-2 ( e.g. , with 117 M parameters) results in significantlyless coherent texts and can hardly serve as training data.
An interesting future direction is to try largergenerator PLMs ( e.g. , GPT-3) which may create training data of better quality.
[Table]
[Table]
E Case Studies
We present concrete examples of generated texts guided by prompts of different labels in Table
[Table]
The generated sequences are not only coherent, but also pertain to the corresponding labels.
Foreasier tasks like SST-2, the generated texts almost always correctly reflect the desired sentimentpolarity specified by the prompt.
For more difficult tasks like MNLI, sometimes the generated textsare not of the correct label.
Table 12 demonstrates more concrete examples where the generated textsmay not correctly pertain to the label of the prompt.
The existence of such label noise motivates ouruse of the regularization techniques in the fine-tuning stage
[Table]
We believe that larger generator PLMs ( e.g. , GPT-3 [5]) can bring about better text generation qualityand improve the accuracy in producing texts that pertain to the desired class.
Furthermore, betterfiltering strategies can be developed in the future to select training data with the correct labels.
F Knowledge Distillation Baseline Details
We show the concrete prompts used for the knowledge distillation baseline in Tables 13 and 14on MNLI and SST-2, respectively.
We use the best prompt (prompt # 1 in both tables) out of thethree according to the zero-shot test set prediction accuracy for generating soft labels to train theclassification model ( i.e. , knowledge distillation).
The classifier is trained with Kullback–Leibler
[Table]
[Table]
as the
[Table]
to approximate the soft labels generated by CTRL on the entiretraining set
[Table]