M ETA -W EIGHTED L ANGUAGE M ODEL T UNING FOR A UGMENTATION -E NHANCED F EW -S HOT L EARNING
Anonymous
authors
Paper under double-blind review
A BSTRACT
Recent studies have revealed the intriguing few-shot learning ability of pretrained language models (PLMs): They can quickly adapt to a new task when fine-tuned on a small amount of labeled data formulated as prompts, without requiring abundant task-specific annotations.
Despite their promising performance, most existing fewshot approaches that only learn from the small training set still underperform fully supervised training by nontrivial margins.
In this work, we study few-shot learning with PLMs from a different perspective: We first tune an autoregressive PLM on the few-shot samples and then use it as a generator to synthesize a large amount of novel training samples which augment the original training set.
To encourage the generator to produce label-discriminative samples, we train it via weighted maximum likelihood where the weight of each token is automatically adjusted based on a discriminative meta-learning objective.
A classification PLM can then be fine-tuned on both the few-shot and the synthetic samples with regularization for better generalization and stability.
Our approach FewGen achieves an overall better result across seven classification tasks of the GLUE benchmark than existing few-shot learning methods, improving no-augmentation methods by 5+ average points, and outperforming augmentation methods by 3+ average points 1 .
I NTRODUCTION
Recent research has demonstrated the appealing few-shot learning potential of pretrained language models (PLMs) (Brown et al., 2020;
Clark et al., 2020; Devlin et al., 2019;
He et al., 2021;
Liu et al., 2019; Meng et al., 2021) on natural language understanding (NLU) tasks (Wang et al., 2019; 2018):
Instead of relying on abundant task-specific annotations, PLMs can effectively leverage a small set of training samples to quickly learn a new task.
Such training data efficiency is usually achieved by formulating downstream tasks as prompts (Brown et al., 2020; Gao et al., 2021; Scao & Rush, 2021;
Schick & Schütze, 2021a;d) which allow the PLM to adapt its language modeling ability acquired through pretraining to new downstream tasks.
The success of prompt-based methods has stimulated numerous explorations along the line of effective few-shot learning with PLMs: The training samples converted to natural language prompts can be used to directly fine-tune PLMs (Gao et al., 2021; Schick & Schütze, 2021a) or as in-context demonstrations to facilitate better inference (Brown et al., 2020; Liu et al., 2022b).
More recent approaches aim to automate the design of prompts by gradient-based searching (Shin et al., 2020) or parameterizing prompts as continuous learnable embeddings (Lester et al., 2021;
Liu et al., 2021b;
Zhang et al., 2022; Zhong et al., 2021).
Other studies investigate and address specific issues in promptbased few-shot learning (Liu et al., 2022a;
Tam et al., 2021; Zhao et al., 2021).
While remarkable, the model performance still has a nontrivial gap from fully supervised models trained on massive labeled data.
Indeed, training deep models is inherently data demanding—model generalization usually benefits from more training samples (Baum & Haussler, 1988).
In this work, we study few-shot learning with PLMs from a different perspective: Instead of proposing new methods for fine-tuning on few-shot samples, we focus on the generation of quality training data based on few-shot samples and using these synthesized training samples to fine-tune the classification models.
Motivated by the strong text generation power of autoregressive PLMs (Brown et al., 2020;
Keskar et al., 2019; Raffel et al., 2019), previous data augmentation methods enlarge the training set by synthesizing new samples based on the few-shot samples.
They either fine-tune the generator on the training set with the standard maximum likelihood objective (Anaby-Tavor et al., 2020; Kumar et al., 2020) or use the training samples as demonstrations (Yoo et al., 2021).
However, these methods do not explicitly model the distinction across different labels and may struggle to generate accurate training samples pertaining to the desired labels for challenging NLU tasks.
In this paper, we study how to use few-shot samples to effectively tune PLMs to generate high quality label-discriminative training samples.
Our contributions are as follows: (1) We analyze the issues of using standard maximum likelihood for tuning the generator and propose a meta-weighted maximum likelihood objective for generator tuning by automatically learning token weights that emphasize label discriminativeness.
(2) We propose a simple and effective training procedure for fine-tuning classification PLMs on generated data by mitigating label noise.
(3) Under the same few-shot learning setting, our method FewGen outperforms existing methods by 3+ average points on seven classification tasks of the GLUE benchmark (Wang et al., 2018).
Ablation studies demonstrate the effectiveness of our proposed meta-weighted training objective and classifier fine-tuning method.
R ELATED W ORK
Few-Shot Learning with PLMs.
Few-shot learning has gained much attention recently due to its minimal resource assumption—Without requiring massive annotated data but only leveraging a few training samples ( e.g. , 16 per label), few-shot methods can be widely adopted in many practical scenarios where obtaining large-scale annotations is unaffordable.
Standard fine-tuning of PLMs for few-shot learning usually performs poorly because the limited training samples may not be sufficient for optimizing the parameters in the newly introduced classification head.
To reuse the language modeling ability of PLMs without introducing randomly initialized parameters, prompt-based approaches (Brown et al., 2020; Gao et al., 2021;
Hu et al., 2022; Logan IV et al., 2021; Min et al., 2022; Schick & Schütze, 2021a;b;d; Tam et al., 2021) formulate training samples as natural language prompt templates so that various downsteam tasks can be solved as a token prediction problem.
They enjoy improved training data efficiency over standard fine-tuning in lowdata regimes (Scao & Rush, 2021) and achieve remarkable few-shot learning performance.
Later developments in prompt-based methods replace the manual design of prompt templates with automatic search or learning (Cui et al., 2022; Hambardzumyan et al., 2021; Lester et al., 2021;
Liu et al., 2021b; Zhang et al., 2022; Zhong et al., 2021).
There are also studies focusing on specific issues in prompt-based methods such as densifying the supervision by revising the training objective (Liu et al., 2022a;
Tam et al., 2021) and calibrating the biased predictions of PLMs before fine-tuning (Zhao et al., 2021).
Instead of focusing on fine-tuning methods for few-shot learning, we study how to effectively generate abundant quality training samples by learning from the few-shot samples and use them to improve the generalization of the classification model.
Data Augmentation.
Data augmentation methods (Chen et al., 2020; Lee et al., 2021; Miyato et al., 2017; Xie et al., 2020) aim to create similar samples to the existing ones so that the enlarged training set can benefit model generalization.
Early approaches simply use manually designed rules ( e.g. , swapping or inserting tokens) for word-level alterations over the given samples to create new ones (Wei & Zou, 2019).
Later methods leverage the strong generation power of PLMs to synthesize novel samples from scratch.
Given a training set, the PLMs can be either fine-tuned on the labeled samples to learn label-conditioned generation probability (Kumar et al., 2020; Lee et al., 2021; Yang et al., 2020) or take the labeled data as demonstrations (Wang et al., 2021; Yoo et al., 2021) to generate similar samples pertaining to the same label.
In this work, we study how to effectively tune generators on few-shot training data for creating new data—standard fine-tuning of PLMs on a small set of training data is prone to overfitting, and the resulting model may struggle to generate accurate, diverse and novel training data.
We address this challenge by leveraging prefix-tuning and proposing a new meta-weighted training objective to emphasize label-discriminative tokens for generator tuning.
Controlled Text Generation.
Generating training samples for different labels can be viewed as a form of controlled text generation (Hu et al., 2017), whose goal is to generate textual contents of desired semantics, styles or attributes.
Such control can be realized through different stages of
PLM training and deployment: During pretraining, control codes (Keskar et al., 2019) can be used as
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
<
[Table]
[Figure]
our proposed meta-weighted maximum likelihood objective and then used to synthesize new training samples.
A classification PLM is finally trained on both the few-shot and the generated samples.
explicit guidance for training the model to generate domain/attribute-specific texts; fine-tuning PLMs with attribute-specific data can also grant high-level control ( e.g. , certain topics or sentiments (Ziegler et al., 2019)), fine-grained control ( e.g. , specific words or phrases (Chan et al., 2021)) or both (Khalifa et al., 2021); at inference time, control over desired attributes can also be enforced without updating the PLM parameters (Dathathri et al., 2020; Krause et al., 2021; Kumar et al., 2021; Liu et al., 2021a;
Pascual et al., 2021; Yang & Klein, 2021).
Recently, a few studies explore fine-tuning autoregressive
PLMs (Anaby-Tavor et al., 2020; Yang et al., 2020) with the standard language modeling objective on the training set or using label-specific prompts (Meng et al., 2022; Schick & Schütze, 2021c;
Wang et al., 2021; Ye et al., 2022) to steer text generation towards the desired label.
Meta-Learning for Sample Weighting.
The idea of weighting training samples in the loss calculation originates from the class imbalance (Wang et al., 2017) and noisy label (Hendrycks et al., 2018) learning scenarios—By assigning higher weights to the samples from minority classes or lower weights to the noisy samples, the learning process is less impacted by the imbalance/label noise issues.
Meta-learning (Andrychowicz et al., 2016; Finn et al., 2017; Franceschi et al., 2018; Wu et al., 2018) is one way to automatically learn the weight for each sample.
Specifically, a meta objective, usually defined as the loss on a clean unbiased validation set (Ren et al., 2018; Shu et al., 2019), can be used to learn the sample weights which become hyperparameters that control the optimization of model parameters.
Our work has a different motivation and formulation of the meta objective for token-wise weighted training: Not all tokens in a training sample are equally label-discriminative.
We thus design a meta objective to emphasize distinction across different labels (instead of using the validation loss as the meta objective) for learning the token weights.
M ETHOD 3. P RELIMINARIES
Overview.
We consider the strict few-shot learning setting (Perez et al., 2021):
The training set
D train =
{ ( x , y ) i } consists of K training samples per label where x = [ x 1 , x 2 , . . .
, x n ] is a text sequence with n tokens.
The development set
D dev is of the same size as D train .
There is no access to additional task-specific unlabeled data.
The number of training samples K is assumed to be very small ( e.g. , K = 16 ), making it challenging to train a classification model C ϕ that generalizes well to unseen data.
To mitigate such a training data scarcity issue, we propose to first train an autoregressive
PLM on D train , and then use it as a generator G θ to synthesize a large amount of novel samples
D gen = { ( ˜ x , ˜ y ) i } that augment the original training set.
Finally, a classification PLM C ϕ is fine-tuned on both
D train and D gen to perform the task.
An overview of our FewGen method is shown in Fig.
Text Generation with Autoregressive PLMs.
In standard fine-tuning for text generation, an autoregressive PLM G θ is trained via the maximum likelihood generation loss of each token in a sequence x conditioned on previous tokens:
[Equation]
where the token generation probability p θ ( · ) is usually parameterized using token embeddings e and hidden states h of a Transformer (Vaswani et al., 2017) model.
After training, G θ can be used to generate novel texts by iteratively sampling tokens from its generation probability distribution.
Prefix-Tuning.
Unlike fine-tuning which updates all model parameters θ of a PLM, prefixtuning (Li & Liang, 2021) freezes all pretrained Transformer parameters and only optimizes prefix vectors θ p that are prepended to each Transformer layer.
We use prefix-tuning for training G θ p on
D train because (1) it offers better effectiveness than fine-tuning for small datasets (Li & Liang, 2021) and (2) the generation models for different labels can share the same backbone Transformer parameters with only the prefix vectors being different, significantly reducing the memory requirement for multi-class classification tasks.
3. L ABEL -D ISCRIMINATIVE T EXT G ENERATOR T UNING WITH M ETA W EIGHTS
Motivation.
To model the conditional text generation probability p ( x | y l ) on different labels, a straightforward way is to parameterize a generation model G θ pl for each label y l via a set of prefix vectors θ p = { θ p l }| Ll
=
so that p ( x | y l ) = p θ pl ( x ) , and then tune θ p l on the training samples x with label y l :
[Equation]
[Figure]
the generator only
[Figure]
does not automatically decrease
L disc .
However, such an approach only optimizes the generative likelihood p ( x | y l ) without accounting for label discriminativeness p ( y l | x ) which is essential for generating unambiguous training samples to benefit the final classification task.
Challenging NLU tasks can have largely similar distributions across different labels, with very nuanced differences reflected by a few key tokens.
For example, a negative review text “ a movie where the ending feels like a cop-out ” may immediately become a positive one by just changing the last word “cop-out” to “revelation”.
Indeed, we find that such subtle distinctions over different labels may not be effectively captured by the generators if they are trained with the standard generation objective in Eq.
As shown in Fig.
L disc (defined in Eq.
(2)) can even increase during training—It is possible that the dominating patterns in the training samples are label-indiscriminate ( e.g. , a movie review dataset may frequently mention “the movie”), making the generators of different labels eventually converge to similar distributions, especially when there are limited training samples per label.
To promote the generation of label-discriminative texts, we encourage each token x j to be more likely generated under the corresponding label y l instead of other labels ( i.e. , maximize p θ pl ( x j | x <j ) and minimize p θ pl ′ ( x j | x <j ) for l ′ ̸ = l ) via a discriminative loss L disc :
[Equation]
Although one can directly combine L disc with L gen to train G θ p to enforce distinction across different labels, doing so will result in two undesirable consequences: (1) A hyperparameter needs to be introduced to balance the weights of the two losses, whose optimal value is likely to vary by task; and (2) the generation-irrelevant loss L disc will unavoidably interfere the language modeling process, making the resulting model prone to generating less fluent and coherent texts.
Weighted Maximum Likelihood Generator Tuning.
To preserve the generative learning of G θ p while emphasizing label-discriminative tokens, we assume each token is associated with a weight in the maximum likelihood loss.
Intuitively, when our goal is to generate distinctive texts across different labels as training samples, not all tokens should contribute equally to generator training.
For example, for sentiment classification tasks, one would expect “good/bad” to be more label-discriminative than “the movie”, and the former should be paid more attention to during training.
It is thus natural to generalize L gen in Eq.
(1) to L w-gen as follows by assuming a weight w j is given for each token.
[Equation]
Note that in L w-gen , w is assumed to be the hyperparameter under which θ p l is optimized.
When w j is the same for every token, Eq.
(3) will be equivalent to Eq.
While it is possible to manually design weighting rules for setting w to promote label-discriminative learning, they will likely necessitate task-specific knowledge and nontrivial tuning.
To facilitate the automatic learning of these weights w , we propose to parameterize them as learnable hyperparameters using the idea of meta-learning.
Meta Weight Learning Setup.
To automatically learn token weights using the idea of metalearning, we formulate a bi-level optimization problem, where the inner objective L w-gen optimizes the generator parameters θ p , and the outer objective L disc optimizes token weights w that are used as hyperparameters by the inner objective.
We parameterize token weights w via a weighting network g ω so that w j = w j ( ω ) .
Details about the implementation of g ω are in Appendix E. Overall, the learning objectives are as follows:
[Equation]
Under the above formulation, the token weights w j ( ω ) are automatically learned such that the resulting generator parameters θ ∗ p ( ω ) capture label-discriminative information ( i.e. , minimize L disc ).
Instead of solving the optimal ω ∗ and θ ∗ p via nested optimization loops, we use an online optimization strategy (Shu et al., 2019) for training efficiency.
It also guarantees convergence to the critical points of both L w-gen and L disc under mild conditions.
The initialization prompts can be found in Appendix C.
The overall training procedure is shown in Algorithm 1.
Analysis of Meta Weight Learning.
We analyze the gradient update of meta weights to study its effect in generator tuning.
The weighting network parameter ω is optimized via Eq.
(4), and its initialization to gradient is as follows (detailed derivation in Appendix A):
[Equation]
∂ L j
[Equation]
( θ p )
[Equation]
(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
( t
[Equation]
It can be seen that the gradient descent direction of ω is determined by a weighted sum of token weight gradient ascent direction ( i.e. , ∂w j ( ω ) ∂ ω ), where the weight d j characterizes the similarity between the gradient of the discriminative objective and the gradient of the generative objective on the j th token.
Therefore, the meta weights will be higher on those tokens where optimizing their generative objective is more beneficial for minimizing the discriminative objective.
3. C LASSIFIER F INE -T UNING
With the trained generator G θ p , we can synthesize novel training samples
D gen that augment D train for fine-tuning a PLM C ϕ for classification.
The major challenge to effectively leverage
D gen is that the label noise ( i.e. , some generated samples may not accurately pertain to the corresponding label) may deteriorate model performance if standard supervised learning is directly used.
We propose a simple noise-robust training procedure to improve the generalization and stability of training: First fine-tune C ϕ on D train with standard supervised training, and then continue fine-tuning it on D gen by applying temporal ensembling (Laine & Aila, 2017) as regularization.
Specifically, given a training sample (˜ x , ˜ y ) ∈ D gen , we minimize the following classification loss:
[Equation]
where p ϕ (˜ x ) is the model prediction on ˜ x ; λ is a regularization weight for temporal ensembling; and ¯ z is the accumulated moving-average model predictions.
We also use the ensembled prediction ¯ z to filter out noisy synthesized samples: We only include those samples for training where ¯ z strongly agrees with the label ˜
y ( i.e. , ¯ z ˜ y > δ where δ > 0 is a threshold parameter).
In Eq.
(5), the first classification term is the cross-entropy loss; the second regularization term corresponds to temporal ensembling, which requires the current model prediction to be close to its past accumulated predictions.
This not only neutralizes the fluctuation in model predictions for better training stability when label noise is present (Nguyen et al., 2020) but also helps prevent catastrophic forgetting (Kirkpatrick et al., 2017) of the information learned previously from the few-shot training set D train .
Please refer to
Appendix C for details about the temporal ensembling implementation.
The overall procedure of classifier fine-tuning is summarized in Algorithm 2.
E XPERIMENTAL S ETUP
Downstream Tasks and Metrics.
We conduct evaluation on all tasks of the GLUE benchmark (Wang et al., 2018)
(more details in Appendix B) except STS-B which is a regression task.
We
[Table] [Table]
[Table] [Table]
follow the same data split and evaluation protocol as Gao et al.
Both D train and D dev contain 16 samples per label and are sampled from the original training set with 5 different random seeds.
The original development sets are used for testing.
For all reported results, we include the average and standard deviation over the 5 different D train / D dev splits.
F1 score is used as the metric for QQP and MRPC, Matthews correlation for CoLA, and accuracy for the remaining tasks.
Models, Training Settings and Hyperparameters.
FewGen is a training data generation method and can be used with any fine-tuning method on any classification model.
We use moderatesized PLMs to ensure our results are reproducible on typical research hardware: CTRL ( 1 . 6 B parameters)
(Keskar et al., 2019) as the generator G θ and RoBERTa Large ( 356 M parameters) (Liu et al., 2019) as the classifier C ϕ .
We use prefix-tuning for training G θ and prompt-based fine-tuning for training C ϕ .
For simplicity, we use the most basic manual prompt version of LM-BFF (Gao et al., 2021).
The only exception is CoLA for which we use the standard fine-tuning since the input data might be out of the distribution of C ϕ (Gao et al., 2021).
The hyperparameter tuning is performed on
D dev .
More details are in Appendix C.
Compared Methods.
No-augmentation baselines include zero-shot prompting, standard finetuning, in-context learning, and the following strong few-shot learning methods: Four versions of
LM-BFF (Gao et al., 2021), P-Tuning (Liu et al., 2021b) and DART (Zhang et al., 2022).
We also compare FewGen with data augmentation methods for few-shot learning: MixText (Chen et al., 2020), using back translation systems to generate paraphrases (UDA-style (Xie et al., 2020) augmentation),
GPT3Mix (Yoo et al., 2021) and standard fine-tuning of generator on the few-shot samples with prompts.
All augmentation methods use LM-BFF (Man.) for fine-tuning the RoBERTa Large classifier.
More details about data augmentation baselines can be found in Appendix D.
E VALUATION 5. M AIN R ESULTS
We present the results of FewGen and baselines in Table 1.
FewGen achieves overall better performance across the GLUE tasks, on average 5+ points higher than the previous best few-shot method without augmentation, and 3+ points better than GPT3Mix 2 (Yoo et al., 2021) which uses a 100 times larger generator model ( 175 B) than FewGen.
The promising results confirm the effectiveness of our
[Table]
proposed FewGen method in generating quality training data and leveraging them in combination with the few-shot training set for fine-tuning the classification model
[Table]
Comparison with Back Translation.
Using back translation to paraphrase the few-shot samples does not improve the results, even with prompt-based fine-tuning to train the classifier – this is probably because it does not produce samples that are sufficiently different from the few-shot training set.
The success of UDA (Xie et al., 2020) is grounded in the augmentations from abundant unlabeled data that improve the classifier generalization.
However, under the strict few-shot learning setup, there is no access to additional task-specific unlabeled data (Gao et al., 2021), making it challenging for paraphrase-based methods to create sufficiently diverse training samples only based on the small few-shot set.
The new training samples produced by our FewGen method are not limited to the paraphrases of the few-shot samples, as the generator is trained via prefix-tuning to preserve the
PLM’s pretraining knowledge, based on which novel training samples can be synthesized.
Comparison with GPT3Mix.
The gigantic size of GPT3 makes it challenging for tuning on fewshot samples.
Therefore, GPT3Mix (Yoo et al., 2021) uses few-shot samples as demonstrations for creating the augmentations.
Such an approach suffers from two limitations: (1) Without any parameter update to the PLM, its learning ability is not fully leveraged to adapt to the few-shot training set.
The PLM can only use a small subset of the few-shot samples at a time for creating each augmentation, as the number of demonstrations received by the model is bounded by its maximum input sequence length.
This makes the quality of the created augmentations more sensitive to the randomly drawn training samples.
Our FewGen method, on the other hand, can use the entire few-shot set for tuning the PLM and achieves overall even better classification results with a much smaller PLM ( < 1% the size of the GPT3 model) which can be deployed much more easily in practice.
5. A BLATION S TUDIES
We further analyze the effectiveness of each important component in FewGen.
Specifically, we compare FewGen with the following ablations: (1) Using the standard L gen in Eq.
(1) instead of our proposed L w-gen in Eq.
(3) for generator tuning (w. L gen ); (2) using the directly combined
L gen and L disc for generator tuning (w. L gen + L disc ); (3) without applying temporal ensembling in Eq.
(5) ( − temporal ensemble); (4) directly fine-tuning the classification model on the combination of D gen and D train (w. fine-tune on D train ∪ D gen ) 3 .
As shown in Table 2, (1) & (2) using the standard maximum likelihood loss or the combination of generation and discrimination losses to tune the generator both yield lower-quality training data and lead to degraded classification performance; (3) not applying temporal ensembling for fine-tuning the classifier is more prone to label noise in the generated samples; (4) fine-tuning the classifier on the combination of D gen and D train significantly underperforms our two-step fine-tuning method.
To study the impact of the amount of generated training samples on the model performance, we plot the MNLI-m accuracy (mean and standard deviation) with different sizes of D gen in Fig.
Both the average model performance and stability improve with more generated samples.
5. A NALYSES OF L OSS F UNCTIONS FOR G ENERATOR T UNING
As shown in Table 2, the choice of generator loss has a significant impact on the synthesized data quality and thus the final model performance.
We conduct further analyses to compare the training processes of the generator under the following three loss functions and the resulting generated
[Figure]
[Figure]
samples: (1) L gen which is the standard language modeling loss; (2) L gen + L disc which directly adds the discriminative loss to generator training; and (3)
L w-gen which is our meta-weighted objective.
Fig.
4 shows the discriminative loss
L disc and the standard language modeling loss on the held-out development set throughout training.
Although using L gen + L disc helps reduce the discriminative loss, it comes at the cost of hindering language modeling—the generator loss on the development set is high.
Using our meta-weighted objective L w-gen for tuning the generator not only encourages discriminativeness but also mitigates overfitting, yielding the lowest validation set loss.
This is probably because the model receives contrastive information from other labels which facilitates more accurate modeling of the texts with the target label.
We present more quantitative analyses of different generator training objectives in Appendix G.
We visualize the token weights w automatically learned and used in L w-gen in Appendix F.
D ISCUSSIONS AND C ONCLUSIONS
Ethical Considerations.
Despite the impressive text generation and representation power of PLMs, they can also come with the risk (Bender et al., 2021; Bender & Koller, 2020; Brown et al., 2020) of generating disinformation (Pagnoni et al., 2021) or exacerbating biases (Prabhumoye et al., 2018).
Instead of improving upon PLM architectures or generation techniques, our work focuses on using existing PLMs to create training data for NLU tasks.
Therefore, our method can be combined with any bias reduction and correction strategies (Gehman et al., 2020; Ma et al., 2020) in practice to reduce the adverse effects of PLMs.
Limitations.
Compared to few-shot learning methods that directly train classification models on the small training set, FewGen requires tuning a generator PLM and using it to synthesize novel training samples, resulting in higher computation costs and longer running time.
Still, we believe that our method may bring more good than harm—when the small training data size becomes the performance bottleneck for NLU tasks, a simple yet costly solution is to obtain more human annotations.
Our method may replace or reduce the human efforts in such training data creation processes.
Conclusions.
In this work, we propose FewGen, which leverages few-shot training samples to tune a generator PLM for synthesizing novel training data.
The generated data can be then used in combination with few-shot samples to fine-tune a classification model for better generalization.
To emphasize label-discriminative information during generator tuning, we propose a weighted maximum likelihood objective where the token weights are automatically learned via a discriminative meta objective.
Since the generated samples may contain label noise, we propose a simple training procedure that first trains classifiers on the few-shot training set and then on the generated set by applying temporal ensembling for noise-robustness.
Across seven classification tasks from the GLUE benchmark, FewGen significantly outperforms existing approaches under the same few-shot learning setting.
The effectiveness of each important component in FewGen is validated via ablation studies.
Future work directions may include: Using larger PLMs as the generator and the classifier, jointly training both models with each other’s high-confident predictions, and developing systematic metrics for evaluating the quality of generated training samples.
A D ERIVATION OF M ETA W EIGHT G RADIENT U PDATE
We first write out the gradient update of ˆ θ ( t ) p (cid:0) ω ( t ) (cid:1) and ω ( t +1) according to Algorithm 1 as follows:
[Equation]
where α and β are step sizes
[Equation]
The gradient in Equation (7) is calculated as:
[Equation]
ω
[Equation]
( t )
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ˆ
[Equation]
p
[Equation]
ˆ
[Equation]
( t ) p  
[Equation]
n (cid:88) j =
[Equation] [Equation]
) ∂
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ⊤
[Equation]
( t )
[Equation]
( ω )
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12) ω
= ω ( t ) 

Plugging in Eq.
(6) =
[Equation]
n
[Equation]
 
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
( t )
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ⊤
[Equation]
( t )
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
d j ∂w j ( ω )
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12) ω
[Equation]
( t )  
Therefore,
[Equation]
∂ ω (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
t )
∝
[Equation]
(cid:88)
j = d
[Equation]
( ω )
[Equation]
ω (cid:12)(cid:12)(cid:12)(cid:12)
[Table]
B GLUE T ASKS
We provide the details of the seven classification tasks included in the GLUE benchmark.
MNLI: Multi-genre Natural Language Inference (Williams et al., 2018) requires predicting whether a given premise sentence entails, contradicts or neutral with respect to a given hypothesis sentence.
QQP: Quora Question Pairs (Shankar et al., 2017) requires judging whether a pair of questions asked are semantically equivalent.
QNLI:
Question Natural Language Inference requires predicting whether a given sentence contains the answer to a given question sentence.
SST-2: Stanford Sentiment Treebank (Socher et al., 2013) requires determining if a movie review has positive or negative sentiment.
CoLA: Corpus of Linguistic Acceptability (Warstadt et al., 2019) requires determining whether a given sentence is linguistically acceptable or not.
RTE: Recognizing Textual Entailment (Bentivogli et al., 2009; Dagan et al., 2005; Giampiccolo et al., 2007; Haim et al., 2006) requires predicting whether a given premise sentence entails a given hypothesis sentence or not.
MRPC: Microsoft Research Paraphrase Corpus (Dolan & Brockett, 2005) requires predicting whether two sentences are semantically equivalent or not.
C I MPLEMENTATION D ETAILS
[Table]
Details of Initialization Prompts Used for Generator Tuning on Different Tasks.
For generator tuning, we find it beneficial to initialize the prefix vectors with task-descriptive prompts, similar to the observations in Li & Liang (2021).
The prefix lengths ( i.e. , number of trained prefix token positions) are equal to the number of tokens in the prompts.
We present details about the prompts used for initializing the prefix vectors for different tasks in Table 3.
For sequence-pair tasks, an additional infix prompt is used between the two sequences, and we also tune the embeddings of the infix ( i.e. , prompt-tuning (Lester et al., 2021)) for generator training.
Details of Generator Tuning.
The meta-weighted generator tuning procedure (Algorithm 1) involves three forward and backward passes, and thus its time complexity is approximately 3 times of standard generator training without meta learning. However, since the few-shot training sets have a small amount of training data, the extra time cost is usually affordable. In practice, our generator tuning with meta weight learning takes 10 minutes to train on each task (the standard generator training time without meta-learning is 3 . 5 minutes). We use a fixed set of hyperparamters for all tasks without task-specific hyperparamter tuning: We set batch size to be 2 , the learning rate for optimizing θ p to be 5 e − 3 , the learning rate for optimizing ω to be 1 e − 2 , and training epoch to be 20 .
Details of Generating Training Data.
Following Meng et al.
(2022), for sequence-pair tasks (MNLI, QQP, QNLI, RTE and MRPC), we randomly sample the first sequence from the pretraining corpus ( e.g. , Wikipedia) and use greedy sampling for generating the second sequence.
For singlesequence tasks (SST-2 and CoLA), we use top- k sampling with temperature to generate training data from scratch where k = 10 .
For all tasks, we generate 5 , 000 samples per label.
For SST-2, we use one of the following tokens to start generation: “a”, “one”, “the”, “this”, “that”, “i”, “you”, “it”, “what”.
For CoLA, we use a random stop word to start generation.
Hyperparameters for Fine-Tuning Classifier PLMs.
For fine-tuning on the few-shot training samples D train , we search among the following hyperparameter ranges based on development set ( D dev ) model performance and pick the best performing model for futher fine-tuning on synthesized data: Learning rate in [1 e − 5 , 2 e − 5] and batch size in [2 , 4 , 8] .
The number of training steps is fixed to be 1000 .
We also find it beneficial to apply label smoothing (smoothing weight set to 0 . 15 ) for fine-tuning on the few-shot training set.
For fine-tuning on the synthesized training samples D gen , we use the following hyperparameters: 5 e − 6 as the learning rate; 16 as the batch size; label smoothing weight ϵ = 0 .
15 ; temporal ensemble momentum γ = 0 . 9 ; temporal ensemble loss weight λ = 20 ; training steps T = 6 , 000 .
Details of Temporal Ensembling for Fine-Tuning Classifier PLMs on Synthetic Data.
We update ensembled predictions ¯ z as follows where p ϕ is the current model prediction, γ is the momentum parameter, ˆ z is the accumulated model prediction before bias correction, ¯ z is the accumulated model prediction after bias correction, and t is the number of updates ¯ z has received:
[Equation]
The accumulated model prediction ˆ z has a zero initialization; the division (1 − γ t ) is for bias correction (Laine & Aila, 2017).
After each update of ˆ z , it will be compared to a threshold value δ ; each synthesized sample (˜ x , ˜ y ) will be included in training only if ¯ z ˜ y > δ .
We update the ensembled predictions ¯ z on all samples in D gen every 200 steps, and set the threshold value for sample filtering δ = 0 . 8 .
Computation Environment.
The experiments are conducted on NVIDIA A100 GPUs.
D D ATA A UGMENTATION B ASELINE D ETAILS
Details About MixText (Chen et al., 2020).
We use the TMix version of MixText to perform data interpolation on the few-shot labeled dataset (since there is no access to unlabeled task-specific data under the strict few-shot learning setting Gao et al.
We adapt the label mix-up operation to fit prompt-based fine-tuning by interpolating the label words instead of categorical labels; we observe that this results in better few-shot performance than the original TMix, probably analogous to why prompt-based fine-tuning outperforms standard fine-tuning for few-shot learning.
We train the classifier with supervised loss combined with consistency loss over the interpolated samples as in the original paper.
We follow the default hyperparameters in MixText.
Details About Back Translation.
We use two trained Marian (Junczys-Dowmunt et al., 2018) models to perform data augmentation via back translation.
We translate our labeled examples from
English to French, and then back to English.
As in UDA (Xie et al., 2020), we employ random sampling with a tunable temperature to generate a diverse set of derivative examples.
We generateexamples from each few-shot training example and let the synthesized samples share the same label with the original few-shot training sample.
After combining with the original examples, we fine-tune the classifier and observe performance.
Details About GPT3Mix (Yoo et al., 2021).
We use the 175 B GPT3 model for generating the augmentations.
For creating each augmentation, we randomly sample k = 4 (the optimal setting according to GPT3Mix) examples from the few-shot training set as demonstrations.
The prompts follow the suggested format proposed in the original paper (Yoo et al., 2021) and are shown in Table 4.
We create 5 , 000 augmented samples per label to make the resulting training set size equal to that of
FewGen.
After obtaining the augmented examples and their pseudo labels (the probability predictions over all labels by GPT3), we use them along with the real few-shot samples for fine-tuning the classifier, following the setting in GPT3Mix (Yoo et al., 2021).
Details About Standard Generator Fine-Tuning.
We fine-tune the same 1 .
6 B CTRL (Keskar et al., 2019) model as used in FewGen with the standard maximum likelihood objective.
Different from previous studies (Anaby-Tavor et al., 2020; Kumar et al., 2020) that prepend categorical labels to the training samples, we enhance the generator fine-tuning with label-descriptive prompts (shown in Table 3) used in FewGen.
We create 5 , 000 augmented samples per label to make the resulting training set size equal to that of FewGen.
E D ETAILS OF W EIGHTING N ETWORK I MPLEMENTATION
[Table]
The default architecture is a feedforward network (FFN) with one hidden layer.
We also explore adding a self-attention layer on top of the generator
PLM’s output hidden states (Self-attention).
We use the same two metrics with Table 6 to evaluate the resulting generators.
[Table]
Since the token weights w used in Eq.
(4) need to characterize the discriminativeness of each token, we use the value of discriminative objective at each token L j disc as the input to the weighting network, and we use softmax to normalize the weights:
[Equation]
Following Shu et al.
(2019), we instantiate g ω to be a feedforward network (FFN) with only one 100 -dimension hidden layer by default.
We explore an alternative instantiation that adds one self-attention layer on top of the generator PLM’s output hidden states.
The meta weights are finally obtained by projecting the outputs of the selfattention layer using another linear layer.
We evaluate the resulting generator quality via the same two metrics as in Appendix G. Table 5 shows that using more complicated architectures ( e.g. , adding another self-attention layer) does not result in a better generator compared to using a simple FFN for meta weight learning.
This is probably because the generator PLM’s output representations are sufficiently contextualized and contain the information necessary for learning the token weights, thus a simple FFN as the weighting network will be enough.
Using more complicated networks, on the other hand, will introduce more randomly initialized new parameters which may not be learned well using the limited amount of few-shot training data.
F V ISUALIZATION OF T OKEN W EIGHT L EARNING
To gain intuitive understanding of what tokens are assigned more weight during generator tuning, we visualize the learned weights in Fig.
The tokens with higher weights ( e.g. , “weak” in the first example and “hates” in the second example) are learned to be important tokens that decide the relation of the second sentence to the first sentence ( i.e. , the label of the training sample).
With such tokens emphasized during training, the generator is encouraged to capture label-discriminative information that facilitates the generation of unambiguous training samples. [Table] [Table] [Table]
[Figure] The generator is trained given the first sentence to generate the second. The tokens associated with higher weights are more label indicative [Table]
[Table]
G Q UANTITATIVE A NALYSES OF G ENERATOR T RAINING O BJECTIVES
Apart from the final classification model performance which indirectly reflects the synthetic data quality, we additionally conduct more direct quantitative analyses of different generator training objectives.
We use two metrics: (1) The accuracy of generated texts, which is judged by fullysupervised RoBERTa Large models fine-tuned on the original training sets of each task.
We choose to adopt such an automatic evaluation instead of human evaluation because it is efficient and reliable— fully-supervised
RoBERTa Large models have comparable or better accuracy than human baselines according to the GLUE benchmark 4 .
The generator’s perplexity on the test sets, which reflects how well the generator models the task distribution.
As shown in Table 6, using L w-gen for generator training consistently outperforms using L gen or L gen + L disc , both in generated text accuracy and in language modeling ability.
Comparing L w-gen with L gen , the meta weights automatically learned emphasize discriminative tokens in generator training and help the generator capture the subtle semantic differences across different labels, resulting in better language modeling quality and more distinctive generated data.
Comparing L w-gen with L gen + L disc , the generator training objective is not directly impacted by the discriminative objective, thus avoiding the gradient interference issue in multi-task learning (Standley et al., 2019)—the gradient for optimizing the generative probability p ( x | y l ) will be interfered by that optimizing the discriminative probability p
( y l | x ) if L gen + L disc is used.
Therefore, using L w-gen results in better language modeling quality and more fluent and coherent generation results.
We also showcase concrete generation results for the three labels of MNLI by models trained with the three different loss functions in Table 7.
The model trained with L gen produces fluent and coherent sentences, but the generated sentences do not accurately pertain to the desired label ( i [Table] e. , the “entailment” and “contradiction” generation results are in fact neutral with respect to the given sentence), lacking label discriminativeness.
When L gen + L disc is used, the generated samples of different labels are more distinctive, but also become less natural and coherent due to the model’s language modeling ability being hampered.
The generator tuned with L w-gen produces both coherent and label-discriminative samples which can serve as quality training data [Table]
[Table] so we [Table] on
[Table] next [Table] love
[Table] this [Table] ultimately , sarah ’s dedication to finding her husband seems more psychotic than romantic , and nothing in the movie makes a convincing case that one
[Table] ’s broken heart outweighs
[Table]
loss we witness
[Table] big [Table] bit
[Table]
about a chocolate eclair and then
[Table]
into it and finding
[Table] missing [Table] this picture
[Table] mostly [Table] of run-of-the-mill profanity sprinkled with a few remarks so geared toward engendering audience sympathy
that you might think he was running for office – or trying to win over a [Table]
just because a walk to remember is shrewd enough to activate girlish tear ducts does n’t mean it ’s good enough for our girls [Table] often lingers just as long on the irrelevant as on the engaging , which gradually turns what time is it there [Table]
this movie , a certain scene in particular , brought me uncomfortably close to losing my lunch [Table] but it would be better to wait for the video [Table]
H C ONCRETE G ENERATION R ESULTS
We present few-shot training samples ( D train ) of SST-2 in Table 8, and some concrete generation results (from D gen ) for all tasks in Tables 9, 10, 11, 12, 13,
[Table]
and
[Table]
As shown in Tables 8 and
[Table]
the synthetic samples are accurate and quite different from the given training samples to serve as effective augmentations.
For sequence-pair tasks, because we randomly sample the first sequence from the pretraining corpus and let the generator create the second sequence given certain labels
[Table]
the resulting generated samples will be certainly different from the given training samples.
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]