R EVISITING F EW - SAMPLE BERT F INE - TUNING
A BSTRACT
This paper is a study of ﬁne-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios.
We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of signiﬁcant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations.
We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process.
In light of these observations, we re-visit recently proposed methods to improve few-sample ﬁne-tuning with BERT and re-evaluate their effectiveness.
Generally, we observe the impact of these methods diminishes signiﬁcantly with our modiﬁed process.
I NTRODUCTION
Fine-tuning self-supervised pre-trained models has signiﬁcantly boosted state-of-the-art performance on natural language processing (NLP) tasks (Liu, 2019; Yang et al., 2019a;
Wadden et al., 2019; Zhu et al., 2020; Guu et al., 2020).
One of the most effective models for this process is BERT (Devlin et al., 2019).
However, despite signiﬁcant success, ﬁne-tuning remains unstable, especially when using the large variant of BERT (BERT Large ) on small datasets, where pre-training stands to provide the most signiﬁcant beneﬁt.
Identical learning processes with different random seeds often result in signiﬁcantly different and sometimes degenerate models following ﬁne-tuning, even though only a few, seemingly insigniﬁcant aspects of the learning process are impacted by the random seed (Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020).
1 As a result, practitioners resort to multiple random trials for model selection.
This increases model deployment costs and time, and makes scientiﬁc comparison challenging (Dodge et al., 2020).
This paper is a study of different aspects of the few-sample ﬁne-tuning optimization process.
Our goal is to better understand the impact of common choices with regard to the optimization algorithm, model initialization, and the number of ﬁne-tuning training iterations.
We identify suboptimalities in common community practices: the use of a non-standard optimizer introduces bias in the gradient estimation; the top layers of the pre-trained BERT model provide a bad initialization point for ﬁnetuning; and the use of a pre-determined , but commonly adopted number of training iterations hurts convergence.
We study these issues and their remedies through experiments on multiple common benchmarks, focusing on few-sample ﬁne-tuning scenarios.
Once these suboptimal practices are addressed, we observe that degenerate runs are eliminated and performance becomes much more stable.
This makes it unnecessary to execute numerous random restarts as proposed in Dodge et al.
Our experiments show the remedies we experiment with for each issue have overlapping effect.
For example, allocating more training iterations can eventually compensate for using the non-standard biased optimizer, even though the combination of a bias-corrected optimizer and re-initializing some of the pre-trained model parameters can reduce ﬁne-tuning computational costs.
This empirically highlights how different aspects of ﬁne-tuning inﬂuence the stability of the process, at times in a similar manner.
In the light of our observations, we re-evaluate several techniques (Phang et al., 2018; Lee et al., 2020; Howard & Ruder, 2018) that Fine-tuning instability is also receiving signiﬁcant practitioner attention. For example: https:/ issues and https: were recently proposed to increase few-sample ﬁne-tuning stability and show a signiﬁcant decrease in their impact.
Our work furthers the empirical understanding of the ﬁne-tuning process, and the optimization practices we outline identify impactful avenues for the development of future methods.
B ACKGROUND AND R ELATED W ORK BERT
The Bidirectional Encoder Representations from Transformers (BERT; Devlin et al., 2019) model is a Transformer encoder (Vaswani et al., 2017) trained on raw text using masked language modeling and next-sentence prediction objectives.
It generates an embedding vector contextualized through a stack of Transformer blocks for each input token.
BERT prepends a special [CLS] token to the input sentence or sentence pairs.
The embedding of this token is used as a summary token for the input for classiﬁcation tasks.
This embedding is computed with an additional fully-connected layer with a tanh non-linearity, commonly referred to as the pooler , to aggregate the information for the [CLS] embedding.
Fine-tuning The common approach for using the pre-trained BERT model is to replace the original output layer with a new task-speciﬁc layer and ﬁne-tune the complete model.
This includes learning the new output layer parameters and modifying all the original weights, including the weights of word embeddings, Transformer blocks, and the pooler.
For example, for sentence-level classiﬁcation, an added linear classiﬁer projects the [CLS] embedding to an unnormalized probability vector over the output classes.
This process introduces two sources of randomness: the weight initialization of the new output layer and the data order in the stochastic ﬁne-tuning optimization.
Existing work (Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020) shows that these seemingly benign factors can inﬂuence the results signiﬁcantly, especially on small datasets (i.e., < 10K examples).
Consequently, practitioners often conduct many random trials of ﬁne-tuning and pick the best model based on validation performance (Devlin et al., 2019).
Fine-tuning Instability The instability of the BERT ﬁne-tuning process has been known since its introduction (Devlin et al., 2019), and various methods have been proposed to address it.
Phang et al.
(2018) show that ﬁne-tuning the pre-trained model on a large intermediate task stabilizes later ﬁne-tuning on small datasets.
Lee et al.
(2020) introduce a new regularization method to constrain the ﬁne-tuned model to stay close to the pre-trained weights and show that it stabilizes ﬁne-tuning.
Dodge et al.
(2020) propose an early stopping method to efﬁciently ﬁlter out random seeds likely to lead to bad performance.
Concurrently to our work, Mosbach et al.
(2020) also show that BERTA DAM leads to instability during ﬁne-tuning.
Our experiments studying the effect of training longer are related to previous work studying this question in the context of training models from scratch (Popel & Bojar, 2018; Nakkiran et al., 2019).
BERT Representation Transferability BERT pre-trained representations have been widely studied using probing methods showing that the pre-trained features from intermediate layers are more transferable (Tenney et al., 2019b;a;
Liu et al., 2019a; Hewitt & Manning, 2019; Hewitt & Liang, 2019) or applicable (Zhang et al., 2020) to new tasks than features from later layers, which change more after ﬁne-tuning (Peters et al., 2019;
Merchant et al., 2020).
Our work is inspired by these ﬁndings, but focuses on studying how the pre-trained weights inﬂuence the ﬁne-tuning process.
Li et al.
(2020) propose to re-initialize the ﬁnal fully-connected layer of a ConvNet and show performance gain for image classiﬁcation.
2 Concurrent to our work, Tamkin et al.
(2020) adopt a similar methodology of weight re-initialization (Section 5) to study the transferability of BERT.
In contrast to our study, their work emphasizes pinpointing the layers that contribute the most in transfer learning, and the relation between probing performance and transferability.
E XPERIMENTAL M ETHODOLOGY
Data We follow the data setup of previous studies (Lee et al., 2020; Phang et al., 2018; Dodge et al., 2020) to study few-sample ﬁne-tuning using eight datasets from the GLUE benchmark (Wang et al., 2019b).
The datasets cover four tasks: natural language inference (RTE, QNLI, MNLI), paraphrase detection (MRPC, QQP), sentiment classiﬁcation (SST-2), and linguistic acceptability (CoLA).
Appendix A provides dataset statistics and a description of each dataset.
We primarily [Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
[Table] [Table]
focus on four datasets (RTE, MRPC, STS-B, CoLA) that have fewer than 10 k training samples, because BERT ﬁne-tuning on these datasets is known to be unstable (Devlin et al., 2019). We also complement our study by downsampling all eight datasets to 1k training examples following Phang et al.
While previous studies (Lee et al., 2020; Phang et al., 2018; Dodge et al., 2020) focus on the validation performance, we split held-out test sets for our study.
3 For RTE, MRPC, STS-B, and CoLA, we divide the original validation set in half, using one half for validation and the other for test.
For the other four larger datasets, we only study the downsampled versions, and split additional 1k samples from the training set as our validation data and test on the original validation set.
Experimental Setup Unless noted otherwise, we follow the hyperparameter setup of Lee et al.
We ﬁne-tune the uncased, 24-layer BERT Large model with batch size 32, dropout 0.1, and peak learning rate 2 × 10 − 5 for three epochs.
We clip the gradients to have a maximum norm of 1 .
We apply linear learning rate warm-up during the ﬁrst 10% of the updates followed by a linear decay.
We use mixed precision training using Apex 4 to speed up experiments.
We show that mixed precision training does not affect ﬁne-tuning performance in Appendix C. We evaluate ten times on the validation set during training and perform early stopping.
We ﬁne-tune with 20 random seeds to compare different settings.
O PTIMIZATION A LGORITHM : D EBIASING O MISSION IN BERTA DAM
The most commonly used optimizer for ﬁne-tuning BERT is BERTA DAM , a modiﬁed version of the A DAM ﬁrst-order stochastic optimization method.
It differs from the original A DAM algorithm (Kingma & Ba, 2014) in omitting a bias correction step.
This change was introduced by Devlin et al.
and subsequently made its way into common open source libraries, including the ofﬁcial implementation, 5 huggingface’s
Transformers (Wolf et al., 2019), 6 AllenNLP (Gardner et al., 2018),
GluonNLP (Guo et al., 2019), jiant (Wang et al., 2019c), MT-DNN (Liu et al., 2020), and FARM.
7 As a result, this non-standard implementation is widely used in both industry and research (Wang et al., 2019a;
Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020; Sun et al., 2019; Clark et al., 2020;
Lan et al., 2020; Houlsby et al., 2019; Stickland & Murray, 2019; Liu et al., 2019b).
We observe that the bias correction omission inﬂuences the learning rate, especially early in the ﬁne-tuning process, and is one of the primary reasons for instability in ﬁne-tuning BERT (Devlin et al., 2019;
Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020).
Algorithm 1 shows the A DAM algorithm, and highlights the omitted line in the non-standard
BERTA DAM implementation.
At each optimization step (lines 4–11), A DAM computes the exponen-
[Figure] [Figure]
[Figure]
tial moving average of the gradients ( m t ) and the squared gradients ( v t ), where β 1 , β 2 parameterize the averaging (lines 7–8). Because A DAM initializes m t and v t to 0 and sets exponential decay rates β 1 and β 2 close to 1 , the estimates of m t and v t are heavily biased towards 0 early during learning when t is small. Kingma & Ba (2014) computes the ratio between the biased and the unbiased estimates of m t and v t as (1 − β t 1 ) and (1 − β t 2 ) .
This ratio is independent of the training data.
The model parameters θ are updated in the direction of the averaged gradient m t divided by the square root of the second moment √ v t (line 11).
BERTA DAM omits the debiasing (lines 9–10), and directly uses the biased estimates in the parameters update.
Figure 1 shows the ratio ˆ m t √ ˆ v t between the update using the biased and the unbiased estimation as a function of training iterations.
The bias is relatively high early during learning, indicating overestimation.
It eventually converges to one, suggesting that when training for sufﬁcient iterations, the estimation bias will have negligible effect.
Therefore, the bias ratio term is most important early during learning to counteract the overestimation of m t and v t during early iterations.
In practice,
A DAM adaptively re-scales the learning rate by √ 1 − β t1 − β t 1 .
This correction is crucial for BERT ﬁnetuning on small datasets with fewer than 10k training samples because they are typically ﬁne-tuned with less than 1k iterations (Devlin et al., 2019).
The ﬁgure shows the number of training iterations for RTE, MRPC, STS-B, CoLA, and MNLI.
MNLI is the only one of this set with a large number of supervised training examples.
For small datasets, the bias ratio is signiﬁcantly higher than one for the entire ﬁne-tuning process, implying that these datasets suffer heavily from overestimation in the update magnitude.
In comparison, for MNLI, the majority of ﬁne-tuning occurs in the region where the bias ratio has converged to one.
This explains why ﬁne-tuning on MNLI is known to be relatively stable (Devlin et al., 2019).
We evaluate the importance of the debiasing step empirically by ﬁne-tuning BERT with both
BERTA DAM and the debiased A DAM 9 for 50 random seeds on RTE, MRPC, STS-B, and CoLA.
Figure 2 summarizes the performance distribution.
The bias correction signiﬁcantly reduces the performance variance across different random trials and the four datasets.
Without the bias correction we observe many degenerate runs, where ﬁne-tuned models fail to outperform the random baseline.
For example, on RTE, 48% of ﬁne-tuning runs have an accuracy less than 55%, which is close to random guessing.
Figure 3 further illustrates this difference by plotting the mean and the range of training loss during ﬁne-tuning across different random trials on RTE.
Figure 11 in Appendix F shows similar plots for MRPC, STS-B, and CoLA.
The biased BERTA DAM consistently leads to worse averaged training loss, and on all datasets to higher maximum training loss.
This indicates models trained with BERTAdam are underﬁtting and the root of instability lies in optimization.
[Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
We simulate a realistic setting of multiple random trials following Dodge et al. We use bootstrapping for the simulation:
given the 50 ﬁne-tuned models we trained, we sample models with replacement, perform model selection on the validation set, and record the test results; we repeat this process 1k times to estimate mean and variance.
Figure 4 shows the simulated test results as a function of the number of random trials.
Appendix E provides the same plots for validation performance.
Using the debiased A DAM we can reliably achieve good results using fewer random trials; the difference in expected performance is especially pronounced when we perform less than 10 trials.
Whereas the expected validation performance monotonically improves with more random trials (Dodge et al., 2020), the expected test performance deteriorates when we perform too many random trials because the model selection process potentially overﬁts the validation set.
Based on these observations, we recommend performing a moderate number of random trials (i.e., 5 or 10).
I NITIALIZATION : R E - INITIALIZING BERT P RE - TRAINED L AYERS
The initial values of network parameters have signiﬁcant impact on the process of training deep neural networks, and various methods exist for careful initialization (Glorot & Bengio, 2010;
He et al., 2015;
Zhang et al., 2019; Radford et al., 2019; Dauphin & Schoenholz, 2019).
During ﬁne-tuning, the BERT parameters take the role of the initialization point for the ﬁne-tuning optimization process, while also capturing the information transferred from pre-training.
The common approach for BERT ﬁne-tuning is to initialize all layers except one specialized output layer with the pre-trained weights.
We study the value of transferring all the layers in contrast to simply ignoring the information learned in some layers.
This is motivated by object recognition transfer learning results showing that lower pre-trained layers learn more general features while higher layers closer to the output specialize more to the pre-training tasks (Yosinski et al., 2014).
Existing methods using BERT show that using the complete network is not always the most effective choice, as we discuss in Section 2.
Our empirical results further conﬁrm this: we observe that transferring the top pre-trained layers slows down learning and hurts performance. [Figure] [Figure]
We test the transferability of the top layers using a simple ablation study.
Instead of using the pretrained weights for all layers, we re-initialize the pooler layers and the top L ∈ N BERT Transformer blocks using the original BERT initialization, N (0 , 0 .
We compare two settings: (a) standard ﬁne-tuning with BERT, and (b) Re-init ﬁne-tuning of BERT.
We evaluate Re-init by selecting
L ∈ { 1 , . .
. , 6 } based on mean validation performance.
All experiments use the debiased A DAM (Section 4) with 20 random seeds.
Re-init Impact on Performance Table 1 shows our results on all the datasets from Section 3.
We show results for the common setting of using 3 epochs, and also for longer training, which we discuss and study in Section 6.
Re-init consistently improves mean performance on all the datasets, showing that not all layers are beneﬁcial for transferring.
It usually also decreases the variance across all datasets.
Appendix F shows similar beneﬁts for pre-trained models other than BERT.
Sensitivity to Number of Layers Re-initialized Figure 5 shows the effect of the choice of L , the number of blocks we re-initialize, on RTE and MRPC.
Figure 13 in Appendix F shows similar plots for the rest of the datasets.
We observe more signiﬁcant improvement in the worst-case performance than the best performance, suggesting that Re-init is more robust to unfavorable random seed.
We already see improvements when only the pooler layer is re-initialized.
Re-initializing further layers helps more.
For larger L though, the performance plateaus and even decreases as re-initialize pre-trained layers with general important features.
The best L varies across datasets.
Effect on Convergence and Parameter Change Figure 6 shows the training loss for both the standard ﬁne-tuning and Re-init on RTE and MRPC.
Figure 13, Appendix F shows the training loss for all other datasets.
Re-init leads to faster convergence.
We study the weights of different
Transformer blocks.
For each block, we concatenate all parameters and record the L 2 distance between these parameters and their initialized values during ﬁne-tuning.
Figure 7 plots the Ldistance for four different transformer blocks as a function of training steps on RTE, and Figures 15– 18 in Appendix F show all transformer blocks on four datasets.
In general, Re-init decreases the Ldistance to initialization for top Transformer blocks (i.e., 18–24).
Re-initializing more layers leads to a larger reduction, indicating that Re-init decreases the ﬁne-tuning workload.
The effect of Re-init is not local; even re-initializing only the topmost Transformer block can affect the whole network.
While setting L = 1 or L = 3 continues to beneﬁt the bottom Transformer blocks, re-initializing too many layers (e.g., L = 10 ) can increase the L 2 distance in the bottom Transformer blocks, suggesting a tradeoff between the bottom and the top Transformer blocks.
Collectively, these results suggest that Re-init ﬁnds a better initialization for ﬁne-tuning and the top L layers of BERT are potentially overspecialized to the pre-training objective.
T RAINING I TERATIONS : F INE - TUNING BERT FOR L ONGER
BERT is typically ﬁne-tuned with a slanted triangular learning rate, which applies linear warm-up to the learning rate followed by a linear decay.
This learning schedule warrants deciding the number of training iterations upfront.
Devlin et al.
(2019) recommend ﬁne-tuning GLUE datasets for three epochs.
This recommendation has been adopted broadly for ﬁne-tuning (Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020).
We study the impact of this choice, and observe that this one-size-ﬁts-all [Figure] [Figure] three-epochs practice for BERT ﬁne-tuning is sub-optimal.
Fine-tuning BERT longer can improve both training stability and model performance.
Experimental setup We study the effect of increasing the number of ﬁne-tuning iterations for the datasets in Section 3.
For the 1k downsampled datasets, where three epochs correspond to 96 steps, we tune the number of iterations in { 200 , 400 , 800 , 1600 , 3200 } .
For the four small datasets, we tune the number of iterations in the same range but skip values smaller than the number of iterations used in three epochs.
We evaluate our models ten times on the validation set during ﬁne-tuning.
This number is identical to the experiments in Sections 4–5, and controls for the set of models to choose from.
We tune with eight different random seeds and select the best set of hyperparameters based on the mean validation performance to save experimental costs.
After the hyperparameter search, we ﬁne-tune with the best hyperparameters for 20 seeds and report the test performance.
Results Table 1 shows the result under the Longer column.
Training longer can improve over the three-epochs setup most of the time, in terms of both performance and stability.
This is more pronounced on the 1k downsampled datasets.
We also ﬁnd that training longer reduces the gap between standard ﬁne-tuning and Re-init, indicating that training for more iterations can help these models recover from bad initializations.
However, on datasets such as MRPC and MNLI, Re-init still improves the ﬁnal performance even with training longer.
We show the validation results on the four downsampled datasets with different number of training iterations in Figure 8.
We provide a similar plot in Figure 14, Appendix G for the other downsampled datasets.
We observe that different tasks generally require different number of training iterations and it is difﬁcult to identify a one-size-ﬁts-all solution.
Therefore, we recommend practitioners to tune the number of training iterations on their datasets when they discover instability in ﬁne-tuning.
We also observe that on most of the datasets,
Re-init requires fewer iterations to achieve the best performance, corroborating that Re-init provides a better initialization for ﬁne-tuning.
R EVISITING E XISTING M ETHODS FOR F EW - SAMPLE BERT F INE - TUNING
Instability in BERT ﬁne-tuning, especially in few-sample settings, is receiving increasing attention recently (Devlin et al., 2019;
Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020).
We revisit these methods given our analysis of the ﬁne-tuning process, focusing on the impact of using the debiased A DAM instead of BERTA DAM
(Section 4).
Generally, we ﬁnd that when these methods are re-evaluated with the unbiased A DAM they are less effective with respect to the improvement in ﬁne-tuning stability and performance. [Table]
7. O VERVIEW
Pre-trained Weight Decay Weight decay (WD) is a common regularization technique (Krogh &
Hertz, 1992).
At each optimization iteration, λ w is subtracted from the model parameters, where λ is a hyperparameter for the regularization strength and w is the model parameters.
Pre-trained weight decay adapts this method for ﬁne-tuning pre-trained models (Chelba & Acero, 2004; Daumé III, 2007) by subtracting λ ( w − ˆ w ) from the objective, where ˆ w is the pre-trained parameters.
Lee et al.
(2020) empirically show that pre-trained weight decay works better than conventional weight decay in BERT ﬁne-tuning and can stabilize ﬁne-tuning.
Mixout Mixout (Lee et al., 2020) is a stochastic regularization technique motivated by Dropout (Srivastava et al., 2014) and DropConnect (Wan et al., 2013).
At each training iteration, each model parameter is replaced with its pre-trained value with probability p .
The goal is to prevent catastrophic forgetting, and (Lee et al., 2020) proves it constrains the ﬁne-tuned model from deviating too much from the pre-trained initialization.
Layer-wise Learning Rate Decay (LLRD) LLRD (Howard & Ruder, 2018) is a method that applies higher learning rates for top layers and lower learning rates for bottom layers.
This is accomplished by setting the learning rate of the top layer and using a multiplicative decay rate to decrease the learning rate layer-by-layer from top to bottom.
The goal is to modify the lower layers that encode more general information less than the top layers that are more speciﬁc to the pre-training task.
This method is adopted in ﬁne-tuning several recent pre-trained models, including XLNet (Yang et al., 2019b) and ELECTRA (Clark et al., 2020).
Transferring via an Intermediate Task Phang et al.
(2018) propose to conduct supplementary ﬁne-tuning on a larger, intermediate task before ﬁne-tuning on few-sample datasets.
They show that this approach can reduce variance across different random trials and improve model performance.
Their results show that transferring models ﬁne-tuned on MNLI (Williams et al., 2018) can lead to signiﬁcant improvement on several downstream tasks including RTE, MRPC, and
STS-B.
In contrast to the other methods, this approach requires large amount of additional annotated data.
7. E XPERIMENTS
We evaluate all methods on RTE, MRPC, STS-B, and CoLA.
We ﬁne-tune a BERT Large model using the A DAM optimizer with debiasing for three epochs, the default number of epochs used with each of the methods.
For intermediate task ﬁne-tuning, we ﬁne-tune a BERT Large model on MNLI and then ﬁne-tune for our evaluation.
For other methods, we perform hyperparameter search with a similar size search space for each method, as described in Appendix H. We do model selection using the average validation performance across 20 random seeds.
We additionally report results for standard ﬁne-tuning with longer training time (Section 6), weight decay, and Re-init (Section 5).
Table 2 provides our results.
Compared to published results (Phang et al., 2018;
Lee et al., 2020), our test performance for Int.
Task (transferring via an intermediate task), Mixout, Pre-trained
WD, and WD are generally higher when using the A DAM with debiasing.
However, we observe less pronounced beneﬁts for all surveyed methods compared to results originally reported.
At times, these methods do not outperform the standard baselines or simply training longer.
Using additional annotated data for intermediate task training continues to be effective, leading to consistent improvement over the average performance across all datasets.
LLRD and Mixout show less consistent performance impact.
We observe no noticeable improvement using pre-trained weight decay and conventional weight decay in improving or stabilizing BERT ﬁne-tuning in our experiments, contrary to existing work (Lee et al., 2020).
This indicates that these methods potentially ease the optimization difﬁculty brought by the debiasing omission in BERTA DAM , and when we add the debiasing, the positive effects are reduced.
C ONCLUSION
We have demonstrated that optimization plays a vital role in the few-sample BERT ﬁne-tuning.
First, we show that the debiasing omission in BERTA DAM is the main cause of degenerate models on small datasets commonly observed in previous work (Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020).
Second, we observe the top layers of the pre-trained BERT provide a detrimental initialization for ﬁne-tuning and delay learning.
Simply re-initializing these layers not only speeds up learning but also leads to better model performance.
Third, we demonstrate that the common one-size-ﬁts-all three-epochs practice for BERT ﬁne-tuning is sub-optimal and allocating more training time can stabilize ﬁne-tuning.
Finally, we revisit several methods proposed for stabilizing BERT ﬁne-tuning and observe that their positive effects are reduced with the debiased A DAM .
In the future, we plan to extend our study to different pre-training objectives and model architectures, and study how model parameters evolve during ﬁne-tuning. A CKNOWLEDGMENTS
We thank Cheolhyoung Lee for his help in reproducing previous work.
We thank Lili Yu, Ethan R.
Elenberg, Varsha Kishore, and Rishi Bommasani for their insightful comments, and Hugging Face for the Transformers project, which enabled our work.
A D ATASETS
Table 3 summarizes dataset statistics and describes our validation/test splits.
We also provide a brief introduction for each datasets:
RTE Recognizing Textual Entailment (Bentivogli et al., 2009) is a binary entailment classiﬁcation task.
We use the GLUE version.
MRPC Microsoft Research Paraphrase Corpus (Dolan & Brockett, 2005) is binary classiﬁcation task.
Given a pair of sentences, a model has to predict whether they are paraphrases of each other.
We use the GLUE version.
STS-B Semantic Textual Similarity Benchmark (Cer et al., 2017) is a regression tasks for estimating sentence similarity between a pair of sentences.
We use the GLUE version.
CoLA Corpus of Linguistic Acceptability (Warstadt et al., 2019) is a binary classiﬁcation task for verifying whether a sequence of words is a grammatically correct English sentence.
Matthews correlation coefﬁcient (Matthews, 1975) is used to evaluate the performance.
We use the GLUE version.
MNLI Multi-Genre Natural Language Inference Corpus (Williams et al., 2018) is a textual entailment dataset, where a model is asked to predict whether the premise entails the hypothesis, predicts the hypothesis, or neither.
We use the GLUE version.
QQP Quora Question Pairs (Iyer et al., 2017) is a binary classiﬁcation task to determine whether two questions are semantically equivalent (i.e., paraphrase each other).
We use the GLUE version.
SST-
The binary version of the Stanford Sentiment Treebank (Socher et al., 2013) is a binary classiﬁcation task for whether a sentence has positive or negative sentiment.
We use the GLUE version.
B I SOLATING THE I MPACT OF D IFFERENT S OURCES OF R ANDOMNESS
The randomness in BERT ﬁne-tuning comes from three sources: (a) weight initialization, (b) data order, and (c) Dropout regularization (Srivastava et al., 2014).
We control the randomness using two separate random number generators: one for weight initialization and the other for both data order and Dropout (both of them affect the stochastic loss at each iteration).
We ﬁne-tune BERT on RTE for three epochs using A DAM with 10 seeds for both random number generators.
We compare the standard setup with Re-init 5, where L = 5 .
This experiment is similar to Dodge et al.
but we use A DAM with debiasing instead of BERTA DAM and control for the randomness in Dropout as well.
When ﬁxing a random seed for weight initialization, Re-init 5 shares the same initialized classiﬁer weights with the standard baseline.
Figure 9 shows the validation accuracy of each individual run as well as the minimum, average, and maximum scores when ﬁxing one of the random seeds.
Figuresummarizes the standard deviations when one of the random seeds is controlled.
We observe several trends.
Re-init 5 usually improves the performance regardless of the weight initialization or data order and Dropout.
Second, Re-init 5 still reduces the instability when one of the sources of randomness is controlled.
Third, the standard deviation of ﬁxing the weight initialization roughly matches the one of controlled data order and Dropout, which aligns with the observation of Dodge et al.
[Table]
[Figure]
[Figure]
o r
I n i t i a li z a t i o n
[Figure]
We show the standard deviation of ﬁxing either the initialization or data order and Dropout.
Re-initconsistently reduces the instability regardless of the sources of the randomness.
C M IXED P RECISION T RAINING
Mixed precision training can accelerate model training while preserving performance by replacing some 32-bit ﬂoating-point computation with 16-bit ﬂoating-point computation.
We use mixed precision training in all our experiments using huggingface’s
Transformers (Wolf et al., 2019).
Transformers uses O1-level optimized mixed precision training implemented with the Apex library.
We evaluate if this mixed precision implementation inﬂuences our results.
We ﬁne-tune BERT withrandom trials on RTE, MRPC, STS-B, and CoLA.
We use two-tailed t -test to test if the distributions of the two methods are statistically different.
Table 4 shows the mean and standard deviation of the
[Table] [Table]
test performance.
The performance of mixed precision matches the single precision counterpart, and there is no statistically signiﬁcant difference.
D B IAS - CORRECTION ON MNLI
The focus of this paper is few-sample learning.
However, we also experiment with the full MNLI dataset.
Table 5 shows that average accuracy over three random runs.
The results conﬁrm that there is no signiﬁcant difference in using bias correction or not on such a large dataset.
While our recommended practices do not improve training on large datasets, this result shows there is no disadvantage to ﬁne-tune such models with the same procedure as we propose for few-sample training.
E S UPPLEMENTARY M ATERIAL FOR S ECTION
Effect of A DAM with Debiasing on Convergence.
Figure 11 shows the training loss as a function of the number of training iterations.
Using bias correction effectively speeds up convergence and reduces the range of the training loss, which is consistent with our observation in Figure 3.
Effect of A DAM with Debiasing on the Expected Validation Performance.
Figure 12 shows the expected validation performance as a function of the number of random trials.
Comparing to Figure 4, we observe several trends.
First, using A DAM with debiasing consistently leads to faster convergence and improved validation performance, which is similar to our observation about the test performance.
Second, we observe that the expected validation performance monotonically increases with the number of random trials, contrary to our observation about the test performance.
This suggests that using too many random trials may overﬁt to the validation set and hurt generalization performance.
F S UPPLEMENTARY M ATERIAL FOR S ECTION
Effect of L on Re-init Figure 13 shows the effect of Re-init in ﬁne-tuning on the eight downsampled datasets.
We observe similar trends in Figure 13 and Figure 5.
Re-init’s improvement is more pronounced in the wort-case performance across different random trials.
Second, the best value of L is different for each dataset.
Effect of Re-init on Model Parameters We use the same setup as in Figure 7 to plot the change in the weights of different Transformer blocks during ﬁne-tuning on RTE, MRPC, STS-B, and CoLA in Figures 15–18.
Effect of Re-init on Other Models We study more recent pre-trained contexual embedding models beyond BERT Large .
We investigate whether Re-init provides better ﬁne-tuning initialization in
XLNet Large Yang et al.
(2019b), RoBERTa Large Liu et al.
(2019c)
, BART Large Lewis et al.
(2019), and
ELECTRA Large Clark et al.
XLNet is an autoregressive language model trained by learning
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Table]
[Figure]
all permutations of natural language sentences.
RoBERTa is similar to BERT in terms of model architecture but is only pre-trained on the mask language modeling task only, but for longer and on
[Table] [Table]
more data.
BART is a sequence-to-sequence model trained as a denoising autoencoder.
ELECTRA is a BERT-like model trained to distinguish tokens generated by masked language model from tokens drawn from the natural distribution.
Together, they represent a diverse range of modeling choices in pre-training, including different model architectures, objectives, data, and training strategies.
We use A DAM with debiasing to ﬁne-tune these models on RTE, MRPC, STS-B, and COLA, using the hyperparameters that are either described in the paper or in the ofﬁcial repository of each model.
Table 6 summarizes the hyper-parameters of each model for each dataset.
We use the huggingface’s
Transformers library (Wolf et al., 2019).
The experimental setup is kept the same as our other experiments.
Table 7 displays the average test performance on these datasets.
We observe that several models suffer from high instability on these datasets and in most cases, Re-init can reduce the performance variance.
We observe that for some models, like XLNet Large or RoBERTa Large , Re-init can improve the average performance and reduce variance.
However, the behavior of Re-init varies signiﬁcantly across different models and Re-init have less signiﬁcant improvement for ELECTRA Large and BART Large .
Further study of the entire model family requires signiﬁcant computational resources, and we leave it as an important direction for future work.
G S UPPLEMENTARY M ATERIAL FOR S ECTION
Figure 14 plots the validation performance as a function of the number of training iteration, using the same setting as in Figure 8.
Similar to our observations in Figure 8, we ﬁnd that training longer generally improves ﬁne-tuning performance and reduces the gap between standard ﬁne-tuning and
Re-init.
On MNLI, Re-init still outperforms standard ﬁne-tuning.
[Figure]
H E XPERIMENTAL D ETAILS IN S ECTION
The hyperparameter search space allocated for each method in our experiments is:
Layerwise Learning Rate Decay (LLRD)
We grid search the initial learning rate in { 2 × 10 − 5 , 5 × 10 − 5 , 1 × 10 − 4 } and the layerwise decay rate in { 0 . 9 , 0 . 95 } .
Mixout We tune the mixout probability p ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 .
Weight decay toward the pre-trained weight We tune the regularization strength λ ∈ { 10 − 3 , 10 − 2 , 10 − 1 , 10 0 } .
Weight decay We tune the regularization strength λ ∈ { 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 } .
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
I
[Figure]
iti
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Figure]
[Table]
[Table] [Table]
[Figure]