M IXOUT : E FFECTIVE R EGULARIZATION TO F INETUNE L ARGE - SCALE P RETRAINED L ANGUAGE M ODELS
A BSTRACT
In natural language processing, it has been observed recently that generalization could be greatly improved by ﬁnetuning a large-scale language model pretrained on a large unlabeled corpus.
Despite its recent success and wide adoption, ﬁnetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available.
In this paper, we introduce a new regularization technique, to which we refer as “mixout”, motivated by dropout.
Mixout stochastically mixes the parameters of two models.
We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory.
We empirically evaluate the proposed mixout and its variants on ﬁnetuning a pretrained language model on downstream tasks.
More speciﬁcally, we demonstrate that the stability of ﬁnetuning and the average accuracy greatly increase when we use the proposed approach to regularize ﬁnetuning of BERT on downstream tasks in GLUE.
I NTRODUCTION
Transfer learning has been widely used for the tasks in natural language processing (NLP)
(Collobert et al., 2011; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Phang et al., 2018).
In particular, Devlin et al.
(2018) recently demonstrated the effectiveness of ﬁnetuning a large-scale language model pretrained on a large, unannotated corpus on a wide range of NLP tasks including question answering and language inference.
They have designed two variants of models, BERT LARGE (340M parameters) and BERT BASE (110M parameters).
Although BERT LARGE outperforms BERT BASE generally, it was observed that ﬁnetuning sometimes fails when a target dataset has fewer than 10, training instances (Devlin et al., 2018; Phang et al., 2018).
When ﬁnetuning a big, pretrained language model, dropout (Srivastava et al., 2014) has been used as a regularization technique to prevent co-adaptation of neurons (Vaswani et al., 2017; Devlin et al., 2018;
Yang et al., 2019).
We provide a theoretical understanding of dropout and its variants, such as Gaussian dropout (Wang & Manning, 2013), variational dropout (Kingma et al., 2015), and dropconnect (Wan et al., 2013), as an adaptive L 2 -penalty toward the origin (all zero parameters 0 ) and generalize dropout by considering a target model parameter u (instead of the origin), to which we refer as mixout ( u ) .
We illustrate mixout ( u ) in Figure 1.
To be speciﬁc, mixout ( u ) replaces all outgoing parameters from a randomly selected neuron to the corresponding parameters of u . mixout ( u ) avoids optimization from diverging away from u through an adaptive L 2 -penalty toward u .
Unlike mixout ( u ) , dropout encourages a move toward the origin which deviates away from u since dropout is equivalent to mixout ( 0 ) .
We conduct experiments empirically validating the effectiveness of the proposed mixout ( w pre ) where w pre denotes a pretrained model parameter.
To validate our theoretical ﬁndings, we train a fully connected network on EMNIST Digits (Cohen et al., 2017) and ﬁnetune it on MNIST.
We observe that a ﬁnetuning solution of mixout ( w pre ) deviates less from w pre in the L 2 -sense than [Figure] that of dropout.
In the main experiment, we ﬁnetune BERT LARGE with mixout ( w pre ) on small training sets of GLUE (Wang et al., 2018).
We observe that mixout ( w pre ) reduces the number of unusable models that fail with the chance-level accuracy and increases the average development (dev) scores for all tasks.
In the ablation studies, we perform the following three experiments for ﬁnetuning BERT LARGE with mixout ( w pre ) : (i) the effect of mixout ( w pre ) on a sufﬁcient number of training examples, (ii) the effect of a regularization technique for an additional output layer which is not pretrained, and (iii) the effect of probability of mixout ( w pre ) compared to dropout.
From these ablation studies, we observe that three characteristics of mixout ( w pre ) : (i) ﬁnetuning with mixout ( w pre ) does not harm model performance even with a sufﬁcient number of training examples; (ii) It is beneﬁcial to use a variant of mixout as a regularization technique for the additional output layer; (iii)
The proposed mixout ( w pre ) is helpful to the average dev score and to the ﬁnetuning stability in a wider range of its hyperparameter p than dropout.
1. R ELATED W ORK
For large-scale pretrained language models (Vaswani et al., 2017; Devlin et al., 2018; Yang et al., 2019), dropout has been used as one of several regularization techniques.
The theoretical analysis for dropout as an L 2 -regularizer toward 0 was explored by Wan et al.
(2013) where 0 is the origin.
They provided a sharp characterization of dropout for a simpliﬁed setting (generalized linear model).
Mianjy & Arora (2019) gave a formal and complete characterization of dropout in deep linear networks with squared loss as a nuclear norm regularization toward 0 .
However, neither Wan et al.
nor Mianjy & Arora (2019) gives theoretical analysis for the extension of dropout which uses a point other than 0 .
Wiese et al.
(2017), Kirkpatrick et al.
(2017), and Schwarz et al.
(2018) used L 2 -penalty toward a pretrained model parameter to improve performance.
They focused on preventing catastrophic forgetting to enable their models to learn multiple tasks sequentially.
They however do not discuss nor demonstrate the effect of L 2 -penalty toward the pretrained model parameter on the stability of ﬁnetuning.
Barone et al.
(2017) introduced tuneout, which is a special case of mixout.
They applied various regularization techniques including dropout, tuneout, and L 2 -penalty toward a pretrained model parameter to ﬁnetune neural machine translation.
They however do not demonstrate empirical signiﬁcance of tuneout compared to other regularization techniques nor its theoretical justiﬁcation.
P RELIMINARIES AND N OTATIONS
Norms and Loss Functions
Unless explicitly stated, a norm (cid:107) · (cid:107) refers to L 2 -norm.
A loss function of a neural network is written as L ( w ) = 1 n (cid:80) ni
=
L i ( w ) , where w is a trainable model parameter.
L i is “a per-example loss function” computed on the i -th data point.
Strong Convexity A differentiable function f is strongly convex if there exists m >
such that
[Equation]
for all x and y .
Weight Decay We refer as “ wdecay( u , λ ) ” to minimizing
[Equation]
instead of the original loss function L ( w ) where λ is a regularization coefﬁcient.
Usual weight decay of λ is equivalent to wdecay( 0 , λ ) .
Probability for Dropout and Dropconnect Dropout (Srivastava et al., 2014) is a regularization technique selecting a neuron to drop with a probability of p .
Dropconnect (Wan et al., 2013) chooses a parameter to drop with a probability of p .
To emphasize their hyperparameter p , we write dropout and dropconnect with a drop probability of p as “ dropout( p ) ” and “ dropconnect( p ) ”, respectively.
dropout( p ) is a special case of dropconnect( p ) if we simultaneously drop the parameters outgoing from each dropped neuron.
Inverted Dropout and Dropconnect
In the case of dropout( p ) , a neuron is retained with a probability of 1 − p during training.
If we denote the weight parameter of that neuron as w during training, then we use (1 − p ) w for that weight parameter at test time (Srivastava et al., 2014).
This ensures that the expected output of a neuron is the same as the actual output at test time.
In this paper, dropout( p ) refers to inverted dropout( p ) which uses w / (1 − p ) instead of w during training.
By doing so, we do not need to compute the output separately at test time.
Similarly, dropconnect( p ) refers to inverted dropconnect( p ) .
A NALYSIS OF D ROPOUT AND I TS G ENERALIZATION
We start our theoretical analysis by investigating dropconnect which is a general form of dropout and then apply the result derived from dropconnect to dropout.
The iterative SGD equation for dropconnect( p ) with a learning rate of η is
[Equation]
where B ( t ) =
diag( B ( t ) 1 , B ( t ) 2 , · · · , B ( t ) d ) and B ( t ) i ’s are mutually independent Bernoulli(1 − p ) random variables with a drop probability of p for all i and t .
We regard equation 2 as ﬁnding a solution to the minimization problem below:
[Equation]
where B = diag( B 1 , B 2 , · · · , B d ) and B
i ’s are mutually independent Bernoulli(1 − p ) random variables with a drop probability of p for all i .
Gaussian dropout (Wang & Manning, 2013) and variational dropout (Kingma et al., 2015) use other random masks to improve dropout rather than Bernoulli random masks.
To explain these variants of dropout as well, we set a random mask matrix M = diag( M 1 , M 2 , · · · , M d ) to satisfy E M i = µ and Var( M i ) =
σ 2 for all i .
Now we deﬁne a random mixture function with respect to w from u and M as
[Equation]
and a minimization problem with “ mixconnect ( u , µ, σ
[Equation]
” as
[Equation]
We can view dropconnect( p ) equation 3 as a special case of equation 5 where u = 0 and M = B .
We investigate how mixconnect ( u , µ, σ 2 ) differs from the vanilla minimization problem
[Equation]
If the loss function L is strongly convex, we can derive a lower bound of E L (cid:0) Φ( w ; u , M ) (cid:1) as in Theorem 1:
Theorem 1.
Assume that the loss function L is strongly convex.
Suppose that a random mixture function with respect to w from u and M is given by Φ( w ; u , M ) in equation 4 where M is diag( M 1 , M 2 , · · · , M d ) satisfying E M i = µ and Var( M i )
=
σ 2 for all i .
Then, there exists m > 0 such that
[Equation]
for all w (Proof in Supplement A).
Theorem 1 shows that minimizing the l.h.s.
of equation 7 minimizes the r.h.s.
of equation 7 when the r.h.s.
is a sharp lower limit of the l.h.s.
The strong convexity of L means that L is bounded from below by a quadratic function, and the inequality of equation 7 comes from the strong convexity.
Hence, the equality holds if L is quadratic, and mixconnect ( u , µ, σ 2 ) is an L 2 -regularizer with a regularization coefﬁcient of mσ 2 /µ 2 .
3. M IXCONNECT TO M IXOUT
We propose mixout as a special case of mixconnect, which is motivated by the relationship between dropout and dropconnect.
We assume that
[Equation]
where w ( N i )
j is the j th parameter outgoing from the neuron N i .
We set the corresponding M to
M
[Equation]
· · · , M
[Equation]
N
[Equation]
) (cid:17) , (8) where E M ( N i ) = µ and Var( M ( N i ) )
=
σ 2 for all i .
In this paper, we set M ( N i ) to Bernoulli(1 − p ) for all i and mixout ( u ) hereafter refers to this correlated version of mixconnect with Bernoulli random masks.
We write it as “ mixout ( u , p ) ” when we emphasize the mix probability p .
Corollary 1.1.
Assume that the loss function L is strongly convex.
We denote the random mixture function of mixout ( u , p ) , which is equivalent to that of mixconnect ( u , 1 − p, p − p 2 ) , as Φ( w ; u , M ) where M is deﬁned in equation 8.
Then, there exists m > 0 such that
[Equation]
for all w .
Corollary 1.1 is a straightforward result from Theorem 1.
As the mix probability p in equationincreases to 1, the L 2 -regularization coefﬁcient of mp/ (1 − p ) increases to inﬁnity.
It means that p of mixout ( u , p ) can adjust the strength of L 2 -penalty toward u in optimization.
mixout ( u ) differs from wdecay( u ) since the regularization coefﬁcient of mixout ( u ) depends on m determined by the current model parameter w .
mixout ( u , p ) indeed regularizes learning to minimize the deviation from u .
We validate this by performing least squares regression in Supplement D.
We often apply dropout to speciﬁc layers.
For instance, Simonyan & Zisserman (2014) applied dropout to fully connected layers only.
We generalize Theorem 1 to the case in which mixout is only applied to speciﬁc layers, and it can be done by constructing M in a particular way.
We demonstrate this approach in Supplement B and show that mixout for speciﬁc layers adaptively
L 2 -penalizes their parameters.
3. M IXOUT FOR P RETRAINED M ODELS
Hoffer et al.
(2017) have empirically shown that
[Equation]
where w t is a model parameter after the t -th SGD step.
When training from scratch, we usually sample an initial model parameter w 0 from a normal/uniform distribution with mean 0 and small variance.
Since w 0 is close to the origin, w t is away from the origin only with a large t by equation 10.
When ﬁnetuning, we initialize our model parameter from a pretrained model parameter w pre .
Since we usually obtain w pre by training from scratch on a large pretraining dataset, w pre is often far away from the origin.
By Corollary 1.1, dropout L 2 -penalizes the model parameter for deviating away from the origin rather than w pre .
To explicitly prevent the deviation from w pre , we instead propose to use mixout ( w pre ) .
V ERIFICATION OF T HEORETICAL R ESULTS FOR M IXOUT ON MNIST
Wiese et al.
(2017) have highlighted that wdecay( w pre ) is an effective regularization technique to avoid catastrophic forgetting during ﬁnetuning.
Because mixout ( w pre ) keeps the ﬁnetuned model to stay in the vicinity of the pretrained model similarly to wdecay( w pre ) , we suspect that the proposed mixout ( w pre ) has a similar effect of alleviating the issue of catastrophic forgetting.
To empirically verify this claim, we pretrain a 784-300-100-10 fully-connected network on EMNIST Digits (Cohen et al., 2017), and ﬁnetune it on MNIST.
For more detailed description of the model architecture and datasets, see Supplement C.1.
In the pretraining stage, we run ﬁve random experiments with a batch size of 32 for { 1 , 2 , · · · , 20 } training epochs.
We use Adam (Kingma & Ba, 2014) with a learning rate of 10 − 4 , β 1 = 0 .
9 , β 2 = 0 .
999 , wdecay( 0 , 0 . 01) , learning rate warm-up over the ﬁrst 10% steps of the total steps, and linear decay of the learning rate after the warm-up.
We use dropout(0 . 1) for all layers except the input and output layers.
We select w pre whose validation accuracy on EMNIST Digits is best (0.992) in all experiments.
For ﬁnetuning, most of the model hyperparameters are kept same as in pretraining, with the exception of the learning rate, number of training epochs, and regularization techniques.
We train with a learning rate of 5 × 10 − 5 for 5 training epochs.
We replace dropout( p ) with mixout ( w pre , p ) .
We do not use any other regularization technique such as wdecay( 0 ) and wdecay( w pre ) .
We monitor (cid:107) w ft − w pre (cid:107) 2 , 1 validation accuracy on MNIST
[Figure]
pre and memorizes the source task better than dropout( p ) for all p .
As shown in Figure 2 (a), after ﬁnetuning with mixout ( w pre , p ) , the deviation from w pre is minimized in the L 2 -sense.
This result veriﬁes Corollary 1.1.
We demonstrate that the validation accuracy of mixout ( w pre , p ) has greater robustness to the choice of p than that of dropout( p ) .
In Figure 2 (b), both dropout( p ) and mixout ( w pre , p ) result in high validation accuracy on the target task (MNIST) for p ∈ { 0 . 1 , 0 . 2 , 0 . 3 } , although mixout ( w pre , p ) is much more robust with respect to the choice of the mix probability p .
In Figure 2 (c), the validation accuracy of mixout ( w pre , p ) on the source task (EMNIST Digits) drops from the validation accuracy of the model at w pre (0.992) to approximately 0.723 regardless of p .
On the other hand, the validation accuracy of dropout( p ) on the source task respectively drops by 0.041, 0.074 and 0.105 which are more than those of mixout ( w pre , p ) for p ∈ { 0 . 1 , 0 . 2 , 0 . 3 } .
F INETUNING A P RETRAINED L ANGUAGE M ODEL WITH M IXOUT
In order to experimentally validate the effectiveness of mixout, we ﬁnetune BERT LARGE on a subset of GLUE (Wang et al., 2018) tasks (RTE, MRPC, CoLA, and STS-B) with mixout ( w pre ) .
We choose them because Phang et al.
(2018) have observed that it was unstable to ﬁnetune BERT LARGE on these four tasks.
We use the publicly available pretrained model released by Devlin et al.
(2018), ported into PyTorch by HuggingFace.
3 We use the learning setup and hyperparameters recommended by Devlin et al.
We use Adam with a learning rate of 2 × 10 − 5 , β 1 = 0 .
9 , β 2 = 0 .
999 , learning rate warmup over the ﬁrst 10% steps of the total steps, and linear decay of the learning rate after the warmup ﬁnishes.
We train with a batch size of 32 for 3 training epochs.
Since the pretrained BERT LARGE is the sentence encoder, we have to create an additional output layer, which is not pretrained.
We initialize each parameter of it with N (0 , 0 .
We describe our experimental setup further in Supplement C.2.
The original regularization strategy used in Devlin et al.
(2018) for ﬁnetuning BERT LARGE is using both dropout(0 . 1) and wdecay( 0 , 0 . 01) for all layers except layer normalization and intermediate layers activated by GELU (Hendrycks & Gimpel, 2016).
We however cannot use mixout ( w pre ) nor wdecay( w pre ) for the additional output layer which was not pretrained and therefore does not have w pre .
We do not use any regularization for the additional output layer when ﬁnetuning BERT LARGE with mixout ( w pre ) and wdecay( w pre ) .
For the other layers, we replace dropout(0 . 1) and wdecay( 0 , 0 . 01) with mixout ( w pre ) and wdecay( w pre ) , respectively.
Phang et al.
(2018) have reported that large pretrained models (e.g., BERT LARGE ) are prone to degenerate performance when ﬁnetuned on a task with a small number of training examples, and that multiple random restarts 4 are required to obtain a usable model better than random prediction.
To compare ﬁnetuning stability of the regularization techniques, we need to demonstrate the distribution of model performance.
We therefore train BERT LARGE with each regularization strategy on each task with 20 random restarts.
We validate each random restart on the dev set to observe the behaviour of the proposed mixout and ﬁnally evaluate it on the test set for generalization.
We present the test score of our proposed regularization strategy on each task in Supplement C.3.
We ﬁnetune BERT LARGE with mixout ( w pre , { 0 . 7 , 0 . 8 , 0 . 9 } ) on RTE, MRPC, CoLA, and STSB.
For the baselines, we ﬁnetune BERT LARGE with both dropout(0 . 1) and wdecay( 0 , 0 . 01) as well as with wdecay( w pre , { 0 . 01 , 0 . 04 , 0 . 07 , 0 . 10 } ) .
These choices are made based on the experiments in Section 6.3 and Supplement F.
In Section 6.3, we observe that ﬁnetuning BERT LARGE with mixout ( w pre , p ) on RTE is signiﬁcantly more stable with p ∈ { 0 . 7 , 0 . 8 , 0 . 9 } while ﬁnetuning with dropout( p ) becomes unstable as p increases.
In Supplement F, we demonstrate that dropout(0 . 1) is almost optimal for all the tasks in terms of mean dev score although Devlin et al.
(2018) selected it to improve the maximum dev score.
In Figure 3, we plot the distributions of the dev scores from 20 random restarts when ﬁnetuning
BERT LARGE with various regularization strategies on each task.
For conciseness, we only show four regularization strategies; Devlin et al.
(2018)’s: both dropout(0 . 1) and wdecay( 0 , 0 . 01) ,
Wiese et al.
(2017)’s: wdecay( w pre , 0 . 01) , ours: mixout ( w pre , 0 . 7) , and ours+Wiese et al.
(2017)’s: both mixout ( w pre , 0 . 7) and wdecay( w pre , 0 . 01) .
As shown in Figure 3 (a–c), we observe many ﬁnetuning runs that fail with the chance-level accuracy when we ﬁnetune BERT LARGE with both dropout(0 . 1) and wdecay( 0 , 0 . 01) on RTE, MRPC, and CoLA.
We also have a bunch of degenerate model conﬁgurations when we use wdecay( w pre , 0 . 01) without mixout ( w pre , 0 . 7) .
Unlike existing regularization strategies, when we use mixout ( w pre , 0 . 7) as a regularization technique with or without wdecay( w pre , 0 . 01) for ﬁnetuning BERT LARGE , the number of degenerate model conﬁgurations that fail with a chance-level accuracy signiﬁcantly decreases.
For example, in
Figure 3 (c), we have only one degenerate model conﬁguration when ﬁnetuning BERT LARGE with mixout ( w pre , 0 . 7) on CoLA while we observe respectively seven and six degenerate models with
Devlin et al.
(2018)’s
and Wiese et al.
(2017)’s regularization strategies.
[Figure]
In Figure 3 (a)
[Figure]
we further improve the stability of ﬁnetuning BERT LARGE by using both mixout ( w pre , 0 . 7) and wdecay( w pre , 0 . 01) .
Figure 3 (d) shows respectively two and one degenerate model conﬁgurations with Devlin et al.
(2018)’s
and Wiese et al.
(2017)’s,
but we do not have any degenerate resulting model with ours and ours+Wiese et al.
(2017)’s.
In Figure 3 (b, c), we observe that the number of degenerate model conﬁgurations increases when we use wdecay( w pre , 0 . 01) additionally to mixout ( w pre , 0 . 7) .
In short, applying our proposed mixout signiﬁcantly stabilizes the ﬁnetuning results of BERT LARGE on small training sets regardless of whether we use wdecay( w pre , 0 . 01) .
In Table 1, we report the average and the best dev scores across 20 random restarts for each task with various regularization strategies.
The average dev scores with mixout ( w pre , { 0 . 7 , 0 . 8 , 0 . 9 } ) increase for all the tasks.
For instance, the mean dev score of ﬁnetuning with mixout ( w pre , 0 .
8) on
CoLA is 57.9 which is 49.2% increase over 38.8 obtained by ﬁnetuning with both dropout( p ) and wdecay( 0 , 0 . 01) .
We observe that using wdecay( w pre , { 0 . 01 , 0 . 04 , 0 . 07 , 0 . 10 } ) also improves the average dev scores for most tasks compared to using both dropout( p ) and wdecay( 0 , 0 . 01) .
We however observe that ﬁnetuning with mixout ( w pre , { 0 . 7 , 0 . 8 , 0 . 9 } ) outperforms that with wdecay( w pre , { 0 . 01 , 0 . 04 , 0 . 07 , 0 . 10 } ) on average.
This conﬁrms that mixout ( w pre ) has a different effect for ﬁnetuning BERT LARGE compared to wdecay( w pre ) since mixout ( w pre ) is an adaptive L 2 -regularizer along the optimization trajectory.
Since ﬁnetuning a large pretrained language model such as BERT LARGE on a small training set frequently fails, the ﬁnal model performance has often been reported as the maximum dev score (Devlin et al., 2018;
Phang et al., 2018) among a few random restarts.
We thus report the best dev score for each setting in Table 1.
According to the best dev scores as well, mixout ( w pre , { 0 . 7 , 0 . 8 , 0 . 9 } ) improves performance for all the tasks compared to using both dropout( p ) and wdecay( 0 , 0 . 01) .
For instance, using mixout ( w pre , 0 . 9) improves the maximum dev score by 0.9 compared to using both dropout( p ) and wdecay( 0 , 0 . 01) on MRPC.
Unlike the average dev scores, the best dev scores achieved by using wdecay( w pre , { 0 . 01 , 0 . 04 , 0 . 07 , 0 . 10 } ) are better than those achieved by using mixout ( w pre , { 0 . 7 , 0 . 8 , 0 . 9 } ) except RTE on which it was better to use mixout ( w pre , { 0 . 7 , 0 . 8 , 0 . 9 } ) than wdecay( w pre , { 0 . 01 , 0 . 04 , 0 .
We only
The ( w greatly pre
[Table] [Table]
[Table]
We investigate the effect of combining both mixout ( w pre ) and wdecay( w pre ) to see whether they are complementary.
We ﬁnetune BERT LARGE with both mixout ( w pre , { 0 . 7 , 0 . 8 , 0 . 9 } ) and wdecay( w pre , 0 . 01) .
This leads not only to the improvement in the average dev scores but also in the best dev scores compared to using wdecay( w pre , { 0 . 01 , 0 . 04 , 0 . 07 , 0 . 10 } ) and using both dropout( p ) and wdecay( 0 , 0 . 01) .
The experiments in this section conﬁrm that using mixout ( w pre ) as one of several regularization techniques prevents ﬁnetuning instability and yields gains in dev scores.
A BLATION S TUDY
In this section, we perform ablation experiments to better understand mixout ( w pre ) .
Unless explicitly stated, all experimental setups are the same as in Section 5.
6. M IXOUT WITH A S UFFICIENT N UMBER OF T RAINING E XAMPLES
We showed the effectiveness of the proposed mixout ﬁnetuning with only a few training examples in Section 5.
In this section, we investigate the effectiveness of the proposed mixout in the case of a larger ﬁnetuning set.
Since it has been stable to ﬁnetune BERT LARGE on a sufﬁcient number of training examples (Devlin et al., 2018; Phang et al., 2018), we expect to see the change in the behaviour of mixout ( w pre ) when we use it to ﬁnetune BERT LARGE on a larger training set.
We train BERT LARGE by using both mixout ( w pre , 0 . 7) and wdecay( w pre , 0 . 01) with 20 random restarts on SST-2.
5 We also train BERT LARGE by using both dropout( p ) and wdecay( 0 , 0 . 01) with 20 random restarts on SST-2 as the baseline.
In Table 2, we report the mean and maximum of
SST-2 dev scores across 20 random restarts with each regularization strategy.
We observe that there is little difference between their mean and maximum dev scores on a larger training set, although using both mixout ( w pre , 0 . 7) and wdecay( w pre , 0 . 01) outperformed using both dropout( p ) and wdecay( 0 , 0 . 01) on the small training sets in Section 5.
TECHNIQUE
[Table]
6. E FFECT OF A R EGULARIZATION T ECHNIQUE FOR AN A DDITIONAL O UTPUT L AYER
In this section, we explore the effect of a regularization technique for an additional output layer.
There are two regularization techniques available for the additional output layer: dropout( p ) and mixout ( w 0 , p ) where w 0 is its randomly initialized parameter.
Either of these strategies differs from the earlier experiments in Section 5 where we did not put any regularization for the additional output layer.
best of each result from all the regularization strategies shown in Table 1.
Bold marks the best of each statistics within each column.
We obtain additional gains in dev scores by varying the regularization technique for the additional output layer.
[Table]
mixout ( w pre , 0 . 7) - 61.6 ( 74.0 ) 87.1 (
[Table]
[Table] [Table]
[Table]
mixout ( w pre , 0 . 7) mixout ( w 0 , 0 . 7)
[Table]
[Table]
mixout ( w pre , 0 . 7)
[Table]
[Table]
The best of each result from Table
[Table]
We report the average and best dev scores across 20 random restarts when ﬁnetuning BERT LARGE with mixout ( w pre , 0 . 7) while varying the regularization technique for the additional output layer in
Table 3.
6 We observe that using mixout ( w 0 , 0 . 7) for the additional output layer improves both the average and best dev score on RTE, CoLA, and
STS-B.
In the case of MRPC, we have the highest best-dev score by using dropout(0 . 7) for the additional output layer while the highest mean dev score is obtained by using mixout ( w 0 , 0 . 7) for it.
In Section 3.2, we discussed how mixout ( w 0 ) does not differ from dropout when the layer is randomly initialized, since we sample w 0 from w whose mean and variance are 0 and small, respectively.
Although the additional output layer is randomly initialized, we observe the signiﬁcant difference between dropout and mixout ( w 0 ) in this layer.
We conjecture that (cid:107) w 0 − 0 (cid:107) is not sufﬁciently small because E (cid:107) w − 0 (cid:107) is proportional to the dimensionality of the layer (2,048).
We therefore expect mixout ( w 0 ) to behave differently from dropout even for the case of training from scratch.
In the last row of Table 3, we present the best of the corresponding result from Table 1.
We have the highest mean and best dev scores when we respectively use mixout ( w pre , 0 . 7) and mixout ( w 0 , 0 . 7) for the pretrained layers and the additional output layer on RTE, CoLA, and STSB.
The highest mean dev score on MRPC is obtained by using mixout ( w pre , 0 .
8) for the pretrained layers which is one of the results in Table 1.
We have the highest best dev score on MRPC when we use mixout ( w pre , 0 . 7) and dropout(0 . 7) for the pretrained layers and the additional output layer, respectively.
The experiments in this section reveal that using mixout ( w 0 ) for a randomly initialized layer of a pretrained model is one of the regularization schemes to improve the average dev score and the best dev score.
6. E FFECT OF M IX P ROBABILITY FOR M IXOUT AND D ROPOUT
We explore the effect of the hyperparameter p when ﬁnetuning BERT LARGE with mixout ( w pre , p ) and dropout( p ) .
We train BERT LARGE with mixout ( w pre , { 0 . 0 , 0 . 1 , · · · , 0 . 9 } ) on RTE with 20 random restarts.
We also train BERT LARGE after replacing mixout ( w pre , p ) by dropout( p ) with 20 random restarts.
We do not use any regularization technique for the additional output layer.
Because we use neither wdecay( 0 ) nor wdecay( w pre ) in this section, dropout(0 . 0) and mixout ( w pre , 0 . 0) are equivalent to ﬁnetuning without regularization.
[Figure]
It is not helpful to vary p for dropout( p ) while mixout ( w pre , p ) helps signiﬁcantly in a wide range of p .
Figure 4 shows distributions of RTE dev scores across 20 random restarts when ﬁnetuning
BERT LARGE with dropout( p ) and mixout ( w pre , p ) for p ∈ { 0 . 0 , 0 . 1 , · · · , 0 . 9 } .
The mean dev score of ﬁnetuning BERT LARGE with mixout ( w pre , p ) increases as p increases.
On the other hand, the mean dev score of ﬁnetuning BERT LARGE with dropout( p ) decreases as p increases.
If p is less than 0.4, ﬁnetuning with mixout ( w pre , p ) does not improve the ﬁnetuning results of using dropout( { 0 . 0 , 0 . 1 , 0 . 2 } ) .
We however observe that mixout ( w pre , { 0 . 7 , 0 . 8 , 0 . 9 } ) yields better average dev scores than dropout( p ) for all p , and signiﬁcantly reduces the number of ﬁnetuning runs that fail with the chance-level accuracy.
We notice that the proposed mixout spends more time than dropout from the experiments in this section.
It takes longer to ﬁnetune a model with the proposed mixout than with the original dropout, although this increase is not signiﬁcant especially considering the waste of time from failed ﬁnetuning runs using dropout.
In Supplement E, we describe more in detail the difference between mixout and dropout in terms of wall-clock time.
C ONCLUSION
The special case of our approach, mixout ( w pre ) , is one of several regularization techniques modifying a ﬁnetuning procedure to prevent catastrophic forgetting.
Unlike wdecay( w pre ) proposed earlier by Wiese et al.
(2017), mixout ( w pre ) is an adaptive L 2 -regularizer toward w pre in the sense that its regularization coefﬁcient adapts along the optimization path.
Due to this difference, the proposed mixout improves the stability of ﬁnetuning a big, pretrained language model even with only a few training examples of a target task.
Furthermore, our experiments have revealed the proposed approach improves ﬁnetuning results in terms of the average accuracy and the best accuracy over multiple runs.
We emphasize that our approach can be used with any pretrained language models such as RoBERTa (Liu et al., 2019) and
XLNet (Yang et al., 2019), since mixout does not depend on model architectures, and leave it as future work.
A CKNOWLEDGMENTS
The ﬁrst and third authors’ work was supported by the National Research Foundation of Korea (NRF) grants funded by the Korea government (MOE, MSIT) (NRF-2017R1A2B4011546, NRF2019R1A5A1028324).
The second author thanks support by AdeptMind, eBay, TenCent, NVIDIA and CIFAR and was partly supported by Samsung Electronics (Improving Deep Learning using
Latent Structure)
S UPPLEMENTARY M ATERIAL A P ROOFS FOR T HEOREM
Theorem 1.
Assume that the loss function L is strongly convex.
Suppose that a random mixture function with respect to w from u and M is given by
[Equation]
where M is diag( M 1 , M 2 , · · · , M d ) satisfying E M i = µ and Var( M i )
=
σ 2 for all i .
Then, there exists m > 0 such that
[Equation]
for all w .
Proof.
Since L is strongly convex [Equation] there exist m > 0 such that
[Equation]
for all w by equation 1.
Recall that E M i = µ and Var( M i ) =
σ 2 for all i .
Then, we have
[Equation]
[Equation]
13) and
[Equation]
[Equation]
By using equation 13 and equation 14, we can rewrite equation 12 as
[Equation]
B A PPLYING TO S PECIFIC L AYERS
We often apply dropout to speciﬁc layers.
For instance, Simonyan & Zisserman (2014) applied dropout to fully connected layers only.
We generalize Theorem 1 to the case in which mixconnect is only applied to speciﬁc layers, and it can be done by constructing M in a particular way.
To better characterize mixconnect applied to speciﬁc layers, we deﬁne the index set I as I =
{ i : M i = 1 } .
Furthermore, we use ˜ w and ˜ u to denote ( w i ) i/ ∈
I and ( u i )
i/ ∈
I , respectively.
Then, we generalize equation 7 to
[Equation]
From equation 15, applying mixconnect ( u , µ, σ 2 ) is to use adaptive wdecay(˜ u ) on the weight parameter of the speciﬁc layers ˜ w .
Similarly, we can regard applying mixout ( u , p ) to speciﬁc layers as adaptive wdecay( ˜ u ) .
C E XPERIMENTAL D ETAILS C. F ROM EMNIST D IGITS TO MNIST
Model Architecture The model architecture in Section 4 is a 784-300-100-10 fully connected network with a softmax output layer.
For each hidden layer, we add layer normalization (Ba et al., 2016) right after the ReLU (Nair & Hinton, 2010) nonlinearity.
We initialize each parameter with
N (0 , 0 . 02 2 ) and each bias with 0.
Regularization
In the pretraining stage, we use dropout(0 . 1) and wdecay( 0 , 0 . 01) .
We apply dropout(0 . 1) to all hidden layers.
That is, we do not drop neurons of the input and output layers.
wdecay( 0 , 0 . 01) does not penalize the parameters for bias and layer normalization.
When we ﬁnetune our model on MNIST, we replace dropout( p ) with mixout ( w pre , p ) .
We use neither wdecay( 0 ) nor wdecay( w pre ) for ﬁnetuning.
Dataset For pretraining, we train our model on EMNIST Digits.
This dataset has 280,000 characters into 10 balanced classes.
These characters are compatible with MNIST characters.
EMNIST
Digits provides 240,000 characters for training and 40,000 characters for test.
We use 240,000 characters provide for training and split these into the training set (216,000 characters) and validation set (24,000 characters).
For ﬁnetuning, we train our model on MNIST.
This has 70,000 characters into 10 balance classes.
MNIST provide 60,000 characters for training and 10,000 characters for test.
We use 60,000 characters given for training and split these into the training set (54,000 characters) and validation set (6,000 characters).
Data Preprocessing We only use normalization after scaling pixel values into [0 , 1] .
We do not use any data augmentation.
C. F INETUNING BERT ON PARTIAL GLUE TASKS
Model Architecture Because the model architecture of BERT LARGE is identical to the original (Devlin et al., 2018), we omit its exhaustive description.
Brieﬂy, BERT LARGE has 24 layers,hidden size, and 16 self-attention heads (total 340M parameters).
We use the publicly available pretrained model released by Devlin et al.
(2018), ported into PyTorch by HuggingFace.
7 We initialize each weight parameter and bias for an additional output layer with N (0 , 0 .
02 2 ) and 0, respectively.
Regularization
In the ﬁnetuning stage, Devlin et al.
(2018) used wdecay( 0 , 0 . 01) for all parameters except bias and layer normalization.
They apply dropout(0 . 1) to all layers except each hidden layer activated by GELU (Hendrycks & Gimpel, 2016) and layer normalization.
We substitute wdecay( w pre ) and mixout ( w pre ) for wdecay( 0 , 0 . 01) and dropout(0 . 1) , respectively.
Dataset We use a subset of GLUE (Wang et al., 2018) tasks.
The brief description for each dataset is as the following:
• RTE (2,500 training examples): Binary entailment task (Dagan et al., 2006) • MRPC (3,700 training examples): Semantic similarity (Dolan & Brockett, 2005) • CoLA (8,500 training examples): Acceptability classiﬁcation (Warstadt et al., 2018)
• STS-B (7,000 training examples): Semantic textual similarity (Cer et al., 2017) • SST-2 (67,000 training examples): Binary sentiment classiﬁcation (Socher et al., 2013)
In this paper, we reported F1 accuracy scores for MRPC, Mattew’s correlation scores for CoLA,
Spearman correlation scores for STS-B, and accuracy scores for the other tasks.
Data Preprocessing We use the publicly available implementation of BERT LARGE by HuggingFace.7 https : / / co bert / bert-large-uncased-pytorch_model.binhttps://github com/
/pytorch-transformers
C. T EST R ESULTS ON GLUE TASKS
We expect that using mixout stabilizes ﬁnetuning results of BERT LARGE on a small training set.
To show this, we demonstrated distributions of dev scores from 20 random restarts on RTE, MRPC,
CoLA and STS-B in Figure 3.
We further obtained the highest average/best dev score on each task in Table 3.
To conﬁrm the generalization of the our best model on the dev set, we demonstrate the test results scored by the evaluation server 9 in Table 4.
[Table]
+ wdecay
[Table]
For all the tasks except MRPC, the test scores obtained by the proposed mixout 10 are better than those reported by Devlin et al.
We explored the behaviour of ﬁnetuning BERT LARGE with mixout by using the learning rate of 2 × 10 − 5 while Devlin et al.
(2018) obtained their results by using the learning rate of { 2 × 10 − 5 , 3 × 10 − 5 , 4 × 10 − 5 , 5 × 10 − 5 } .
We thus present the test scores obtained by the regularization strategy of Devlin et al.
(2018) when the learning rate is 2 × 10 − 5 .
The results in this section show that the best model on the dev set generalizes well, and all the experiments based on dev scores in this paper are proper to validate the effectiveness of the proposed mixout.
For the remaining GLUE tasks such as SST-2 with a sufﬁcient number of training instances, we observed that using mixout does not differs from using dropout in Section 6.1.
We therefore omit the test results on the other tasks in GLUE.
D V ERIFICATION OF C OROLLARY 1.1 WITH L EAST S QUARES R EGRESSION
Corollary 1.1 shows that mixout ( u , p ) regularizes learning to minimize the deviation from the target model parameter u , and the strength of regularization increases as p increases when the loss function is strongly convex.
In order to validate this, we explore the behavior of least squares regression with mixout ( u , p ) on a synthetic dataset.
For randomly given w ∗ 1 and w ∗ 2 , we generated an observation y satisfying y =
w ∗ 1 x + w ∗ 2 + (cid:15) where (cid:15) is Gaussian noise.
We set the model to ˆ y =
w 1 x + w 2 .
That is, the model parameter w is given by ( w 1 , w 2 ) .
We randomly pick u as a target model parameter for mixout ( u , p ) and perform least squares regression with mixout ( u , { 0 . 0 , 0 . 3 , 0 . 6 , 0 . 9 } ) .
As shown in Figure 5, w converges to the target model parameter u rather than the true model parameter w ∗ =
( w ∗ 1 , w ∗ 2 ) as the mix probability p increases.
[Figure]
E T IME U SAGE OF M IXOUT C OMPARED TO D ROPOUT
We recorded the training time of the experiment in Section 6.3 to compare the time usage of mixout and that of dropout.
It took about 843 seconds to ﬁnetune BERT LARGE with mixout ( w pre ) .
On the other hand, it took about 636 seconds to ﬁnetune BERT LARGE with dropout.
mixout ( w pre ) spends 32.5% more time than dropout since mixout ( w pre ) needs an additional computation with the pretrained model parameter w pre .
However, as shown in Figure 4, at least 15 ﬁnetuning runs among 20 random restarts fail with the chance-level accuracy on RTE with dropout( p ) for all p while only 4 ﬁnetuning runs out of 20 random restarts are unusable with mixout ( w pre , 0 .
From this result, it is reasonable to ﬁnetune with the proposed mixout although this requires additional time usage compared to dropout.
F E XTENSIVE H YPERPARAMETER S EARCH FOR D ROPOUT
Devlin et al.
(2018) ﬁnetuned BERT LARGE with dropout(0 . 1) on all GLUE (Wang et al., 2018) tasks.
They chose it to improve the maximum dev score on each downstream task, but we have reported not only the maximum dev score but also the mean dev score to quantitatively compare various regularization techniques in our paper.
In this section, we explore the effect of the hyperparameter p when ﬁnetuning BERT LARGE with dropout( p ) on RTE, MRPC, CoLA, and STS-B to show dropout(0 . 1) is optimal in terms of mean dev score.
All experimental setups for these experiments are the same as Section 6.3.
[Figure]
As shown in Figure 6, we have the highest average dev score on MRPC with dropout(0 . 1) as well as on STS
-B. We obtain the highest average dev scores with dropout(0 . 0) on RTE and CoLA, but we get the second-highest average dev scores with dropout(0 .
1) on them.
These experiments conﬁrm that the drop probability 0.1 is almost optimal for the highest average dev score on each task.