A
VARIANCE PRINCIPLE EXPLAINS WHY DROPOUT FINDS FLATTER MINIMA
Anonymous authors
Paper under double-blind review
A BSTRACT
Although dropout has achieved great success in deep learning, little is known about how it helps the training ﬁnd a good generalization solution in the highdimensional parameter space.
In this work, we show that the training with dropout ﬁnds the neural network with a ﬂatter minimum compared with standard gradient descent training.
We further study the underlying mechanism of why dropout ﬁnds ﬂatter minima through experiments.
We propose a Variance Principle that the variance of a noise is larger at the sharper direction of the loss landscape.
Existing works show that SGD satisﬁes the variance principle, which leads the training to ﬂatter minima.
Our work show that the noise induced by the dropout also satisﬁes the variance principle that explains why dropout ﬁnds ﬂatter minima.
In general, our work points out that the variance principle is an important similarity between dropout and SGD that lead the training to ﬁnd ﬂatter minima and obtain good generalization.
I NTRODUCTION
Dropout is used with gradient-descent-based algorithms for training DNNs (Hinton et al., 2012;
Srivastava et al., 2014).
During training, the output of each neuron is multiplied with a random variable with probability p as one and 1 − p as zero.
Note that p is called dropout rate, and every time for computing concerned quantity, the variable is randomly sampled at each feedforward operation.
Dropout has been an indispensable trick in the training of deep neural networks (DNNs), however, with very little understanding.
Similar to SGD, training with dropout is equivalent to that with some speciﬁc noise.
To understand what kind of noise beneﬁts the generalization of training, we proposes a variance principle of a noise, that is,
Variance Principle: the variance of a noise is larger at the sharper direction of the loss landscape.
If a noise satisﬁes the variance principle, it can help the training select ﬂatter minima and leads the training to better generalization.
As shown in Zhu et al.
; Feng & Tu (2021), the noise in SGD satisﬁes the variance principle and SGD can ﬁnd ﬂatter minima and obtain better generalization (Keskar et al., 2016; Neyshabur et al., 2017).
In this work, we study the characteristic of minima learned with dropout.
We show that compared with the standard gradient descent (GD), the GD with dropout selects ﬂatter minima.
As suggested by many existing works (Keskar et al., 2016; Neyshabur et al., 2017; Zhu et al., 2018), ﬂatter minima are more likely to have better generalization and stability.
We then put efforts to show that the noise induced by the dropout satisﬁes the variance principle, which explains why dropout ﬁnds ﬂatter minima.
To examine the variance principle, we explore the relation between the ﬂatness of the loss landscape and the noise structure induced by dropout at minima through three methods and obtain a consistent result that the noise is larger at the sharper direction of the loss landscape to help the training select ﬂatter minima.
First, we examine the inverse variance-ﬂatness relation, similar to Feng & Tu (2021).
We deﬁne the ﬂatness of a minimum at one direction by the length of the largest interval in the considered direction, which covers the minimum and no point in the interval has loss larger than twice of the loss of the minimum, denoted by F p for the direction p .
We then consider two deﬁnitions of the noise structure, i.e., random trajectory covariance Σ t and random gradient covariance Σ g .
For the random trajectory covariance, we train the network to an “exploration phase” (Shwartz-Ziv & Tishby, 2017), where the loss decreases with a very slow speed and then sample parameter sets { θ i } Ni =1 from N consecutive training steps to compute the covariance, where θ i is the network parameter set at step i .
For the gradient covariance, we train the network until the loss is very small and then freeze the training.
We sample N gradients { g i } Ni =1 with different dropout variables to compute the covariance.
In each sample, the dropout rate is ﬁxed.
For both random trajectory covariance and random gradient covariance, we perform principal component analysis (PCA) and obtain similar results.
We ﬁnd that at the direction of larger variance (larger eigen-value), the loss landscape of the minimum is sharper, i.e., inverse variance-ﬂatness relation.
Second, we study the relation between the Hessian and the noise structure induced by dropout.
The eigenvalues of the Hessian of the loss at a minimum are also often used to indicate the ﬂatness.
The landscape at an eigen-direction is claimed sharper if the corresponding eigen-value is larger.
For each eigen-direction v j , we project the parameter trajectory { θ i } Ni =1 or gradients { g i } Ni =1 to the direction of v j and compute the variance.
We ﬁnd that the noise variance is larger at the direction with a larger eigen-value.
Third, we show that the Hessian matrix aligns well with the random gradient covariance of gradients { g i } Ni =1 , i.e., their eigen directions of large eigen-values are close, similar to Zhu et al.
These empirical works show that the noise structure induced by the dropout tends to have larger variance in order to escape the sharper direction, i.e., variance principle, thus, leading to ﬂatter minima.
These characteristics of dropout are very similar to SGD (Keskar et al., 2016; Zhu et al., 2018;
Feng & Tu, 2021).
The similarity between dropout and SGD suggests that modelling their similarity may be a key to understanding how and what stochasticity beneﬁts the training.
R ELATED WORKS
Dropout was proposed as a simple way to prevent neural networks from overﬁtting, and thus improving the generalization of the network, to a certain extent (Hinton et al., 2012; Srivastava et al., 2014).
Many works aim to ﬁnd an explicit regularization form of dropout.
Wager et al.
(2013) studies the explicit form of dropout on linear regression and logistic problem, but for studying non-linear neural network, it is still unclear how to characterize the effect of dropout by an explicit regularization term.
McAllester (2013) presents PAC-Bayesian bounds, and Wan et al.
, Mou et al.
(2018) derives
Rademacher generalization bounds.
These results show that the reduction of complexity brought by dropout is O ( p ) , where p is the probability of keeping an element in dropout.
Mianjy & Arora (2020) show that dropout training with logistic loss achieves (cid:15) -suboptimality in test error in O (1 /(cid:15) ) iterations.
All of the above works need speciﬁc settings, such as norm assumptions and logistic loss, and they only give a rough estimate of the generalization error bound, which usually consider the worst case.
However, it is not clear what is the characteristic of the dropout training process and how to bridge the training with the generalization.
In this work, we show that dropout noise has a special structure, which closely relates with the loss landscape.
The structure of the effective noise induced by the dropout may be a key reason why dropout can ﬁnd solutions with better generalization.
Many researches have empirically shown that SGD can improve the generalization performance of neural networks through ﬁnding a ﬂatter solution (Li et al., 2017; Jastrzebski et al., 2017; 2018).
This work utilizes the current understanding of SGD to study dropout and shows that much similarity is shared between SGD and dropout.
The ﬂatness of the solution is an important aspect of understanding the generalization of neural networks (Keskar et al., 2016; Neyshabur et al., 2017; Zhu et al., 2018).
A number of works suggested that the learning rate and batch size determine the ﬂatness of the solutions (Jastrzebski et al., 2017; 2018; Wu et al., 2018).
Li et al.
(2017) propose a visualization method of the loss landscape at 1-d cross-section to visualize the ﬂatness.
Feng & Tu (2021) investigate the connection between SGD learning dynamics and the loss landscape through the principal component analysis (PCA), and show that SGD dynamics follow a lowdimensional drift-diffusion motion in the weight space.
Through characterizing the loss landscape by its ﬂatness in each PCA direction around the solution found by SGD, they also reveal a robust in-
[Table]
λ (Σ) vs. F v , Fig. 2,
[Table]
{ Var(Proj v i ( S )) , λ i ( H ) } , Fig.
Alignment: Tr ( H Σ g ) , Fig.
verse relation between the weight variance and the landscape ﬂatness in PCA directions, thus ﬁnding that SGD serves as a landscape dependent annealing algorithm to search for ﬂat minima.
Zhu et al.
(2018) study a general form of gradient based optimization dynamics with unbiased noise to analyze the behavior of SGD on escaping from minima and its regularization effects.
They also introduce an indicator to characterize the efﬁciency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function and thus revealing the anisotropic noise of SGD.
P RELIMINARY 3. D ROPOUT
Consider an L -layer neural network f θ ( x ) .
With dropout (Srivastava et al., 2014), the feedforward operation in a network f θ ( x ) is
[Equation] [Equation]
[Equation] [Equation]
4) where p is the dropout rate, W [ l ] ∈ R m l +1 × m l , b [ l ] =
R m l
[Equation]
m {+
+}
=
d in = d , m L = d o , σ is a scalar function and “ ◦ ” means entry-wise operation.
We denote the set of parameters by
[Equation]
3. I NTERVAL FLATNESS
We use the deﬁnition of ﬂatness in Feng & Tu (2021).
For convenience, we call it interval ﬂatness
Around a speciﬁc solution θ ∗ 0 , we compute the loss function proﬁle
L v along the direction v :
[Equation]
The interval ﬂatness F v is deﬁned as the width of the region within which L v ( δθ ) ≤ 2 L v (0) .
We determine F v by ﬁnding two closest points θ l v < 0 and θ r v > 0 on each side of the minimum that satisfy L v ( θ l v ) =
L v ( θ r v ) =
2 L v (0) .
The scale factor 2 is used in Feng & Tu (2021), and after our test, the result is not sensitive to the selection of this factor.
In this work, we follow their experimental scheme to show the similarity between dropout and SGD.
The interval ﬂatness is deﬁned as:
[Equation]
A larger value of F v means a ﬂatter landscape in the direction v .
3. R ANDOMNESS INDUCED BY DROPOUT 3.3. R ANDOM TRAJECTORY DATA
The training process of neural networks are usually divided into two phases, fast convergence and exploration phase (Shwartz-Ziv & Tishby, 2017).
Feng & Tu (2021)’s work focuses on the behavior of networks in the exploration phase.
In this work, we follow the experimental scheme in Feng &
Tu (2021) to show the similarity between dropout and SGD.
This can be understood by frequency principle (Xu et al., 2019; 2020), which states that DNNs fast learn low-frequency components but slowly learn high-frequency ones.
We collect parameter sets S para =
{ θ i } Ni =1 from N consecutive training steps in the exploration phase, where θ i is the network parameter set at step i .
3.3. R ANDOM GRADIENT DATA
We train the network until the loss is very small and then freeze the training.
We sample N gradients of the loss function w.r.t.
the parameters with different dropout variables, i.e., S grad =
{ g i } Ni =1 .
In each sample, the dropout rate is ﬁxed.
3. I NVERSE VARIANCE - FLATNESS RELATION
We study the inverse variance ﬂatness relation for both random trajectory data and random gradient data.
For convenience, we denote data as S and its covariance as Σ .
3.4. I NTERVAL FLATNESS VS . VARIANCE
Following Feng & Tu (2021), we perform principal component analysis (PCA) for Σ .
Denote λ i (Σ) as the i -th eigenvalue of Σ , and denote its corresponding eigen-vector as v i (Σ) .
We then compute the ﬂatness of the loss landscape in the direction v i (Σ) , and denote it as F v i (Σ) .
Finally, we display the scatter plot of { λ i (Σ) , F v i (Σ) } .
3.4. P ROJECTED VARIANCE VS . H ESSIAN FLATNESS
The method in Feng & Tu (2021), requires many training steps to obtain the covariance.
To obtain the variance induced by the dropout at a ﬁxed position θ , we propose another way to characterize the inverse variance-ﬂatness relation.
We use the the eigen-value λ i ( H ) of the Hessian H of the loss landscape at θ to denote the sharpness.
Hessian matrix is the matrix obtained by the second derivative of the loss function of the neural network with respect to the parameter vector of neural network.
Here, the parameter vector is a vector consisting of all the parameters of network.
The variance induced by dropout at the corresponding eigen-direction v i ( H ) is computed by the following procedure.
For each eigen-direction v i of Hessian H , we project the sampled parameters or the gradients of sampled parameter S to direction v i by inner product, denoted by Proj v i ( S ) .
Then, the variance at direction v i ( H ) is the variance of dataset Proj v i ( S ) , denoted by Var(Proj v
i ( S )) .
Finally, we display the scatter plot of { Var(Proj v i ( S )) , λ i ( H ) } .
3. A LIGNMENT BETWEEN H ESSIAN AND GRADIENT COVARIANCE
We use the method in Zhu et al.
(2018) to quantify the alignment between the noise structure and the curvature of loss surface.
For each training step i , we calculate the alignment parameter T i :
[Equation]
where Σ i is the i th-step covariance matrix generated by dropout layers and H i is the i th-step Hessian matrix of network parameters.
E XPERIMENTAL SETUP
To understand the effects of dropout, we trained a number of networks with different structures.
We consider two types of neural networks: 1) Fully-connected neural networks (FNNs) that are optimized for performance on MNIST.
For FNN, all parameters are initialized by Xavier initialization(Glorot & Bengio, 2010).
Convolutional neural networks (CNNs) that are optimized for performance on CIFAR-10.
For CNN, all parameters are initialized by He initialization (He et al., 2015).
The loss of all our experiments is cross entropy loss.
Deep Residual neural networks (ResNets)(He et al., 2016) that are optimized for performance on CIFAR-100.
For ResNet, all parameters are initialized by He initialization (He et al., 2015).
The loss of all our experiments is cross entropy loss.
For Fig.
1 (a), we use the FNN with size 784 − 1024 − 1024 − 10 .
We add dropout layers behind the ﬁrst and the second layers with dropout rate of 0.8 and 0.5, respectively.
We train the network using default Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.0001.
For Fig.
1 (b), we use vgg-9 (Simonyan & Zisserman, 2014) to compare the loss landscape ﬂatness w/o dropout layers.
Models are trained using GD with Nesterov momentum, training-size 2048 for 300 epochs.
The learning rate was initialized at 0.1, and divided by a factor of 10 at epochs 150,and 275.
We only use 2048 examples for training to compromise with the computational burden.
For Fig.
1 (c), 3, 5, we use ResNet-20 (He et al., 2016) to compare the loss landscape ﬂatness w/o dropout layers.
Models are trained using GD, training-size 50000 for 1200 epochs.
The learning rate was initialized at 0.01.
Since ResNet’s Hessian calculation takes a lot of time, for the ResNet experiment, we only perform it at a speciﬁc dropout rate and learning rate.
For Fig. 2, 4, 6, we use the FNN with size 784 − 50 − 50 − 10 .
We train the network using GD with the ﬁrst 10,000 training data as the training set.
We add a dropout layer behind the second layer.
The dropout rate and learning rate are speciﬁed and unchanged in each experiment.
D ROPOUT FINDS FLATTER MINIMA
Dropout is almost ubiquitous in training deep networks.
It is interesting and important to understand what makes dropout improve the generalization of training neural networks.
Inspired by the study of SGD (Keskar et al., 2016), we explore the ﬂatness of the minima found by dropout.
The loss landscape of DNNs is highly non-convex and complicate (Skorokhodov & Burtsev, 2019), but with certain characteristics, such as embedding principle (Zhang et al., 2021) shows that the loss landscape of any network “contains” all critical points of all narrower networks, in the sense that, any critical point of any narrower network can be embedded to a critical point of the target network preserving its output function.
It is impractical to visualize the loss landscape in the high-dimensional space.
To compare two minima, θ and θ (cid:48) , in a loss landscape, one simple way (Keskar et al., 2016) is to visualize 1-d cross-section of the interpolation between θ and θ (cid:48) .
However, empirical studies (Li et al., 2017) show that this simple way could be misleading when θ and θ (cid:48) have difference of orders of magnitude.
Li et al.
(2017) visualizes loss functions using ﬁlter-wise normalized directions to remove the scaling effect.
We adopt this method (Li et al., 2017) in this work as follows.
To obtain a direction for a network with parameters θ , we begin by producing a random Gaussian direction vector d with dimensions compatible with θ .
Then, we normalize each ﬁlter in d to have the same norm of the corresponding ﬁlter in θ .
In other words, we make the replacement d
i,j ← d i,j (cid:107) d i,j (cid:107) (cid:107) θ i,j (cid:107) , where d i,j represents the j th ﬁlter (not the j th weight, for FNN, one layer is one ﬁlter.)
of the i th layer of d , and (cid:107) · (cid:107) denotes the Frobenius norm.
We use f ( α ) =
L ( θ + α d ) to characterize the loss landscape around the minima obtained with dropout layers θ ∗ d and without dropout layer θ ∗ .
For both network structures shown in Fig.
1, dropout can improve the generalization of the network and ﬁnd a ﬂatter minimum.
I NVERSE VARIANCE - FLATNESS RELATION
Similar to SGD, the effect of dropout can be equivalent to imposing a speciﬁc noise on the gradient.
A random noise, such as isotropic noise, can help the training escape local minima, but can not robustly improve generalization (An, 1996; Zhu et al., 2018).
The noise induced by the dropout should have certain properties that can lead the training to good minima.
In this section, we show that the noise induced by the dropout satisﬁes the variance principle, that is, the noise variance is larger along the sharper direction of the loss landscape at a minimum. The
[Figure] landscape-dependent structure helps the training escape sharp minima.
We utilize three methods to examine the variance principle for dropout, as summarized in Table
6. I NTERVAL FLATNESS VS . VARIANCE
We use the principal component analysis (PCA) to study the weight variations when the accuracy is nearly 100% .
For FNNs, networks are trained on MNIST with the ﬁrst 10000 examples as the training set for computational efﬁciency.
For ResNets, networks are trained on CIFAR-100 with 50000 examples as the training set to make a more convincing conclusion on modern datasets and models.
The networks are trained with full batch for different learning rates and dropout rates under the same random seed (that is, with the same initialization parameters).
After the loss is small enough, we sample the parameters or gradients of parameters N times ( N = 3000 in this experiment) and use the method introduced in Sec.
3.3 to construct covariance matrix Σ by the weights S para or gradients S grad between two hidden layers.
The PCA was done for the covariance matrix Σ .
We then compute the interval ﬂatness of the loss function landscape at eigen-directions, i.e., { F v i (Σ) } N i =1 .
Note that the PCA spectrum { λ i (Σ) } N i =1 indicate the variance of weights S para or gradients S grad at corresponding eigen-directions.
As shown in Fig.
2,3, for different learning rates and dropout rates, there is an inverse relationship between the interval ﬂatness of the loss function landscape { F v i (Σ) } Ni =1 and the dropout variance, i.e., the PCA spectrum
{ λ i (Σ) } Ni =1 .
We can approximately see a power-law relationship between { F v i (Σ) } Ni =1 and { λ i (Σ) } Ni =1 for different dropout rates and learning rates.
More detailed, for small ﬂatness part, the variance of noise induced by dropout is generally large, which indicates that the noise induced by dropout has larger variance in sharp directions, for large ﬂatness part, as the loss landscape ﬂatter, the linear relationship more obvious, we can see a clearer asymptotic behavior in the results.
Overall, we can observe the negative correlation between the eigenvalues and ﬂatness in
Fig.
6. P ROJECTED VARIANCE VS . H ESSIAN FLATNESS
The eigenvalues of the Hessian of the loss at a minimum are also often used to indicate the ﬂatness.
A large eigenvalue correspond to a sharper direction.
In this section, we study the relationship between eigen-values of the Hessian H of parameters θ at the end point of training and the variances of dropout at corresponding eigen-directions.
As mentioned in the Preliminary section, we sample [Figure] [Figure] the parameters or gradients of parameters 1000 times, that is N = 1000 .
For each eigen-direction v i of Hessian H , we project the sampled parameters or the gradients of sampled parameter to direction v i by inner product, denoted by Proj v i ( S ) .
Then, we compute the variance of the projected data, i.e., Var(Proj v i ( S )) .
As shown in Fig.
4, we ﬁnd that there is also a power-law relationship between { λ i ( H ) } Di =1 and { Var(Proj v i ( S )) } Di =1 for different dropout rates and learning rates, no matter if S is sampled from parameters or gradients of parameters.
The positive correlation between the eigenvalue and the projection variance show the structure of the dropout noise, which helps the network escape the bad minima.
6. A LIGNMENT BETWEEN H ESSIAN AND GRADIENT COVARIANCE
Zhu et al.
(2018) show that the alignment indicator
Tr( H Σ) plays an crucial role for stochastic processes escaping from minima, where H is the Hessian and Σ is the noise covariance.
In this subsection, we study the alignment between the Hessian and the random gradient covariance at each training step.
Note that the training is performed by GD without dropout.
At step i , we sample the gradients of parameters { g ji } Nj =1 by tentatively adding a dropout layer between the hidden layers.
For each step i , we the compute
Tr( H i Σ i ) , where H i is the Hessian of the loss at the parameter set at step i and Σ
i is the covariance of { g j i } N j =1 .
[Figure] [Figure] In order to show the anisotropic structure, we construct the isotropic noise for comparison, i.e., ¯ Σ i = TrΣ
i D I of the covariance matrix Σ i , where D is the number of parameters.
In our experiments, D = 2500 .
As shown in Fig.
6 , in the whole training process under different learning rates and dropout rates,
Tr( H i Σ i ) is much larger than Tr( H i ¯Σ i ) , indicating the anisotropic structure of dropout noise and its high alignment with the Hessian matrix.
C ONCLUSION AND DISCUSSION
In this work, we propose a variance principle that noise is larger at the sharper direction of the loss landscape.
If a noise satisﬁes the variance principle, it helps the training select ﬂatter minima and leads the training to good generalization.
We empirical show that dropout ﬁnds ﬂat minima during the training and examine that the dropout satisﬁes the variance principle to explain why dropout ﬁnds ﬂat minima by the following three perspectives: interval ﬂatness vs. variance, projected variance vs.
Hessian ﬂatness and alignment between Hessian and gradient covariance.
The dropout and the SGD are common in sharing the variance principle.
Based on this, there could be a general theory framework that can model both dropout and SGD to understand why they have better generalization.
The research on understanding the good generalization of dropout and SGD [Figure] lr [Figure]
lr [Figure] is far from complete. As a starting point, the variance principle shows a promising and reasonable direction for understanding the stochastic training of neural networks.