J UMPOUT : I MPROVED D ROPOUT FOR D EEP N EURAL N ETWORKS WITH R ECTIFIED L INEAR U NITS
Anonymous authors
Paper under double-blind review
A BSTRACT
Dropout is a simple yet effective technique to improve the generalization performance and prevent overﬁtting in deep neural networks (DNNs).
In this paper, we discuss three novel observations about dropout to better understand the generalization of DNNs with rectiﬁed linear unit (ReLU) activations: 1) dropout is a smoothing technique that encourages each local linear model of a DNN to be trained on data points from nearby regions; 2) a constant dropout rate can result in effective neural-deactivation rates that are signiﬁcantly different for layers with different fractions of activated neurons ; and 3) the rescaling factor of dropout causes an inconsistency to occur between the normalization during training and testing conditions when batch normalization is also used.
The above leads to three simple but nontrivial improvements to dropout resulting in our proposed method “Jumpout.”
Jumpout samples the dropout rate using a monotone decreasing distribution (such as the right part of a truncated Gaussian), so the local linear model at each data point is trained, with high probability, to work better for data points from nearby than from more distant regions.
Instead of tuning a dropout rate for each layer and applying it to all samples, jumpout moreover adaptively normalizes the dropout rate at each layer and every training sample/batch, so the effective dropout rate applied to the activated neurons are kept the same.
Moreover, we rescale the outputs of jumpout for a better trade-off that keeps both the variance and mean of neurons more consistent between training and test phases, which mitigates the incompatibility between dropout and batch normalization.
Compared to the original dropout, jumpout shows signiﬁcantly improved performance on CIFAR10,
CIFAR100, Fashion-MNIST, STL10, SVHN, ImageNet-1k, etc., while introducing negligible additional memory and computation costs.
I NTRODUCTION
Deep learning has achieved remarkable success on a variety of machine learning tasks (Russakovsky et al., 2015; Rajpurkar et al., 2016).
Deep neural networks (DNN), however, are often able to ﬁt the training data perfectly — this can result in the overﬁtting problem, thereby weakening the generalization performance on unseen data.
Dropout (Srivastava et al., 2014; Huang et al., 2016) is a simple yet effective technique to mitigate such problems by randomly setting the activations of hidden neurons to 0 , a strategy that reduces co-adaptation amongst neurons.
Dropout applies to any layer in a DNN without causing signiﬁcant additional computational overhead.
Dropout, however, has several drawbacks.
Firstly, dropout rates, constituting extra hyper-parameters at each layer, need to be tuned to get optimal performance.
Too high a dropout rate can slow the convergence rate of the model, and often hurt ﬁnal performance.
Too low a rate yields few or no improvements on generalization performance.
Ideally, dropout rates should be tuned separately for each layer and also during various training stages.
In practice, to reduce computation, we often tune a single dropout rate and keep it constant for all dropout layers and throughout the training process.
If we treat dropout as a type of perturbation on each training sample, it acts to generalize the DNN to noisy samples having that speciﬁc expected amount of perturbation (due to the ﬁxed dropout rate) with high probability.
The ﬁxed rate rules out samples typical having less perturbation, i.e., those potentially more likely to be closer to the original samples and thus that are potentially more helpful to improve generalization.
Also, when a constant dropout rate is applied to layers and samples having different fractions of activated neurons, the effective dropout rate (i.e., the proportion of the activated neurons that are deactivated by dropout) varies, which might result in too much perturbation for some layers and samples and too little perturbation for others.
Another deﬁciency of dropout lies in its incompatibility with batch normalization (BN) (Ioffe &
Szegedy, 2015) (more empirical evidence of this is shown in Section 3.3).
As dropout randomly shuts down activated neurons, it needs to rescale the undropped neurons to match the original overall activation gain of the layer.
Unfortunately, such rescaling breaks the consistency of the normalization parameters required between training and test phases 1 and may cause poor behavior when used with
BN.
Since BN, and its variants (Ba et al., 2016; Ulyanov et al., 2016; Wu & He, 2018), has become an almost indispensable component of modern DNN architectures to keep the training stable and to accelerate convergence, dropout itself often gets dropped out in the choice between these two non-complementary options and has recently become less popular.
1. O UR A PPROACH
We propose three simple modiﬁcations to dropout in order to overcome the drawbacks mentioned above.
These modiﬁcations lead to an improved version of dropout we call “jumpout.”
Our approach is motivated by three observations about how dropout results in improved generalization performance for DNNs with rectiﬁed linear unit (ReLU) activations, which covers a frequently used class of DNNs.
Firstly, we note that any DNN with ReLU is a piecewise linear function which applies different linear models to data points from different polyhedra deﬁned by the ReLU activation patterns.
Based on this observation, applying dropout to a training sample randomly changes its ReLU activation patterns and hence the underlying polyhedral structure and corresponding linear models.
This means that each linear model is trained not only to produce correct predictions for data points in its associated polyhedron, but also is trained to work for data points in nearby polyhedra; what precisely “nearby” means depends on the dropout rate used.
This partially explains why dropout improves generalization performance.
The problem, however, is that with a ﬁxed dropout rate, say p , and on a layer with n units, the typical number of units dropped out is np as that is the mode of a Binomial distribution with parameter p .
It is relatively rare that either very few (closer to zero) or very many (closer to n ) units are dropped out.
Thus, with high probability, each linear model is smoothed to work on data points in polyhedra at a typical distance away.
The probability of smoothing over closer distances is potentially much smaller, thus not achieving the goal of local smoothness.
In jumpout, by contrast, p rather than being ﬁxed is itself a random variable; we sample p from a distribution that is monotone decreasing (e.g., a truncated half-Gaussian).
This achieves the property that Pr ( i units dropping out ) ≥ Pr ( i+1 units dropping out ) for all i ∈ { 1 , 2 , . . . , n } .
That is, a smaller dropout rate has a higher probability of being chosen.
Hence, the probability of smoothing polyhedra to other points decreases as the points move farther away.
Secondly, we notice that in dropout, the fraction of activated neurons in different layers, for different samples and different training stages, can be different.
Although we are using the same dropout rate, since dropping out neurons that are already quiescent by ReLU changes nothing, the effective dropout rate, i.e., the fraction of the activated neurons that are dropped, can vary signiﬁcantly.
In jumpout, we adaptively normalize the dropout rate for each layer and each training sample/batch, so the effective neural-deactivation rate applied to the activated neurons are consistent over different layers and different samples as training proceeds.
Lastly, we address the incompatibility problem between dropout and BN by rescaling the outputs of jumpout in order to keep the variance unchanged after the process of neural deactivation.
Therefore, the BN layers learned in the training phase can be directly applied in the test phase without an inconsistency, and we can reap the beneﬁts of both dropout and BN when training a DNN.
In our implementation, similar to dropout, jumpout also randomly generates a 0/1 mask over the hidden neurons to drop activations.
It does not require any extra training, can be easily implemented and incorporated into existing architectures with only a minor modiﬁcation to dropout code.
In our experiments on a broad range of benchmark datasets including CIFAR10, CIFAR100, Fashion-
MNIST, SVHN, STL10 and ImageNet-1k, jumpout shows almost the same memory and computation costs as the original dropout, but signiﬁcantly and consistently outperforms dropout on a variety of tasks, as we show below.
1. R ELATED W ORK
Jumpout is not the ﬁrst approach to address the ﬁxed dropout rate problem.
Indeed, recent work has proposed different methods to generate adaptive dropout rates.
Ba & Frey (2013) proposed “standout” to adaptively change the dropout rates for various layers and training stages.
They utilized a binary belief network and trained it together with the original network to control the dropout rates.
Zhuo et al.
(2015) further extend the model so that adaptive dropout rates can be learned for different neurons or group of neurons.
Zhai & Wang (2018) showed that the Rademacher complexity of a DNN is bounded by a function related to the dropout rate vectors, and they proposed to adaptively change dropout rates according to the Rademacher complexity of the network.
In contrast to the above methods, jumpout does not rely on additional trained models: it adjusts the dropout rate solely based on the ReLU activation patterns.
Moreover, jumpout introduces negligible computation and memory overhead relative to the original dropout methods, and can be easily incorporated into existing model architectures.
Wang & Manning (2013) showed that dropout has a Gaussian approximation called Gaussian dropout and proposed to optimize the Gaussian dropout directly to achieve faster convergence.
The Gaussian dropout was also extended and studied from the perspective of variational methods.
Kingma et al.
(2015) generalized Gaussian dropout and proposed variational dropout, where they connected the global uncertainty with the dropout rates so that dropout rates can be adaptive for every neuron.
Molchanov et al.
(2017) further extended variational dropout to reduce the variance of the gradient estimator and achieved sparse dropout rates.
Other recent variants of dropout include Swapout (Singh et al., 2016), which combines dropout with random skipping connections to generalize to different neural network architectures, and Fraternal Dropout (Zolna et al., 2018), which trains two identical
DNNs using different dropout masks to produce the same outputs and tries to shrink the gap between the training and test phases of dropout.
In this paper, we focus on changes to the original dropout that do not require any extra training/optimization costs or introduce more parameters to learn.
Jumpout involves orthogonal and synergistic contributions to most of the above methods, and targets different problems of dropout.
Indeed, jumpout can be applied along with most other previous variants of dropout.
R E LU DNN S ARE COMPRISED OF LOCAL LINEAR MODELS
We study a feed-forward deep neural networks of the form:
[Equation]
where W j is the weight matrix for layer j , ψ j is the corresponding activation function (ReLU in this paper), x ∈ X is an input data point of d in dimensions and ˆ y ( x ) is the network’s output prediction of d out dimensions, e.g., the logits before applying softmax.
We denote the hidden nodes on layer j to be h j , i.e., h j = W j ψ j − 1 ( W j − 1 ψ j − 2 ( . . .
ψ 1 ( W 1 x ))) , whose dimensionality is d j ; they represent the nodes after applying the activation function as ¯ h j = ψ ( h j ) .
The above DNN formalization can generalize many DNN architectures used in practice.
Clearly,
Eqn.
(1) can represent a fully-connected network of m layers.
Note Eqn.
(1) covers the DNNs with bias terms at each layer since the bias terms can be written in the matrix multiplication as well by introducing dummy dimensions on the input data (append m 1’s to input data).
Moreover, the convolution operator is essentially a matrix multiplication, where every row of the matrix corresponds to applying a convolutional ﬁlter on a certain part of the input, and therefore the resulting weight matrix is very sparse and has tied parameters, and typically has an enormous (compared to input size) number of rows.
The average-pooling is a linear operator and therefore representable as a matrix multiplication, and max-pooling can be treated as an activation function.
Finally, we can represent the residual network block by appending an identity matrix at the bottom of a weight matrix so that we can retain the input values, and add the retained input values later through another matrix operation.
Therefore, we can also write a DNN with short-cut connections in the form of Eqn.
For piecewise linear activation functions such as ReLU, the DNN in Eqn.
(1) can be written as a piecewise linear function, i.e., the DNN in a region surrounding a given data point x is a linear model having the following form:
[Equation]
where W xj is the equivalent weight matrix after combining the resultant activation pattern with W j .
For instance, suppose we use ReLU activation ψ ReLU ( z ) = max(0 , z ) ; at every layer, we have an activation pattern for the input a j ( x ) ∈ { 0 , 1 } d j , and a j ( x )
[ p ] = 0 indicates that ReLU sets the unit p to 0 or otherwise preserves the unit value.
Then, ψ ReLU ( W 1 x ) =
W x 1 x , where W x 1 is modiﬁed from W 1 by setting the rows, whose corresponding activation patterns are 0 , to be all zero vectors.
We can continue such a process to the deeper layers, and in the end we can eliminate all the ReLU functions and produce a linear model as shown in Eqn.
In addition, the gradient ∂
ˆ y ( x ) ∂x is the weight vector of the linear model.
Note that the linear model in Eqn. 2 is speciﬁcally associated with the activation patterns { a j ( x ) } mj =1 on all layers for a data input x , which is equal to a set of linear constraints that deﬁnes a convex polyhedron containing x .
In a DNN with ReLU activations, for every dimension i in layer 1, if a 1 ( x )[ i ] = 1 , we have a linear equation W
[ i ] x > 0 and otherwise we have W
[ i ] x ≤ 0 .
As a result, we have d 1 linear constraints for layer 1.
Similarly, we can follow the same procedure on layer j with input changed to W x j − 1 . . .
W x 1 x , so we have d j linear constraints for layer j .
Therefore, a DNN with piecewise linear activation functions is a piecewise linear function deﬁned by a number of local linear models (on a set of input data points) and the corresponding convex polyhedra, each represented by a set of linear constraints ( (cid:80) mj
=
d j constraints in speciﬁc).
An analysis based on a similar perspective can be found in Raghu et al.
Although the above analysis can be easily extended to DNNs with general piecewise linear activation functions, we focus on DNNs with ReLU activations in the rest of the paper for clarity.
In addition to the piecewise linear property, ReLU units are cheap to compute, as is their gradient, and are widely applicable to many different tasks while achieving good performance
(He et al., 2016a; Zagoruyko & Komodakis, 2016).
In the following, we will study how dropout improves the generalization performance of a complicated DNN by considering how it generalizes each local linear model to its nearby convex polyhedra.
This is easier to analyze and acts as the inspiration for our modiﬁcations to the original dropout.
We will further elaborate the understandings of dropout based on the above insights of local linear models in the next section.
T HREE M ODIFICATIONS TO D ROPOUT LEAD TO J UMPOUT 3. M ODIFICATION I: M ONOTONE D ROPOUT R ATE FOR L OCAL S MOOTHNESS
There have been multiple explanations for how dropout improves the performance of DNNs.
Firstly, dropout prevents the co-adaptation of the neurons in the network, or in other words, encourages the independence or diversity amongst the neurons.
Secondly, by randomly dropping a portion of neurons during training, we effectively train a large number of smaller networks, and during test/inference, the network prediction can be treated as an ensemble of the outputs from those smaller networks, and thus enjoys the advantages of using an ensemble such as variance reduction.
Here we provide another perspective for understanding how dropout improves generalization performance by inspecting how it smooths each local linear model described in the previous section.
As mentioned above, for a DNN with ReLUs, the input space is divided into convex polyhedra, and for any data point in every convex polyhedron of the ﬁnal layer (a polyhedron that is not divided further into smaller regions), the DNN behaves exactly as a linear model.
For large DNNs with thousands of neurons per layer, the number of convex polyhedra can be exponential in the number of neurons.
Hence, there is a high chance that the training samples will be dispersedly situated amongst the different polyhedra, and every training data point is likely to be given its own distinct local linear model.
Moreover, it is possible that two nearby polyhedra may correspond to arbitrarily different linear models, since they are the results of consecutively multiplying a series of weight matrices
W m W xm − 1 . . .
W x 1 of different x (as shown in Eqn.
(2)), where each weight matrix W xj is W j with some rows setting to be all-zero according to the activation pattern a j ( x ) of a speciﬁc data point x .
If the activation patterns of two polyhedra differ on some critical rows of the weight matrices, the resulting linear models may differ a lot.
Therefore, it is possible that the linear model of one polyhedron can only work for one or a few training data points strictly within the polyhedron, and may fail when applied to any nearby test data point (i.e., a lack of smoothness).
This might make
DNN fragile and perform unstably on new data, and thus weaken its generalization ability.
Given the problems of dropout mentioned in Section 1.1, we propose to sample a dropout rate from a truncated half-normal distribution (to get a positive value), which is the positive part of an ordinary
Gaussian distribution with mean zero.
In particular, we ﬁrstly sample p ∼ N (0 , σ ) from a Gaussian distribution, and then take the absolute value | p | as the dropout rate.
We further truncate | p | so that | p | ∈ [ p min , p max ] , where 0 ≤ p min < p max ≤ 1 .
These determine the lower and upper limits of the dropout rate and are used to ensure that the sampled probability does not get either too small, which makes jumpout ineffective, or too large, which may yield poor performance.
Overall, this achieves a monotone decreasing probability of a given dropout rate as mentioned above.
Other distributions (such as a Beta distribution) could also be used for this purpose, but we leave that to future work.
We utilize the standard deviation σ as the hyper-parameter to control the amount of generalization enforcement.
By using the above method, smaller dropout rates are sampled with higher probabilities so that a training sample will be more likely to contribute to the linear models of closer polyhedra.
Therefore, such a Gaussian-based dropout rate distribution encourages the smoothness of the generalization performance of each local linear model, i.e., it will still perform well on points in closer polyhedra, but its effectiveness will diminish for a point farther away from the polyhedron it belongs to.
3. M ODIFICATION II: D ROPOUT R ATE ADAPTED TO THE NUMBER OF A CTIVATED N EURONS
The dropout rate for each layer is a hyper-parameter, and as stated above, it controls a form of smoothness amongst nearby local linear models.
Ideally, the dropout rates of different layers should be tuned separately to improve network performance.
In practice, it is computationally expensive or infeasible to tune so many hyper-parameters.
One widely adopted approach is therefore to set the same drop rate for all layers and to tune one global dropout rate.
R e L U
A c t i v a t i o n
P o r t i o n
Epoch convconvconvfc
Using a single global dropout rate is suboptimal because the proportion of active neurons (i.e., neurons with positive values) of each layer at each training stage and for each sample can be dramatically different (see Fig. 1).
When applying the same dropout rate to different layers, different fractions of active neurons get deactivated, so the effective dropout rate applied to the active neurons varies signiﬁcantly.
Suppose the fraction of active neurons in layer j is q + j = ( (cid:80) i =1: d 1 h
j [ i ] > 0 ) /
|
h j | .
Since dropping the inactive neurons has no effects (neurons with values ≤ 0 have already been set to 0 by ReLU), the effective dropout rate of every layer is p
j q + j , where p j is the dropout rate of layer j .
Thus, to better control the behavior of dropout for different layers and across various training stages, we normalize the dropout rate by q + j and use an actual dropout rate of p (cid:48)
j = p j /q + j .
By doing so, the hamming distance between the changed activation pattern and the original pattern is more consistent, and we can more precisely achieve the desirable level of smoothing encouragement by tuning the dropout rate as a single hyper-parameter.
3. M ODIFICATION III: R ESCALE O UTPUTS TO WORK WITH B ATCH N ORMALIZATION
In standard dropout, if the dropout rate is p , we scale the neurons by 1 /p during training and keeps the neuron values unchanged during the test/inference phase.
The scaling factor 1 /p keeps the mean of the neurons the same between training and test; this constitutes a primary reason for the incompatibility between dropout and batch normalization (BN) (Ioffe & Szegedy, 2015).
Speciﬁcally, though the mean of neurons is consistent, the variance can be dramatically different between the training and test phases, in which case the DNN might have unpredictable behavior as the BN layers cannot adapt to the change of variance from training to test condition.
We consider one possible setting of combining dropout layers with BN layers where one linear computational layer (e.g., a fully-connected or a convolutional layer without activation function) is followed by a BN layer, then a ReLU activation layer, and then followed by a dropout layer.
For layer j , without loss of generality, we may treat the value of a neuron i after ReLU, i.e., ¯ h j [ i ] as a random variable with q + j probability of being 1 and 1 − q + j probability of being 0.
If dropout is not applied, ¯ h
j
[ i ] then gets multiplied by certain entry in the weight matrix W j +
[ i (cid:48) , i ] , and contributes to the value of the i (cid:48) neuron of layer j +1 .
Since we consider any index i and i (cid:48) , we rename the following terms for simplicity: x j := ¯ h j , w j := W j +
[ i (cid:48) , i ] , y j :=
h
j +
[ i (cid:48) ] .
As neuron i (cid:48) of layer j + 1 (before ReLU) then gets fed into a BN layer, we will focus on the change of mean and variance as we add the dropout layer.
Suppose we apply a dropout rate of p j , then
[Equation]
Hence, dropout changes both the scales of the mean and variance of neurons during training.
Since the following BN’s parameters are trained based on the scaled mean and variance, which however are not scaled by dropout during test/inference (because dropout is not used during testing), the trained
BN is not consistent with the test phase.
An easy ﬁx of the inconsistency is to rescale the output y j to counteract dropout’s on the scales of mean and variance.
In order to recover the original scale of the mean, we should rescale the dropped neurons by (1 − p j ) − 1 .
However, the rescaling factor should be (1 − p j ) − 0 .
5 instead for recovering the scale of the variance if E ( y j ) is small and thus the second term of the variance can be ignored.
[Figure]
Ideally, we can also take into account the value of
E [ w j ]
[Figure]
and scale the un-dropped nuerons by (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
However, computing information about w j , which is the weight matrix of the following layer, requires additional computation and memory cost.
In addition, such a scaling factor is only correct for the variance of y j .
To make the mean consistent, we should instead use (1 − p j ) − 1 (the original dropout scaling factor).
No simple scaling method can resolve the shift in both mean and variance, as the mean rescaling (1 − p j ) −does not solve the variance shift.
[Figure]
n R a t i o
Epoch (1 -p)^(-1) (1 -p)^(-0.5) (1 -p)^(-0.75) 0.8 0.
V a r i a n c e R a t i o
Epoch (1 -p)^(-1) (1 -p)^(-0.5) (1 -p)^(-0.75)
When the mean E ( y j ) is large in magnitude, so that the second term in the variance is comparable with the ﬁrst term, in which case the variance is small, we should use the rescaling factor close to (1 − p j ) − 1 , which makes the mean exactly unchanged for training and test.
In contrast, when the mean
E ( y j ) is small in magnitude and close to 0 , the second term in the variance is ignorable, and we should use (1 − p j ) − 0 .
5 as the rescaling factor, to make the variance unchanged.
In practice, it is not efﬁcient to compute E ( y j ) during training, so we propose to use a trade-off point (1 − p j ) − 0 .
75 between (1 − p j ) − 1 and (1 − p j ) − 0 .
In Figure 3, we show that (1 − p j ) − 0 .
75 makes both the mean and variance sufﬁciently consistent for the cases of using dropout and not using dropout.
In Figure 2, we compare the performance of the original dropout and dropout using our rescaling factor (1 − p j ) − 0 .
75 , when they input : h j , σ , p max , p min 1 q + j :=
( (cid:80) i =1: d j 1 h j [ i ] > 0 ) / |
h j | ; // Compute the fraction of activated neurons 2 p ∼ N (0 , σ ) , p j := min( p min + | p | , p max ) ; //
Sample a Gaussian dropout rate 3 p (cid:48)
j :=
p j /q + j ; // Normalize the dropout rate according to the fraction of activated neurons 4 Randomly generate a 0/1 mask z j for h j , with probability p (cid:48)
j to be 0; //
Sample the dropout mask 5 s j :=
(1 − p (cid:48) ) − 0 .
75 ; // Compute the rescaling factor 6 h (cid:48)
j :=
s j ∗ diag ( z j )
h j ; //
Rescale the outputs 7 return h (cid:48)
j are used with and without BN in a convolutional networks.
It shows that using dropout with BN can potentially improve the performance, and larger dropout might result in more improvement.
However, using the original dropout with BN leads to a signiﬁcant decrease in the accuracy once increasing the dropout rate over 0 .
In contrast, the performance of dropout using our rescaling keeps improving with increasing dropout rate (until reaching 0 . 25 ), and is the best among the four conﬁgurations.
3. J UMPOUT L AYER
We combine the three modiﬁcations speciﬁcally designed to overcome the drawbacks of the original dropout in our proposed improved dropout, which we call “Jumpout” as shown in Alg. 1.
Similar to the original dropout, jumpout essentially generates a 0/1 mask for the input neurons, and randomly drop a portion of the neurons based on the mask.
Summarizing the novelty of jumpout, instead of using a ﬁxed dropout rate as in the original dropout, jumpout samples from a monotone decreasing distribution as mentioned above to get a random dropout rate.
Also, jumpout normalizes the dropout rate adaptively based on the number of active neurons, which enforces consistent regularization and generalization effects on different layers, across different training stages, and on different samples.
Finally, jumpout further scales the outputs by (1 − p ) − 0 .
75 , as opposed to (1 − p ) − 1 during training, in order to trade-off the mean and variance shifts and synergize well with batchnorm operations.
Jumpout requires one main hyper-parameter σ to control the standard deviation of the half-normal distribution, and two auxiliary truncation hyperparameters ( p min , p max ) .
Though ( p min , p max ) can also be tunned, they serve to bound the samples from the half-normal distribution; in practice, we set p min = 0 .
01 and p max = 0 .
6 , which work consistently well over all datasets and models we tried.
Hence, jumpout has three hyperparameters, although we only tuned σ and achieved good performance, as can be seen below.
Also, note that here we consider the input h j to be the features of layer j corresponding to one data point.
For a mini-batch of data points, we can either estimate q + j separately for each single data point in the mini-batch or apply the average q + j over data points as the estimate for the mini-batch.
In practice, we utilize the latter option as we ﬁnd that it gives comparable performance to the ﬁrst while using less computation and memory.
Jumpout has almost the same memory cost as the original dropout, which is the additional 0/ drop mask.
For computation, jumpout requires counting the number of active neurons, which is insigniﬁcant compared to the other layers of a deep model, and sampling from the distribution, which is also insigniﬁcant compared to the other computation in DNN training.
E XPERIMENTS
In this section, we apply dropout and jumpout to different popular DNN architectures and compare their performance on six benchmark datasets at different scales.
In particular, these DNN architectures include a small CNN with four convolutional layers 2 applied to CIFAR10 (Krizhevsky & Hinton, 2009), WideResNet-28-10 (Zagoruyko & Komodakis, 2016) applied to CIFAR10 and
CIFAR100 (Krizhevsky & Hinton, 2009), “pre-activation” version of ResNet-20 (He et al., 2016b) applied to Fashion-MNIST (“Fashion” in all tables) (Xiao et al., 2017), WideResNet-16-8 applied to SVHN (Netzer et al., 2011) and STL10 (Coates et al., 2011), and ResNet-18 (He et al., 2016a) applied to ImageNet (Deng et al., 2009; Russakovsky et al., 2015).
The information about the all the datasets can be found in Table 4 at Appendix. [Table] Dropout
[Table] [Table] [Table] For all the experiments about CIFAR and Fashion-MNIST, we follow the standard settings, data preprocessing/augmentation, and hyperparameters used in an existing GitHub repository 3 .
On ImageNet, we starts from a pre-trained ResNet18 model 4 , and train two copies of it with dropout and jumpout respectively for the same number of epochs.
The reason for not starting from random initialized model weights is that training DNNs on ImageNet usually does not have overﬁtting problem if one follows the standard data augmentation methods used to train most modern models, but both dropout and jumpout are most effective in the case of overﬁtting.
Therefore, we choose to start from the pre-trained model, on which training accuracy is relatively high (but still not overﬁt and very close to the test accuracy) 5 . [Table]
We summarize the experimental results in Table 2 which shows that jumpout consistently outperforms dropout on all datasets and all the DNNs we tested.
Moreover, for Fashion-MNIST and CIFAR on which the test accuracy is already > 95% , jumpout can still bring appreciable improvements.
In addition, on CIFAR100 and ImageNet (on which a great number of DNNs and training methods are heavily tunned), jumpout achieves the improvement that can only be obtained by signiﬁcantly process, is be increasing the model size in the past.
These verify the effectiveness of jumpout and its advantage comparing to the original dropout.
In addition, we conduct a thorough ablation study of all the possible combinations of the three proposed modiﬁcations, with results reported in Table 1. It further veriﬁes the effectiveness of each modiﬁcation: 1) each modiﬁcation improves the vanilla dropout; 2) adding any modiﬁcation to another brings further improvements; and 3) applying the three modiﬁcations together (i.e., jumpout) achieves the best performance.
We also provide the learning curves and convergence plots of dropout and jumpout equipped DNNs during training in Figure 4.
In all the ﬁgures, “Jumpout” applies adaptive dropout rate per minibatch.
Jumpout exhibits substantial advantages over dropout in early learning stages, and reaches a reasonably good accuracy much faster.
In the future, it may be possible to ﬁnd a better learning rate schedule method speciﬁcally for jumpout, so it can reach the ﬁnal performance earlier than dropout  A PPENDIX M e a n R a t i o
Epoch (1 -p)^(-1) (1 -p)^(-0.5) (1 -p)^(-0.75) 0.
V a r i a n c e R a t i o
Epoch (1 -p)^(-1) (1 -p)^(-0.5) (1 -p)^(-0.75) (a) p = 0 .
[Table]
[Table]