S HAKE D ROP REGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes a powerful regularization method named ShakeDrop regularization .
ShakeDrop is inspired by Shake-Shake regularization that decreases error rates by disturbing learning.
While Shake-Shake can be applied to only ResNeXt which has multiple branches, ShakeDrop can be applied to not only ResNeXt but also ResNet, Wide ResNet and PyramidNet in a memory efﬁcient way.
Important and interesting feature of ShakeDrop is that it strongly disturbs learning by multiplying even a negative factor to the output of a convolutional layer in the forward training pass. The effectiveness of ShakeDrop is conﬁrmed by experiments on
CIFAR-10/100 and Tiny ImageNet datasets.
I NTRODUCTION
Recent advances in generic object recognition have been brought by deep neural networks.
After
ResNet (He et al., 2016) opened the door to very deep CNNs of over a hundred layers by introducing the residual block, its improvements such as Wide ResNet (Zagoruyko & Komodakis, 2016),
PyramdNet (Han et al., 2017a;b) and ResNeXt (Xie et al., 2017) have broken the records of lowest error rates.
On the other hand, in learning, they often suffer from problems such as vanishing gradients. Hence, regularization methods help to learn and boost the performance of such base network architectures.
Stochastic Depth (ResDrop) (Huang et al., 2016) and Shake-Shake (Gastaldi, 2017) are known to be effective regularization methods for ResNet and its improvements. Among them, Shake-Shake applied to ResNeXt is the one achieving the lowest error rates on CIFAR-10/ datasets (Gastaldi, 2017).
Shake-Shake, however, has following two drawbacks.
(1) Shake-Shake can be applied to only multibranch architectures (i.e. ResNeXt).
(2) Shake-Shake is not memory efﬁcient.
Both drawbacks come from the same root.
That is, Shake-Shake requires two branches of residual blocks to apply. If it is true, it is not difﬁcult to conceive its solution: a similar disturbance to Shake-Shake on a single residual block.
It is, however, not trivial to realize it.
The current paper addresses the problem of realizing a similar disturbance to Shake-Shake on a single residual block, and proposes a powerful regularization method, named ShakeDrop regularization .
While the proposed ShakeDrop is inspired by Shake-Shake, the mechanism of disturbing learning is completely different.
ShakeDrop disturbs learning more strongly by multiplying even a negative factor to the output of a convolutional layer in the forward training pass.
In addition, a different factor from the forward pass is multiplied in the backward training pass.
As a byproduct, however, learning process gets unstable.
Our solution to this problem is to stabilize the learning process by employing ResDrop in a different usage from the usual.
Based on experiments using various base network architectures, we reveal the condition that the proposed ShakeDrop successfully works.
E XISTING M ETHODS R EQUIRED TO I NTRODUCE THE P ROPOSED M ETHOD 2. D EEP N ETWORK ARCHITECTURES
ResNet (He et al., 2016) opened the door to very deep CNNs of over a hundred layers by introducing the residual block, given as
[Equation]
[Figure]
(a) ResNeXt (Xie et al., 2017), in which some process- ing layers omitted for conciseness.
[Figure]
[Figure] ) Shake [Figure] Shake (ResNeXt + Shake-Shake) (Gastaldi, 2017), in which some processing layers omitted for conciseness.
[Figure]
[Figure]
intermediate regularization method to the proposed ShakeDrop.
[Figure]
where x is the input of the residual block, G ( x ) is the output of the residual block and F ( x ) is the output of the residual branch on the residual block.
While ResNet realized deep networks, it is imperfect.
He et al.
(2016) pointed out that high error rates obtained by the 1202-layer ResNet on the CIFAR datasets (Krizhevsky, 2009) is caused by overﬁtting.
Veit et al.
(2016) experimentally showed that some residual blocks whose channels greatly increase cause high error rates.
PyramidNet (Han et al., 2017a;b) overcame the problem of ResNet by gradually increasing channels on each residual block.
It has almost same residual block as Eqn.
It successfully realized deep CNNs of up to 272 layers and achieved the lowest error rates among the vanilla residual networks on the CIFAR datasets.
2. N ON -D EEP NETWORK ARCHITECTURES
Wide ResNet (Zagoruyko & Komodakis, 2016) improved error rates by simply increasing channels of ResNet.
While it has almost same residual block as Eqn.
(1), it has wider and shallower architecture than ResNet.
Jastrzebski et al.
(2017) experimentally showed that Wide ResNet overcomes the problem of ResNet like PyramidNet.
ResNeXt (Xie et al., 2017) achieved the better error rates than Wide ResNet on almost same number of parameters.
The basic architecture of ResNeXt is given as
[Equation]
where F 1 ( x ) and F 2 ( x ) are the outputs of the residual branches as shown in Figure 1(a).
The number of residual branches is not limited to 2, and the number is the most important factor to control the result.
2. R EGULARIZATION METHODS
Stochastic Depth (Huang et al., 2016) is a regularization method which overcame problems of
ResNet such as vanishing gradients.
It makes the network apparently shallow in learning by dropping residual blocks stochastically selected.
On the l th residual block from input layer, the Stochastic Depth process is given as
[Equation]
where b l ∈ { 0 , 1 } is a Bernoulli random variable with the probability of p l .
Huang et al. (2016) tested two rules for determining p l : uniform rule and linear decay rule. The uniform rule using a constant for all p l did not work well. The linear decay rule which deﬁnes p l as
[Equation]
where L is the number of all layers and p L is the initial parameter, worked well.
It was originally proposed for improving ResNet.
It was also applied to PyramidNet; the combination of PyramidNet and Stochastic Depth was named PyramidDrop (Yamada et al., 2016).
Shake-Shake (Gastaldi, 2017) is a powerful regularization method for improving ResNeXt architectures.
Its architecture is almost same as Eqn.
(2) and is given as
[Equation]
where α is a random coefﬁcient given as α ∈ [0 , 1] .
As shown in Figure 1(b), calculation of the backward pass is disturbed by another random coefﬁcient β ∈ [0 , 1] instead of α .
As a result, Shake-Shake decreased error rates than ResNeXt.
P ROPOSED M ETHOD 3. I NTERPRETATION OF S HAKE -S HAKE REGULARIZATION AND ITS DRAWBACKS
We give an intuitive interpretation of the forward pass of Shake-Shake regularization.
To the best of our knowledge, it has not been given yet, while the phenomenon in the backward pass is experimentally investigated by Gastaldi (2017). As shown in Eqn.
(5) (and also in Fig. 1(b)), in the forward pass, Shake-Shake interpolates the outputs of two residual branches (i.e., F 1 ( x ) and F 2 ( x ) ) with a random variable α that controls the degree of interpolation. As DeVries & Taylor (2017a) demonstrated that interpolation of two data in the feature space can synthesize reasonable augmented data, the interpolation of two residual blocks of Shake-Shake in the forward pass can be interpreted as synthesizing data.
Use of a random variable α generates many different augmented data. On the other hand, in the backward pass, a different random variable β is used to disturb learning to make the network learnable long time.
Gastaldi (2017) demonstrated how the difference between α and β affects. The regularization mechanism of Shake-Shake relies on two or more residual branches, so that it can be applied only to 2-branch network architectures such as ResNeXt.
It also means that due to memory amount required to apply Shake-Shake, it is hard to make networks deeper like 1-branch network architectures such as ResNet and PyramidNet. In our estimation, memory consumption increased by 8% by changing a 1-branch network (e.g., ResNeXt 1-64d) to a corresponding 2-branch network (e.g., ResNeXt 2-40d) with keeping the number of learnable parameters. 3. S IMILAR REGULARIZATION TO S HAKE -S HAKE ON 1- BRANCH NETWORK ARCHITECTURES
In order to realize a similar regularization of Shake-Shake on 1-branch network architectures such as ResNet, Wide ResNet and PyramidNet, in the forward pass, we need a different mechanism from interpolation to synthesize augmented data in the feature space.
A known algorithm that generates augmented data in the feature space is virtual adversarial training (VAT) (Miyato et al., 2016). It generates virtual adversarial examples in the feature space, from the current example, in the direction that most severely damages class label prediction. However, since Shake-Shake does not intend to take into account the direction to generate a new example, following the way, we just apply random perturbation, using α , to the output of a residual branch (i.e., F ( x ) of Eqn.
Differently from
VAT, multiplication by α , a scalar value, does not change the direction but controls dissimilarity of generated data. One exception is the case α takes negative values (i.e., the opposite direction). In the backward pass, we can use the same way as Shake-Shake for 1-branch network architectures.
We call the regularization method mentioned above 1-branch Shake . It is given as
[Equation]
where α is a coefﬁcient that disturbs learning in the forward pass.
As shown in Figure 1(c), β is a coefﬁcient similar to the one of Shake-Shake and used in the backward pass.
The 1-branch Shake is expected to realize powerful generalization like Shake-Shake regularization.
However, by applying it to 110-layer PyramidNet, the result on the CIFAR-100 dataset, where α ∈ [0 , 1] and β ∈ [0 , 1] following Shake-Shake, was hopelessly bad (i.e., an error rate of 77.99%).
Failure of 1-branch Shake is caused by too strong perturbation. However, weakening the perturbation would also weaken the effect of regularization.
Thus, we need a trick to promote learning under strong perturbation.
Our idea is to use the mechanism of Stochastic Depth (ResDrop) for solving the issue. As is written in Sec.2.3, in the original form, ResDrop promotes learning by dropping some residual blocks.
In other words, it makes a network apparently shallow in learning.
In our situation, however, the original usage of ResDrop does not contribute much because a shallower network to which 1-branch Shake is applied would also suffer from strong perturbation.
Thus, we use the mechanism of ResDrop as a probabilistic switch of two network architectures: the original network (e.g., PyramidNet) and the one to which 1-branch Shake is applied (e.g., PyramidNet+ 1-branch Shake ). The following effects are expected by mixing up two networks.
If the original network is selected, learning is correctly promoted. If the network with strong perturbation is selected, learning is disturbed. To achieve a good performance, two networks should be in a good balance. The proposed ShakeDrop is given as
[Equation]
where b l is a Bernoulli random variable following the linear decay rule.
We call the new method “ShakeDrop.”
According to the value of b l , Eqn.
(7) is deformed as
[Equation]
In the case that the base network is PyramidNet, ShakeDrop is equivalent to PyramidNet if b l = 1 , and it is equivalent to PyramidNet + 1-branch Shake if b l = 0 .
We found that the linear decay rule of ResDrop (Eqn.
(4)) works well, while we also tested the uniform rule which did not work well.
As shown in Figure 1(d), similar to Shake-Shake, β is used in the backward pass instead of α .
It is noteworthy that regardless of the value of β , the weights of the network are updated.
For example, let us consider the case of β = 0 . In such a case, the weights of the residual blocks selected to apply 1-branch Shake are not updated. However, the inputs to their succeeding residual blocks are changed.
E XPERIMENTS
See Sec.
A for implementation details.
4. P RELIMINARY EXPERIMENTS : SEARCH FOR PARAMETER SETTINGS OF THE PROPOSED METHOD
The best parameter settings of the proposed method, ShapeDrop, are searched when it is applied to
PyramidNet. They include parameter ranges of α and β , and the timing these parameters are updated.
In the preliminary experiments, networks were trained on the CIFAR-100 dataset.
PyramidNet had 110 layers, which consisted of a convolution layer, 54 additive pyramidal residual blocks and a fully connected layer.
The number of channels at the last residual block was 286.
Tableshows representative parameter ranges of α and β we tested and their results. From the results, we can see how the combinations of α and β affect.
First of all, except PyramidDrop ( α = 0 , β = 0 ), only case F ( α ∈ [ − 1 , 1] , β = 0 ) and case H ( α ∈ [ − 1 , 1] , β ∈ [0 , 1] ) were better than the original
[Table] [Table] PyramidNet (case A: α = 1 , β = 1 ). Among them, case H was the best.
It is interesting to observe three cases E ( α ∈ [ − 1 , 1] , β = 1 ), F and H all of which take the same range of α . Since cases F and H were better than PyramidNet [Table] we conﬁrm strong perturbation is effective.
However, case E was even worse than PyramidNet. This implies that if strong perturbation is applied, the residual blocks selected to apply 1-branch Shake strongly suffer from the perturbation, so that the weights of the residual blocks should not be fully updated. On the other hand, the best was case H whose range of β was in-between cases E and F. This implies that the values of β should be in a good balance.
Next, the best strategy to update the parameters (referred as scaling coefﬁcient) among “Batch and “Pixel” are explored “Batch” the scaling
“Image” means means channel
“Pixel” means that the same scaling coefﬁcients are used for the each element for each residual block.
ShakeDrop was trained using the best parameters found in the experiment above, i.e, ranges α ∈ [ − 1 , 1] , β ∈ [0 , 1] .
Table 2 shows that “Pixel” level was the best. However, it required a lot of memory.
Hence, we selected “Image” level, the second best, considering memory efﬁciency.
Through the preliminary experiments, we found that the most effective parameters for α and β are α ∈ [ − 1 , 1] and β ∈ [0 , 1] , and the effective and memory efﬁcient level is “Image” level.
We conduct the following experiments using these parameters.
4. C OMPARISON WITH REGULARIZATION METHODS
The proposed ShakeDrop is compared with vanilla (without regularization), ResDrop and 1-branch
Shake in different network architectures including ResNet, PyramidNet, Wide ResNet and ResNeXt.
Tables 3 and 4 show experimental results on CIFAR-100 dataset (Krizhevsky, 2009) and Tiny ImageNet dataset 1 , respectively.
Table 3 shows that the proposed ShakeDrop can be applied to not only PyramidNet but also ResNet,
Wide ResNet and ResNeXt.
However, to successfully apply it, we found that the residual blocks have to end with batch normalization (BN) (Ioffe & Szegedy, 2015).
We believe that its reason is tightly related to the ranges of α and β .
Without BN, since the outputs of residual blocks are not in a certain range, sometimes the factors α and β could be too big.
This seems to cause diverge in
[Table]
< Conv-BN-ReLU-Conv-BN-add- (ReLU) >
[Table]
< Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
ResNeXt-29 8-64d Bottleneck < Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
PyramidNet-
[Table]
200 Bottleneck < BN-Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add
[Table]
ResNeXt-164 2-1-40d Bottleneck < Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
ResNeXt-29 2-4-64d
[Table]
< Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
×
[Table]
×
[Table]
learning.
With this regard, EraseReLU
(Dong et al., 2017) which eliminates the last ReLU is a very convenient for us because in EraseReLU versions of ResNet and ResNeXt, the residual blocks ends with BN.
On the other hand, in the case of Wide ResNet, we need to intentionally add BN in the end of the residual blocks.
The table shows that as long as the residual blocks end with BN,
the proposed
ShakeDrop achieved the lowest error rates.
[Table]
Conv-BN-ReLU-Conv-BN-add- (ReLU) >
[Table]
< Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
<
BN
[Table]
Conv-BN-ReLU-Conv-BN-add >
[Table]
< BN-Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add >
[Table]
Table 4 shows that in ResNet and ResNeXt, vanilla networks of EraseReLU achieved lowest error rates.
This could be caused by use of inappropriate base networks for the task since learning ended with relatively high training loss and no regularization methods correctly worked.
However, the table shows that the proposed ShakeDrop achieved the lowest error rates in PyramidNet and Wide
ResNet.
4. C OMPARISON WITH STATE - OF - THE - ART METHODS
The proposed method was compared with state-of-the-arts on the CIFAR-10/100 datasets.
Stateof-the-art methods introduced some techniques that can be applied to many methods in the learning process.
One is longer learning .
While most of methods related to ResNet use 300-epoch scheduling for learning like the preliminary experiment in Sec.
4.1, Shake-Shake use 1800-epoch cosine annealing, on which the initial learning rate is annealed using a cosine function without restart (Gastaldi, 2017).
Another one is image preprocessing .
DeVries & Taylor (2017b) and Zhong et al.
(2017) showed that accuracy is improved by data augmentation which randomly ﬁlls a part of learning images.
For fair comparison with these methods, we also applied them to the proposed method.
ShakeDrop (including PyramidNet and PyramidDrop) had 272 layers, which consisted of a convolution layer, 90 additive pyramidal “bottleneck” blocks (“bottleneck” is a special residual block, which is proposed by He et al.
and a fully connected layer.
The number of channels at the last pyramidal residual block was 864.
Table 5 shows the error rates.
The proposed method, ShakeDrop, without longer learning and image preprocessing , was 3.41% on the CIFAR-10 dataset and [Table] (cid:88) [Table]
- (Gastaldi, 2017) [Table] [Table] ResNeXt+Shake-Shake+Cutout
SS (cid:88)
CO
[Table]
- (DeVries & Taylor, 2017b)
[Table]
[Table]
PyramidNet
[Table]
[Table]
(Han et al., 2017b) (cid:88)
[Table]
[Table]
PyramidDrop
[Table]
[Table]
(Yamada et al., 2016)
[Table]
SD
[Table]
PyramdNet+ShakeDrop
[Table]
[Table]
(Proposed)
SD
[Table]
[Table]
SD
[Table]
RE
[Table]
14.90% on the CIFAR-100 dataset.
We make following comparisons with existing methods.
Note that the proposed method used less numbers of parameters than rival methods in all conditions.
(1) Fair comparison with Shake-Shake in the same condition ShakeDrop combined with longer training achieved error rates of 2.67% (better by 0.19%) on the CIFAR-10 dataset and 13.99% (better by 1.86%) on the CIFAR-100 dataset.
(2) Fair comparison with Cutout in the same condition ShakeDrop combined with both longer learning and image preprocessing achieved error rates of 2.31% (better by 0.25%) and 12.19% (better by 3.01%) on the CIFAR-10/100 datasets, respectively.
(3) Comparison with the state-of-the-arts, Cutout on CIFAR-10 and Coupled Ensemble on CIFAR
ShakeDrop combined with both longer learning and image preprocessing achieved error rates of 2.31% (better by 0.25% than Cutout) and 12.19% (better by 2.85% than Coupled Ensemble) on the
CIFAR-10/100 datasets, respectively.
C ONCLUSION
Based on Stochastic Depth and Shake-Shake, we proposed a new stochastic regularization method ShakeDrop and experimentally conﬁrmed its effectiveness. As a result, as shown in Table 5, the proposed method, ShakeDrop, achieved the state-of-the-art performance in the CIFAR-10/100 datasets A I MPLEMENTATION DETAILS
All implementations used in the experiments were based on the publicly available code of fb.resnet.torch 2 . ShakeDrop was trained using back-propagation by stochastic gradient descent with an approximate version 3 of Nesterov accelerated gradient (Nesterov, 1983) and momentum method (Rumelhart et al., 1986). 4 GPUs were used for learning acceleration; due to parallel processing, even the same layer had different values of parameters b l , α and β depending on GPUs. CIFAR-10/100 datasets
Input images of CIFAR-10/100 datasets were produced in the following manner.
An original image of 32 × 32 pixels was color-normalized, followed by horizontally ﬂipped with a 50% probability. Then, it was zero-padded to be 40 × 40 pixels and randomly cropped to be an image of 32 ×pixels. On PyramidNet, the initial learning late was set to 0.5 for both CIFAR-10/100 datasets following the version 2 of Han et al.
(2017b), while they use 0.1 for the CIFAR-10 dataset and 0.5 for the CIFAR-100 dataset since version 3.
Other than PyramidNet, the initial learning late was set to 0.1.
The initial learning rate was decayed by a factor of 0.1 at 1 / 2 and 3 / 4 of the entire learning process (300 epochs), respectively, following Han et al.
(2017b). As the ﬁlter parameters initializer, “MSRA” (He et al., 2015) was used.
In addition, a weight decay of 0.0001, a momentum of 0.9, and a batch size of 128 were used with 4 GPUs. On PyramidDrop and “PyramidNet + ShakeDrop,” the linear decay parameter p L = 0 . 5 was used following Huang et al. 1-branch Shake used α = [0 , 1] , β = [0 , 1] .
ShakeDrop used parameters of α =
[ − 1 , 1] , β = [0 , 1] and “Image” level scaling coefﬁcient.
Input images of Tiny ImageNet were produced in the following manner. An original image of 64 × 64 pixels was distorted aspect ratio and randomly cropped to be an image of 56 × 56 pixels. Then, brightness, contrast, and saturation of the image were randomly changed.
After that, the image was color-normalized by mean and standard deviation of ImageNet, followed by horizontally ﬂipped with a 50% probability. Learning settings of 110-layer PyramidNet on Tiny ImageNet were almost same as on CIFAR10/100. The initial learning late was set to 0.5. The initial learning rate was decayed by a factor of 0.1 at 1 / 2 and 3 / 4 of the entire learning process (300 epochs), respectively, following Han et al.
(2017b). As the ﬁlter parameters initializer, “MSRA” (He et al., 2015) was used. In addition, a weight decay of 0.0001, a momentum of 0.9, and a batch size of 128 were used with 4 GPUs. Stride size of the ﬁrst convolution layer was set to 2, and stride of average Pooling layer was set to 7.
Learning settings of other than 200-layer PyramidNet on Tiny ImageNet were almost same as 110-layer PyramidNet. The initial learning late was set to 0.1. Learning settings of 200-layer PyramidNet on Tiny ImageNet were almost same as on ImageNet. The initial learning late was set to 0.0125.
The initial learning rate was decayed by a factor of 0. at 1 / 2 , 3 / 4 and 7 / 8 of the entire learning process (120 epochs), respectively, following Han et al. (2017b). As the ﬁlter parameters initializer, “MSRA” (He et al., 2015) was used.
In addition, a weight decay of 0.0001, a momentum of 0.9, and a batch size of 32 were used with 4 GPUs. Stride size of the ﬁrst convolution layer was set to 1, and Stride of Max Pooling layer was set to 1. [Figure]
B D ISCUSSION
In order to ﬁnd out why the proposed method achieved the low error rates, we investigate what kind of difference exists with the conventional methods, particularly with PyramidNet and PyramidDrop. B. T RAINING LOSS
Fig.
2 shows change of training loss.
Fig.
2(a) is of the 300-epoch multi-step learning rate decay schedule with 110-layer networks.
Fig. 2(b) is of 1800-epoch cosine annealing schedule with 272layer networks.
PyramidDrop and ShakeDrop to which probabilistic regularization is introduced decreased the training loss more slowly than PyramidNet in both ﬁgures.
Even at the ﬁnal epoch, the loss of PyramidDrop and ShakeDrop are much larger than 0 in contrast to PyramidNet whose loss is close to 0. B. M EAN AND VARIANCE OF GRADIENTS
Since change of training loss does not ﬂuently explain the reason we are seeking, we examined the mean and variance of gradients on each iteration for each method.
Under the same conditions as the preliminary experiment in Sec. 4.1, we compared PyramidNet, PyramidDrop and ShakeDrop in the “Batch” level.
We focused on 3 residual blocks out of 54 residual blocks and named “ﬁrst,” “middle” and “ﬁnal.”
They mean the ﬁrst, 27th and ﬁnal residual blocks, respectively.
For each, we checked the 2nd convolutional layer.
Figs. 3 and 4 show change of mean and variance of gradients, respectively.
Although PyramidDrop and ShakeDrop were similar in Fig.
2, ShakeDrop took much larger values than PyramidDrop in Figs.
3 and 4.
Particularly, the difference was signiﬁcant in “ﬁrst,” and also large in “middle.”
They would indicate that relatively intensive training lasts in ShakeDrop.
[Figure]
[Figure]