S HAKE D ROP REGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes a powerful regularization method named ShakeDrop regularization .
ShakeDrop is inspired by Shake-Shake regularization that decreases error rates by disturbing learning.
While Shake-Shake can be applied to only ResNeXt which has multiple branches, ShakeDrop can be applied to not only ResNeXt but also ResNet, Wide ResNet and PyramidNet in a memory efﬁcient way.
Important and interesting feature of ShakeDrop is that it strongly disturbs learning by multiplying even a negative factor to the output of a convolutional layer in the forward training pass.
The effectiveness of ShakeDrop is conﬁrmed by experiments on
CIFAR-10/100 and Tiny ImageNet datasets.
I NTRODUCTION
Recent advances in generic object recognition have been brought by deep neural networks.
After
ResNet (He et al., 2016) opened the door to very deep CNNs of over a hundred layers by introducing the residual block, its improvements such as Wide ResNet (Zagoruyko & Komodakis, 2016),
PyramdNet (Han et al., 2017a;b) and ResNeXt (Xie et al., 2017) have broken the records of lowest error rates.
On the other hand, in learning, they often suffer from problems such as vanishing gradients.
Hence, regularization methods help to learn and boost the performance of such base network architectures.
Stochastic Depth (ResDrop) (Huang et al., 2016) and Shake-Shake (Gastaldi, 2017) are known to be effective regularization methods for ResNet and its improvements.
Among them, Shake-Shake applied to ResNeXt is the one achieving the lowest error rates on CIFAR-10/ datasets (Gastaldi, 2017).
Shake-Shake, however, has following two drawbacks.
(1) Shake-Shake can be applied to only multibranch architectures (i.e., ResNeXt).
(2) Shake-Shake is not memory efﬁcient.
Both drawbacks come from the same root.
That is, Shake-Shake requires two branches of residual blocks to apply.
If it is true, it is not difﬁcult to conceive its solution: a similar disturbance to Shake-Shake on a single residual block.
It is, however, not trivial to realize it.
The current paper addresses the problem of realizing a similar disturbance to Shake-Shake on a single residual block, and proposes a powerful regularization method, named ShakeDrop regularization .
While the proposed ShakeDrop is inspired by Shake-Shake, the mechanism of disturbing learning is completely different.
ShakeDrop disturbs learning more strongly by multiplying even a negative factor to the output of a convolutional layer in the forward training pass.
In addition, a different factor from the forward pass is multiplied in the backward training pass.
As a byproduct, however, learning process gets unstable.
Our solution to this problem is to stabilize the learning process by employing ResDrop in a different usage from the usual.
Based on experiments using various base network architectures, we reveal the condition that the proposed ShakeDrop successfully works.
E XISTING M ETHODS R EQUIRED TO I NTRODUCE THE P ROPOSED M ETHOD 2. D EEP N ETWORK ARCHITECTURES
ResNet (He et al., 2016) opened the door to very deep CNNs of over a hundred layers by introducing the residual block, given as
[Equation]
[Figure]
(a) ResNeXt (Xie et al., 2017), in which some process- ing layers omitted for conciseness.
[Figure] [Figure]
) Shake
[Figure]
Shake (ResNeXt + Shake-Shake) (Gastaldi, 2017), in which some processing layers omitted for conciseness.
[Figure]
intermediate regularization
[Figure]
the proposed ShakeDrop.
[Figure]
where x and G ( x ) are the input and output of the residual block, respectively, and F ( x ) is the output of the residual branch on the residual block.
While ResNet realized deep networks, it is imperfect.
He et al.
(2016) pointed out that high error rates obtained by the 1202-layer ResNet on the CIFAR datasets (Krizhevsky, 2009) is caused by overﬁtting.
Veit et al.
(2016) experimentally showed that some residual blocks whose channels greatly increase cause high error rates.
PyramidNet (Han et al., 2017a;b) overcame the problem of ResNet by gradually increasing channels on each residual block.
It has almost same residual block as Eqn.
It successfully realized deep CNNs of up to 272 layers and achieved the lowest error rates among the vanilla residual networks on the CIFAR datasets.
2. N ON -D EEP NETWORK ARCHITECTURES
Wide ResNet (Zagoruyko & Komodakis, 2016) improved error rates by simply increasing channels of ResNet.
While it has almost same residual block as Eqn.
(1), it has wider and shallower architecture than ResNet.
Jastrzebski et al.
(2017) experimentally showed that like PyramidNet, Wide
ResNet overcomes the problem of ResNet.
ResNeXt (Xie et al., 2017) achieved lower error rates than Wide ResNet on almost same number of parameters.
The basic architecture of ResNeXt is given as
[Equation]
where F 1 ( x ) and F 2 ( x ) are the outputs of the residual branches as shown in Figure 1(a).
The number of residual branches is not limited to 2, and the number is the most important factor to control the result.
2. R EGULARIZATION METHODS
Stochastic Depth (Huang et al., 2016) is a regularization method which overcame problems of
ResNet such as vanishing gradients.
It makes the network apparently shallow in learning by dropping residual blocks stochastically selected.
On the l th residual block from input layer, the Stochastic Depth process is given as
[Equation]
where b l ∈ { 0 , 1 } is a Bernoulli random variable with the probability of p l .
Huang et al.
(2016) tested two rules for determining p l : uniform rule and linear decay rule.
The uniform rule using a constant for all p l did not work well.
The linear decay rule which deﬁnes p l as
[Equation]
where L is the number of all layers and p L is the initial parameter, worked well.
It was originally proposed for improving ResNet.
It was also applied to PyramidNet; the combination of PyramidNet and Stochastic Depth was named PyramidDrop (Yamada et al., 2016).
Shake-Shake (Gastaldi, 2017) is a powerful regularization method for improving ResNeXt architectures.
It is given as
[Equation]
where α is a random coefﬁcient given as α ∈ [0 , 1] .
As shown in Figure 1(b), calculation of the backward pass is disturbed by another random coefﬁcient β ∈ [0 , 1] instead of α .
As a result,
Shake-Shake decreased error rates than ResNeXt.
P ROPOSED M ETHOD 3. I NTERPRETATION OF S HAKE -S HAKE REGULARIZATION AND ITS DRAWBACKS
We give an intuitive interpretation of the forward pass of Shake-Shake regularization.
To the best of our knowledge, it has not been given yet, while the phenomenon in the backward pass is experimentally investigated by Gastaldi (2017).
As shown in Eqn.
(5) (and also in Fig. 1(b)), in the forward pass, Shake-Shake interpolates the outputs of two residual branches (i.e., F 1 ( x ) and F 2 ( x ) ) with a random variable α that controls the degree of interpolation.
As DeVries & Taylor (2017a) demonstrated that interpolation of two data in the feature space can synthesize reasonable augmented data, the interpolation of two residual blocks of Shake-Shake in the forward pass can be interpreted as synthesizing data.
Use of a random variable α generates many different augmented data.
On the other hand, in the backward pass, a different random variable β is used to disturb learning to make the network learnable long time.
Gastaldi (2017) demonstrated how the difference between α and β affects.
The regularization mechanism of Shake-Shake relies on two or more residual branches, so that it can be applied only to 2-branch networks architectures.
In addition, 2-branch network architectures consume more memory than 1-branch network architectures. One may think the number of learnable parameters of ResNeXt can be kept in 1-branch and 2-branch network architectures by controlling its cardinality and the number of channels (ﬁlters). For example, a 1-branch network (e.g., ResNeXt 1-64d) and its corresponding 2-branch network (e.g., ResNeXt 2-40d) have almost same number of learnable parameters. However, even so, it increases memory consumption due to the overhead to keep the inputs of residual blocks and so on.
By comparing ResNeXt 1-64d and 2-40d, the latter requires more memory than the former by 8% in theory (for one layer) and by 11% in measured values (for 152 layers).
3. S IMILAR REGULARIZATION TO S HAKE -S HAKE ON 1- BRANCH NETWORK ARCHITECTURES
In order to realize a similar regularization to Shake-Shake on 1-branch network architectures, in the forward pass, we need a mechanism, different from interpolation, to synthesize augmented data in the feature space.
A known algorithm that generates augmented data in the feature space is virtual adversarial training (VAT) (Miyato et al., 2016).
It generates virtual adversarial examples in the feature space, from the current example, in the direction that most severely damages class label prediction.
However, since Shake-Shake does not intend to take into account the direction to generate a new example, following the way, we just apply random perturbation, using α , to the output of a residual branch (i.e., F ( x ) of Eqn.
Differently from VAT, multiplication by α , a scalar value, does not change the direction but controls dissimilarity of generated data.
One exception is the case α takes negative values (i.e., the opposite direction).
In the backward pass, we can use the same way as Shake-Shake for 1-branch network architectures.
We call the regularization method mentioned above 1-branch Shake .
It is given as
[Equation]
where α is a coefﬁcient that disturbs learning in the forward pass.
As shown in Figure 1(c), β is a coefﬁcient similar to the one of Shake-Shake and used in the backward pass.
The 1-branch Shake is expected to realize powerful generalization like Shake-Shake regularization.
However, by applying it to 110-layer PyramidNet with α ∈ [0 , 1] and β ∈ [0 , 1] following Shake-Shake, the result on the CIFAR-100 dataset was hopelessly bad (i.e., an error rate of 77.99%).
3. S TABILIZING LEARNING WITH INTRODUCTION OF MECHANISM OF R ES D ROP
Failure of 1-branch Shake is caused by too strong perturbation.
However, weakening the perturbation would also weaken the effect of regularization.
Thus, we need a trick to promote learning under strong perturbation.
Our idea is to use the mechanism of ResDrop for solving the issue.
As is written in Sec.2.3, in the original form, ResDrop promotes learning by dropping some residual blocks.
In other words, it makes a network apparently shallow in learning.
In our situation, however, the original usage of ResDrop does not contribute because a shallower network to which 1-branch Shake is applied would also suffer from strong perturbation.
Thus, we use the mechanism of ResDrop as a probabilistic switch of two network architectures: the original network (e.g., PyramidNet) and the one to which 1-branch Shake is applied (e.g., PyramidNet + 1-branch Shake ).
By mixing up two networks, the following effects are expected.
When the original network (e.g., PyramidNet) is selected, learning is correctly promoted.
When the network with strong perturbation (e.g., PyramidNet + 1-branch Shake ) is selected, learning is disturbed.
To achieve a good performance, two networks should be in a good balance.
The proposed ShakeDrop is given as
[Equation]
where b l is a Bernoulli random variable following the linear decay rule.
We call the new method “ShakeDrop.”
According to the value of b l , Eqn.
(7) is deformed as
[Equation]
In the case that the base network is PyramidNet, ShakeDrop is equivalent to PyramidNet if b l = 1 , and it is equivalent to “PyramidNet + 1-branch Shake ” if b l = 0 .
We found that the linear decay rule of ResDrop (Eqn.
(4)) works well, while we also tested the uniform rule which did not work well.
As shown in Figure 1(d), similar to Shake-Shake, β is used in the backward pass instead of α .
It is noteworthy that regardless of the value of β , the weights of the network are updated.
For example, let us consider the case of β = 0 .
In such a case, the weights of the residual blocks selected to perturbate (i.e., the layers with b l = 0 ) are not updated.
However, the inputs to their succeeding residual blocks are perturbated and the weights of the residual blocks are updated reﬂecting the perturbation.
E XPERIMENTS
See Sec.
A for implementation details.
4. P RELIMINARY EXPERIMENTS : SEARCH FOR PARAMETER SETTINGS OF THE PROPOSED METHOD
The best parameter settings of the proposed method, ShapeDrop, are searched when it is applied to PyramidNet.
They include parameter ranges of α and β , and the timing these parameters are updated.
In the preliminary experiments, the CIFAR-100 dataset was used. PyramidNet had [Table]
[Table] layers, which consisted of a convolution layer, 54 additive pyramidal residual blocks and a fully connected layer.
The number of channels at the last residual block was 286.
Table 1 shows representative parameter ranges of α and β we tested and their results.
From the results, we can see how the combinations of α and β affect.
First of all, except PyramidDrop ( α = 0 , β = 0 ), only case F ( α ∈ [ − 1 , 1] , β = 0 ) and case H ( α ∈ [ − 1 , 1] , β ∈ [0 , 1] ) were better than the original PyramidNet (case A: α = 1 , β = 1 ).
Among them, case H was the best.
It is interesting to observe three cases E ( α ∈ [ − 1 , 1] , β = 1 ), F and H all of which take the same range of α .
Since cases F and H were better than PyramidNet, we conﬁrm strong perturbation is effective.
However, case E was even worse than PyramidNet.
This implies that if strong perturbation is applied, the residual blocks selected to perturbate strongly suffer from the perturbation, so that the weights of the residual blocks should not be fully updated (i.e., β = 1 should be avoided).
On the other hand, the range of β of the best case H was in-between cases E and F. This implies that the values of β should be well balanced.
The best strategy to update the parameters (referred as scaling coefﬁcient) among “Batch,” “Image,” “Channel” and “Pixel” are explored.
“Batch” means that the same scaling coefﬁcients are used for all the images in the mini-batch for each residual block. “Image” means that the same scaling coefﬁcients are used for each image for each residual block. “Channel” means that the same scaling coefﬁcients are used for the each channel for each residual block.
“Pixel” means that the same scaling coefﬁcients are used for each element for each residual block.
ShakeDrop was trained using the best parameters found in the experiment above, i.e, ranges α ∈ [ − 1 , 1] , β ∈ [0 , 1] .
Table 2 shows that “Pixel” level was the best.
However, it required a lot of memory.
Hence, we selected “Image” level, which was the second best, considering memory efﬁciency.
Through the preliminary experiments, we found that the most effective parameters for α and β are α ∈ [ − 1 , 1] and β ∈ [0 , 1] , and the effective and memory efﬁcient level is “Image” level.
We conduct the following experiments using these parameters.
4. C OMPARISON WITH REGULARIZATION METHODS
The proposed ShakeDrop is compared with vanilla (without regularization), ResDrop and 1-branch
Shake in different network architectures including ResNet, PyramidNet, Wide ResNet and ResNeXt.
Tables 3 and 4 show experimental results on CIFAR-100 dataset (Krizhevsky, 2009) and Tiny ImageNet dataset
[Table]
< Conv-BN-ReLU-Conv-BN-add- (ReLU) >
[Table]
< Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
ResNeXt-29 8-64d Bottleneck < Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
PyramidNet-
[Table]
200 Bottleneck < BN-Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add >
[Table]
ResNeXt-164 2-1-40d Bottleneck < Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
ResNeXt-29 2-4-64d
[Table]
< Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
×
[Table]
×
[Table]
Table 3 shows that the proposed ShakeDrop can be applied to not only PyramidNet but also ResNet,
Wide ResNet and ResNeXt. However, to successfully apply it, we found that the residual blocks have to end with batch normalization (BN) (Ioffe & Szegedy, 2015). Since without BN, the outputs of residual blocks are not in a certain range, sometimes factors α and β could be too big. This seems to cause diverge in learning.
With this regard, EraseReLU
(Dong et al., 2017) which elimi-
[Table]
< Conv-BN-ReLU-Conv-BN-add- (ReLU) >
[Table]
< Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add-(ReLU) >
[Table]
< BN-Conv-BN-ReLU-Conv-BN-add >
[Table]
PyramidNet- [Table]
[Table]
< BN-Conv-BN-ReLU-Conv-BN-ReLUConv-BN-add >
[Table]
nates the last ReLU is a very convenient for us because in the EraseReLU versions of ResNet and
ResNeXt, the residual blocks ends with BN.
On the other hand, in the case of Wide ResNet, we need to intentionally add BN in the end of the residual blocks.
In ResNeXt, we examined two ways, referred as “Type A” and “Type B,” to apply ResDrop and ShakeDrop.
They mean the regularization module is inserted before and after “add” module, respectively.
As far as we examined, Type B was always better than Type A. We also conﬁrmed that ShakeDrop Type B outperformed Shake-Shake in RexNeXt.
As a conclusion, the table shows that as long as the residual blocks end with BN, the proposed ShakeDrop achieved the lowest error rates.
Table 4 shows the results on Tiny ImageNet.
To conduct the experiments, it is better to use networks tuned for ImageNet.
However, except PyramidNet, we could apply the same networks tuned for
CIFAR-100 dataset were used due to time constraint.
As a result, in ResNet, the lowest error rates were achieved by vanilla EraseReLU networks.
This could be caused by use of inappropriate base networks for the task since learning ended with relatively high training loss and no regularization methods correctly worked.
However, in PyramidNet and Wide ResNet, the proposed ShakeDrop achieved the lowest error rates. This conﬁrms that the proposed ShakeDrop works on Tiny ImageNet dataset.
4. C OMPARISON WITH STATE - OF - THE - ART METHODS
The proposed method was compared with state-of-the-arts on the CIFAR-10/100 datasets.
Stateof-the-art methods introduced some techniques that can be applied to many methods in the learning process.
One is longer learning .
While most of methods related to ResNet use 300-epoch scheduling for learning like the preliminary experiment in Sec.
4.1, Shake-Shake use 1800-epoch cosine annealing, on which the initial learning rate is annealed using a cosine function without restart (Gastaldi, 2017).
Another one is image preprocessing .
DeVries & Taylor (2017b) and Zhong et al.
(2017) showed that accuracy is improved by data augmentation which randomly ﬁlls a part of learning images.
For fair comparison with these methods, we also applied them to the proposed method.
PyramidNet (including PyramidDrop and “PyramidDrop + ShakeDrop”) had 272 layers, which consisted of a convolution layer, 90 additive pyramidal “bottleneck” blocks (He et al., 2016) and a fully connected layer.
The number of channels at the last pyramidal residual block was 864.
Tableshows the error rates.
The proposed method, “PyramidNet + ShakeDrop,” without longer learning and image preprocessing , was 3.41% on the CIFAR-10 dataset and 14.90% on the CIFARdataset.
We make following comparisons with existing methods.
Note that the proposed method used less numbers of parameters than rival methods in all conditions.
(1) Fair comparison with Shake-Shake (ResNeXt + Shake-Shake) in the same condition “PyramidNet + ShakeDrop” combined with longer training achieved error rates of 2.67% (better by 0.19%) on the CIFAR-10 dataset and 13.99% (better by 1.86%) on the CIFAR-100 dataset.
(2) Fair comparison with Cutout (ResNeXt + Shake-Shake + Cutout) in the same condition “PyramidNet + ShakeDrop” combined with both longer learning and image preprocessing achieved error rates of 2.31% (better by 0.25%) and 12.19% (better by 3.01%) on the CIFAR-10/100 datasets, respectively.
(3) Comparison with the state-of-the-arts, Cutout on CIFAR-10 and Coupled Ensemble on
CIFAR- “PyramidNet + ShakeDrop” combined with both longer learning and image preprocessing achieved error rates of 2.31% (better by 0.25% than Cutout) and 12.19% (better by 2.85% than Coupled
Ensemble) on the CIFAR-10/100 datasets, respectively.
C ONCLUSION
We proposed a new stochastic regularization method ShakeDrop which can be successfully applied to ResNet and its improvements as long as the residual blocks ends with BN.
Its effectiveness was conﬁrmed through the experiments on CIFAR-10/100 and Tiny ImageNet datasets. ShakeDrop achieved the state-of-the-art performance in the CIFAR-10/100 datasets and the best results among existing regularization methods on Tiny ImageNet dataset except some cases using inappropriate networks.
A I MPLEMENTATION DETAILS
All implementations used in the experiments were based on the publicly available code of fb.resnet.torch 2 .
ShakeDrop was trained using back-propagation by stochastic gradient descent with an approximate version 3 of Nesterov accelerated gradient (Nesterov, 1983) and momentum method (Rumelhart et al., 1986).
4 GPUs were used for learning acceleration; due to parallel processing, even the same layer had different values of parameters b l , α and β depending on GPUs.
CIFAR-10/100 datasets
Input images of CIFAR-10/100 datasets were processed in the following manner.
An original image of 32 × 32 pixels was color-normalized, followed by horizontally ﬂipped with a 50% probability.
Then, it was zero-padded to be 40 × 40 pixels and randomly cropped to be an image of 32 ×pixels.
On PyramidNet, the initial learning late was set to 0.5 for both CIFAR-10/100 datasets following the version 2 of Han et al.
(2017b), while they use 0.1 for the CIFAR-10 dataset and 0.5 for the
CIFAR-100 dataset since version 3.
Other than PyramidNet, the initial learning late was set to 0.1.
The initial learning rate was decayed by a factor of 0.1 at 1 / 2 and 3 / 4 of the entire learning process (300 epochs), respectively, following Han et al.
(2017b).
As the ﬁlter parameters initializer, “MSRA” (He et al., 2015) was used.
In addition, a weight decay of 0.0001, a momentum of 0.9, and a batch size of 128 were used with 4 GPUs.
On PyramidDrop and “PyramidNet + ShakeDrop,” the linear decay parameter p L = 0 . 5 was used following Huang et al.
1-branch Shake used α =
[0 , 1] , β = [0 , 1] .
ShakeDrop used parameters of α =
[ − 1 , 1] , β = [0 , 1] and “Image” level scaling coefﬁcient.
Tiny ImaneNet dataset
Input images of Tiny ImageNet were processed in the following manner.
An original image of 64 × 64 pixels was distorted aspect ratio and randomly cropped to be an image of 56 × 56 pixels.
Then, brightness, contrast, and saturation of the image were randomly changed.
After that, the image was color-normalized by mean and standard deviation of ImageNet, followed by horizontally ﬂipped with a 50% probability.
Learning settings of 110-layer PyramidNet on Tiny ImageNet were almost same as on CIFAR10/100.
The initial learning late was set to 0.5.
The initial learning rate was decayed by a factor of 0.1 at 1 / 2 and 3 / 4 of the entire learning process (300 epochs), respectively, following Han et al.
(2017b).
As the ﬁlter parameters initializer, “MSRA” (He et al., 2015) was used.
In addition, a [Figure] weight decay of 0.0001, a momentum of 0.9, and a batch size of 128 were used with 4 GPUs.
Stride size of the ﬁrst convolution layer was set to 2, and stride of average Pooling layer was set to 7.
Learning settings of other than 200-layer PyramidNet on Tiny ImageNet were almost same as 110-layer PyramidNet.
The initial learning late was set to 0.1.
Learning settings of 200-layer PyramidNet on Tiny ImageNet were almost same as on ImageNet.
The initial learning late was set to 0.0125.
The initial learning rate was decayed by a factor of 0.
at 1 / 2 , 3 / 4 and 7 / 8 of the entire learning process (120 epochs), respectively, following Han et al.
(2017b).
As the ﬁlter parameters initializer, “MSRA” (He et al., 2015) was used.
In addition, a weight decay of 0.0001, a momentum of 0.9, and a batch size of 32 were used with 4 GPUs.
Stride size of the ﬁrst convolution layer was set to 1, and Stride of Max Pooling layer was set to 1.
B D ISCUSSION
In order to ﬁnd out why the proposed method achieved the low error rates, we investigate what kind of difference exists with the conventional methods, particularly with PyramidNet and PyramidDrop.
B. T RAINING LOSS
Fig.
2 shows change of training loss.
Fig.
2(a) is of the 300-epoch multi-step learning rate decay schedule with 110-layer networks.
Fig.
2(b) is of 1800-epoch cosine annealing schedule with 272layer networks.
ResDrop and ShakeDrop, which are probabilistic regularization methods, decreased the training loss more slowly than vanilla PyramidNet in both ﬁgures.
Even at the ﬁnal epoch, the loss of ResDrop and ShakeDrop are much larger than 0 in contrast to vanilla PyramidNet whose loss is close to 0.
B. M EAN AND VARIANCE OF GRADIENTS
In addition to the training loss, we examined the mean and variance of gradients on each iteration for each regularization method.
Under the same conditions as the preliminary experiment in Sec. 4.1, we compared PyramidNet with no regularization method (vanilla), ResDrop and ShakeDrop in the “Batch” level.
We focused on 3 residual blocks out of 54 residual blocks and named “ﬁrst,” “middle” and “ﬁnal.”
They mean the ﬁrst, 27th and ﬁnal residual blocks, respectively.
For each, we checked the 2nd convolutional layer.
Figs.
3 and 4 show change of mean and variance of gradients, respectively.
Although ResDrop and ShakeDrop were similar in Fig.
2, ShakeDrop took much larger values than ResDrop in Figs.
3 and 4.
Particularly, the difference was signiﬁcant in “ﬁrst,” and also large in “middle.”
They would indicate that relatively intensive training lasts in ShakeDrop.
[Figure] [Figure]