E FFECTIVE AND E FFICIENT B ATCH N ORMALIZATION U SING F EW U NCORRELATED D ATA FOR S TATISTICS ’ E STIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep Neural Networks (DNNs) thrive in recent years in which Batch Normalization (BN) plays an indispensable role.
However, it has been observed that BN is costly due to the reduction operations.
In this paper, we propose alleviating the BN’s cost by using only a small fraction of data for mean & variance estimation at each iteration.
The key challenge to reach this goal is how to achieve a satisfactory balance between normalization effectiveness and execution efﬁciency. We identify that the effectiveness expects less data correlation while the efﬁciency expects regular execution pattern.
To this end, we propose two categories of approach: sampling or creating few uncorrelated data for statistics’ estimation with certain strategy constraints.
The former includes “Batch Sampling (BS)” that randomly selects few samples from each batch and “Feature Sampling (FS)” that randomly selects a small patch from each feature map of all samples, and the latter is “Virtual Dataset Normalization (VDN)” that generates few synthetic random samples. Accordingly, multi-way strategies are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime.
All the proposed methods are comprehensively evaluated on various DNN models, where an overall training speedup by up to 21.7% on modern
GPUs can be practically achieved without the support of any specialized libraries, and the loss of model accuracy and convergence rate are negligible.
Furthermore, our methods demonstrate powerful performance when solving the well-known “micro-batch normalization” problem in the case of tiny batch size.
I NTRODUCTION
Recent years, Deep Neural Networks (DNNs) have achieved remarkable success in a wide spectrum of domains such as computer vision (Krizhevsky et al., 2012) and language modeling (Collobert & Weston, 2008).
The success of DNNs largely relies on the capability of presentation beneﬁt from the deep structure (Delalleau & Bengio, 2011).
However, training a deep network is so difﬁcult to converge that batch normalization (BN) has been proposed to solve it (Ioffe & Szegedy, 2015).
BN leverages the statistics (mean & variance) of mini-batches to standardize the activations.
It allows the network to go deeper without signiﬁcant gradient explosion or vanishing (Santurkar et al., 2018; Ioffe & Szegedy, 2015).
Moreover, previous work has demonstrated that BN enables the use of higher learning rate and less awareness on the initialization (Ioffe & Szegedy, 2015), as well as produces mutual information across samples (Morcos et al., 2018) or introduces estimation noises (Bjorck et al., 2018) for better generalization.
Despite BN’s effectiveness, it is observed that BN introduces considerable training overhead due to the costly reduction operations.
The use of BN can lower the overall training speed (mini second per image) by > 45% (Wu et al., 2018), especially in deep models.
To alleviate this problem, several methods were reported.
Range Batch Normalization (RBN) (Banner et al., 2018) accelerated the forward pass by estimating the variance according to the data range of activations within each batch.
A similar approach, L 1 -norm BN (L1BN) (Wu et al., 2018), simpliﬁed both the forward and backward passes by replacing the L 2 -norm variance with its L 1 -norm version and re-derived the gradients for backpropagation (BP) training.
Different from the above two methods, Self-normalization (Klambauer et al., 2017) provided another solution which totally eliminates the need of BN operation with an elaborate activation function called “scaled exponential linear unit” (SELU).
SELU can automatically force the activation towards zero mean and unit variance for better convergence.
Nevertheless, all of these methods are not sufﬁciently effective.
The strengths of L1BN & RBN are very limited since GPU has sufﬁcient resources to optimize the execution speed of complex arithmetic operations such as root for the vanilla calculation of L 2 -norm variance.
Since the derivation of SELU is based on the plain convolutional network, currently it cannot handle other modern structures with skip paths like ResNet and DenseNet.
In this paper, we propose mitigating BN’s computational cost by just using few data to estimate the mean and variance at each iteration. Whereas, the key challenge of this way lies at how to preserve the normalization effectiveness of the vanilla BN and improve the execution efﬁciency in the meantime, i.e. balance the effectiveness-efﬁciency trade-off.
We identify that the effectiveness preservation expects less data correlation and the efﬁciency improvement expects regular execution pattern.
This observation motivates us to propose two categories of approach to achieve the goal of effective and efﬁcient BN: sampling or creating few uncorrelated data for statistics’ estimation with certain strategy constraints.
Sampling data includes “Batch Sampling (BS)” that randomly selects few samples from each batch and “Feature Sampling (FS)” that randomly selects a small patch from each feature map (FM) of all samples; creating data means “Virtual Dataset Normalization (VDN)” that generates few synthetic random samples, inspired by Salimans et al.
Consequently, multi-way strategies including intra-layer regularity, inter-layer randomness, and static execution graph during each epoch, are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime. All the proposed approaches with single-use or joint-use are comprehensively evaluated on various DNN models, where the loss of model accuracy and convergence rate is negligible.
We practically achieve an overall training speedup by up to 21.7% on modern GPUs.
Note that any support of specialized libraries is not needed in our work, which is not like the network pruning (Zhu et al., 2018) or quantization (Hubara et al., 2017) requiring extra library for sparse or low-precision computation, respectively.
Most previous acceleration works targeted inference which remained the training inefﬁcient (Wen et al., 2016; Molchanov et al., 2016; Luo et al., 2017; Zhang et al., 2018b; Hu et al., 2018), and the rest works for training acceleration were orthogonal to our approach (Lin et al., 2017;
Goyal et al., 2017; You et al., 2017).
Additionally, our methods further shows powerful performance when solving the well-known “micro-batch normalization” problem in the case of tiny batch sizes.
In summary, the major contributions of this work are summarized as follows.
• We propose a new way to alleviate BN’s computational cost by using few data to estimate the mean and variance, in which we identify that the key challenge is to balance the normalization effectiveness via less data correlation and execution efﬁciency via regular execution pattern.
• We propose two categories of approach to achieve the above goal: sampling (BS/FS) or creating (VDN) few uncorrelated data for statistics’ estimation, in which multi-way strategies are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime.
The approaches can be used alone or jointly.
• Various benchmarks are evaluated, on which up to 21.7% practical acceleration is achieved for overall training on modern GPUs with negligible accuracy loss and without specialized library support.
• Our methods are also extended to the micro-BN problem and achieve advanced performance 1 .
In order to make this paper easier for understanding, we present the organization of the whole paper in Fig.
14, Appendix G.
M OTIVATION , O PPORTUNITY , AND C HALLENGE 2. M OTIVATION : C OSTLY B ATCH N ORMALIZATION & THE B OTTLENECK A NALYSIS
The activations in one layer for normalization can be described by a d -dimensional activation feature
XXX =
( x (1) , .., x ( d ) ) , where for each feature we have x ( k ) =
( x ( k ) 1 , .., x ( k ) m ) .
Note that in convolutional (Conv) layer, d is the number of FMs and m equals to the number of points in each FM across all the samples in one batch; while in fully-connected (FC) layer, d and m are the neuron number and batch size, respectively.
BN uses the statistics (mean E [ x ( k ) ] & variance V ar [ x ( k ) ] ) of the intra-batch data for each feature to normalize activation by [Equation]
[Equation]
where γ ( k ) & β ( k ) are trainable parameters introduced to recover the representation capability, (cid:15) is a small constant to avoid numerical error, and E [ x ( k ) ] & V ar [ x ( k ) ] can be calculated by
[Equation]
The detailed operations of a BN layer in the backward pass can be found in Appendix C.
Model w/o BN w/ BN
C 26.
D 7.
E 5.
x ∑ ∑ 𝜇𝜇 𝜎𝜎 �𝒙𝒙 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕
∑ ∑ 𝜕𝜕𝜕𝜕 𝜕𝜕𝜇𝜇 𝜕𝜕𝜕𝜕 𝜕𝜕𝜎𝜎 𝜕𝜕𝜕𝜕 𝜕𝜕�𝜕𝜕 x ∑ ∑ 𝜇𝜇 𝜎𝜎 �𝒙𝒙 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕
∑ ∑ 𝜕𝜕𝜕𝜕 𝜕𝜕𝜇𝜇 𝜕𝜕𝜕𝜕 𝜕𝜕𝜎𝜎 𝜕𝜕𝜕𝜕 𝜕𝜕
�𝜕𝜕 𝜕𝜕
𝑠𝑠 pad (b) (c) (d) (a)
Iter.
per second.
From Fig. 1(a), we can see that adding BN will signiﬁcantly slow down the training speed (iterations per second) by 32%-43% on ImageNet.
The reason why BN is costly is that it contains several ”reduction operations”, i.e. (cid:80) mj
=1 .
We offer more thorough data analysis in Appendix E.
If the reduction operations are not optimized, it’s computational complexity should be O ( m ) .
With the optimized parallel algorithm proposed in (Che et al., 2008), the reduction operation is transformed to cascaded adders of depth of log ( m ) as shown in Fig. 1(b).
However, the computational cost is still high since we usually have m larger than one million.
As shown in Fig.
1(c), the red ” (cid:80) ”
s represent operations that contain summations, which cause the BN inefﬁciency.
2. O PPORTUNITY : U SING F EW D ATA FOR S TATISTICS ’ E STIMATION
Motivated by the above analysis, decreasing the effective value of m at each time for statistics estimation seems a promising way to reduce the BN cost for achieving acceleration.
To this end, we propose using few data to estimate the mean and variance at each iteration.
For example, if m changes to a much smaller value of s , equation (2) can be modiﬁed as
[Equation]
where x ( k ) s denotes the small fraction of data, s is the actual number of data points, and we usually have s (cid:28) m .
Here we denote s/m as Sampling Ratio (it includes both the cases of sampling and creating few data in Section 3.1 and 3.2, respectively). Since the reduction operations in the backward pass can be parallelized whereas in the forward pass, the variance can not be calculated until mean is provided (which makes it nearly twice as slow as backward pass), we just use few data in the forward pass.
The computational graph of BN using few data is illustrated in Fig. 1(d).
The key is how to estimate E [ x ( k ) ] & V ar [ x ( k ) ] for each neuron or FM within one batch with much fewer data.
The inﬂuence on the backward pass is discussed in Appendix C. 2. C HALLENGE : E FFECTIVENESS -E FFICIENCY B ALANCE
Although using few data can reduce the BN’s cost dramatically, it will meet an intractable challenge: how to simultaneously preserve normalization effectiveness and improve the execution efﬁciency. On one side, using few data to estimate the statistics will increase the estimation error. By regarding the layers with high estimation error as unnormalized ones, we did contrast test in Fig.
The mean & variance will be scaled up exponentially as network deepens, which causes the degradation of BN’s effectiveness. This degradation can be recovered from two aspects.
• Intra-layer . For the reason that the estimation error is not only determined by the amount of data but also the correlation between them, we can sample less correlated data within each layer to improve the estimation accuracy. • Inter-layer . As depicted in by Fig.
2, the intermittent BN conﬁguration (i.e. discontinuously adding BN in different layers) can also prevent the statistics scaling up across layers. This motivates us that as long as layers with high estimation error are discontinuous, the statistics shift can still be constrained to a smaller range. Therefore, to reduce the correlation between estimation errors in different layers can also be beneﬁcial to improve the accuracy of the entire model, which can be achieved by sampling less correlated data between layers.
On the other side, less data correlation indicates more randomness which usually causes irregular memory access irregular degrading the running efﬁciency. In this paper, we recognize that the overhead of sampling can be well reduced by using regular and static execution patterns, which is demonstrated with ablation study at Fig.
9 (c), Sec. 4.2. [Figure] In a nutshell, careful designs are needed to balance the normalization effectiveness via less data correlation and the execution efﬁciency via regular execution pattern. Only in this way, it is possible to achieve practical acceleration with little accuracy loss, which is our major target in this work.
Based on the above analysis, we summarize the design considerations as follows. • Using few data for statistics’ estimation can effectively reduce the computational cost of BN operations.
Whereas, the effectiveness-efﬁciency trade-off should be well balanced. • Less data correlation is promising to reduce the estimation error and then guarantees the normalization effectiveness. • More regular execution pattern is expected for efﬁcient running on practical platforms.
A PPROACHES
To reach the aforementioned goal, we propose two categories of approach in this section: sampling or creating few uncorrelated data for statistics’ estimation. Furthermore, multi-way strategies to balance the data correlation and execution regularity.
[Figure]
3. S AMPLING U NCORRELATED D ATA : B ATCH S AMPLING AND F EATURE S AMPLING
Uncorrelated Sampling . Here “sampling” means to sample a small fraction of data from the activations at each layer for statistics’ estimation. As discussed in the previous section, mining uncorrelated data within and between layers is critical to the success of sampling-based BN. However, the correlation property of activations in deep networks is complex and may vary across network structures and datasets.
Instead, we apply a hypothesis-testing approach in this work. We ﬁrst make two empirical assumptions:
• Hypothesis 1 . Within each layer, data belonging to the different samples are more likely to be uncorrelated than those within the same sample. • Hypothesis 2 . Between layers, data belonging to different locations and different samples are less likely to be correlated.
Here “location” means coordinate within FMs.
These two assumptions are based on the basic nature of real-world data and the networks, thus they are likely to hold in most situations. They are further evaluated through experiments in Section 4.1. Based on above hypotheses , we propose two uncorrelated-sampling strategies: Batch Sampling (BS) and Feature Sampling (FS) .
The detailed Algorithms can be found in Alg. 1 and 2, respectively, Appendix A. • BS (Fig. 3(b)) randomly selects few samples from each batch for statistics’ estimation. To reduce the inter-layer data correlation, it selects different samples across layers following Hypothesis 2.
• FS (Fig. 3(c)) randomly selects a small patch from each FM of all samples for statistics’ estimation.
Since the sampled data come from all the samples thus it has lower correlation within each layer following Hypothesis 1. Furthermore, it samples different patch locations across layers to reduce the inter-layer data correlation following Hypothesis 2.
A Naive Sampling (NS) is additionally proposed as a comparison baseline, as shown in Fig. 3(a). NS is similar to BS while the sampling index is ﬁxed across layers,
i
e. consistently samples ﬁrst few samples within each batch.
Regular and Static Sampling .
In order to achieve practical acceleration on GPU, we expect more regular sampling pattern and more static sampling index.
Therefore, to balance the estimation effectiveness and execution efﬁciency, we carefully design the following sampling rules: (1) In BS, the selected samples are continuous and the sample index for different channels are shared, while they are independent across layers; (2) In FS, the patch shape is rectangular and the patch location is shared by different channels and samples within each layer but variable as layer changes.
Furthermore, all the random indexes are updated only once for each epoch, which guarantees a static computational graph during the entire epoch.
3. C REATING U NCORRELATED D ATA : V IRTUAL D ATASET N ORMALIZATION
Instead of sampling uncorrelated data, another plausible solution is to directly create uncorrelated data for statistics’ estimation.
We propose Virtual Dataset Normalization (VDN) 2 to implement it, as illustrated in Fig. 3(d).
VDN is realized with the following three steps: (1) calculating the statistics of the whole training dataset ofﬂine; (2) generating s virtual samples 3 at each iteration to concatenate with the original real inputs as the ﬁnal network inputs. (3) using data from only virtual samples at each layer for statistics’ estimation. Due to the independent property of the synthesized data, they are more uncorrelated than real samples thus VDN can produce much more accurate estimation.
The detailed implementation algorithm can be found in Alg. 3, Appendix A.
The sampling approach and the creating approach can be used either in a single way or in a joint way. Besides the single use, a joint use can be described as
[Equation]
where x s denotes the sampled real data while x v represents the created virtual data.
β is a controlling variable, which indicates how large the sampled data occupy the whole data for statistics’ estimation within each batch: (1) when β = 0 or 1, the statistics come from single approach (VDN or any sampling strategy); (2) when β ∈ (0 , 1) , the ﬁnal statistics are a joint value as shown in equation 4.
3. C OMPARISON BETWEEN D IFFERENT U SING W AYS
A comparison between different using ways is presented in Table 1, where “d.s.” denotes “different samples”; “d.l.” stands for “different locations”, and “g.i.” indicates “generated independent data”. Compared to NS, BS reduces the inter-layer correlation via selecting different samples across layers;
FS reduces both the intra-layer and inter-layer correlation via using data from all samples within each layer and selecting different locations across layers, respectively.
Though VDN has a similar inter-layer correlation with NS, it slims the intra-layer correlation with strong data independence. A combination of BS/FS and VDN can inherit the strength of different approaches, thus achieve much lower accuracy loss. [Table]
E XPERIMENTS
[Table]
Experimental Setup .
All of our proposed approaches are validated on image classiﬁcation task using
CIFAR-10, CIFAR-100 and ImageNet datasets from two perspectives: (1) effectiveness evaluation and (2) efﬁciency execution.
To demonstrate the scalability and generality of our approaches on deep networks, we select ResNeton CIFAR-10 & CIFAR-100 and select ResNet-18 and DenseNet-121 on ImageNet 4 . The model [Table]
Dataset: generated
On ImageNet, most experiments are conducted on ResNet-18 due to the faster training time, and only joint approaches are demonstrated on on DenseNet-121.
The accuracy and speedup results are on DenseNet-121 are enough to evidence the effectiveness and efﬁciency of our approaches for deep models.
conﬁguration can be found in Table 2.
The means and variances for BN are locally calculated in each GPU without inter-GPU synchronization as usual.
We denote our approaches as the format of “Approach-Sampled size/Original size-sampling ratio(%)”.
For instance, if we assume batch size is 128, “BS-4/128-3.1%” denotes only 4 samples are sampled in BS and the sampling ratio equals to 4128 = 3 .
Similarly, “FS-1/32-3.1%” implies a 132 = 3 . 1% patch is sampled from each FM, and “VDN-1/128-0.8%” indicates only one virtual sample is added.
The traditional BN is denoted as “BN-128/128-100.0%”.
Other experimental conﬁgurations can be found in Appendix B.
4. E FFECTIVENESS E VALUATION
Convergence Analysis .
Fig.
4 shows the top-1 validation accuracy and conﬁdential interval of
ResNet-56 on CIFAR-10 and CIFAR-100.
On one side, all of our approaches can well approximate the accuracy of normal BN when the sampling ratio is larger than 2% , which evidence their effectiveness.
On the other side, all the proposed approaches perform better than the NS baseline.
In particular, FS performs best, which is robust to the sampling ratio with negligible accuracy loss (e.g. at sampling ratio=1.6%, the accuracy degradation is [--0.
 %-]-0.087% on CIFAR-10 and +0.396% on
CIFAR-100).
VDN outperforms BS and NS with a large margin in extremely small sampling ratio (e.g. 0.8%), whereas the increase of virtual batch size leads to little improvement on accuracy.
BS is constantly better than NS.
Furthermore, an interesting observation is that the BN sampling could even achieve better accuracy sometimes, such as NS-8/128 (72.6 ± 1.5%) , BS-8/128 (72.3 ± 1.5%) , and FS-1/64 (71.2 ± 0.96%) against the baseline (70.8 ± 1%) on CIFAR-100.
Fig.5 further shows the training curves of ResNet-56 on CIFAR-10 under different approaches.
It reveals that FS and VDN would not harm the convergence rate, while BS and NS begin to degrade the convergence when the sampling ratio is smaller than 1.6% and 3.1%, respectively.
Table 3 shows the top-1 validation error on ImageNet under different approaches.
With the same sampling ratio, all the proposed approaches signiﬁcantly outperform NS, and FS surpasses VDN and
BS.
Under the extreme sampling ratio of 0.78%, NS and BS don’t converge.
Due to the limitation of
FM size, the smallest sampling ratio of FS is 1.6%, which has only -0.5% accuracy loss.
VDN can still achieve relatively low accuracy loss (1.4%) even if the sampling ratio decreases to 0.78%.
This implies that VDN is effective for normalization.
Moreover, by combining FS-1/64 and VDN-2/128,
[Table]
we get the lowest accuracy loss (-0.2%).
This further indicates that VDN can be combined with other sampling strategies to achieve better results.
Since training DenseNet-121 is time-consuming, we just report the results with FS/BS-VDN joint use.
Although DenseNet-121 is more challenging than ResNet-18 due to the much deeper structure
[Table]
the “FS-1/64 + VDN-2/64” can still achieve very low accuracy loss (-0.6%).
“BS-1/64 + VDN-2/64” has a little higher accuracy loss, whereas it still achieves better result than NS.
In fact, we observed gradient explosion if we just use VDN on very deep network (i.e. DenseNet-121), which can be conquered through jointly applying VDN and other proposed sampling approach (e.g. FS+VDN).
Fig.
6 illustrates the training curves for better visualization of the convergence.
Except for the BS with extremely small sampling ratio (0.8%) and NS, other approaches and conﬁgurations can achieve satisfactory convergence.
Here, we further evaluate the fully random sampling (FRS) strategy, which samples completely random points in both the batch and FM dimensions.
We can see that FRS is less stable compared with our proposed approaches (except the NS baseline) and achieves much lower accuracy.
One possible reason is that under low sampling ratio, the sampled data may occasionally fall into the worse points, which lead to inaccurate estimation of the statistics.
Correlation Analysis .
In this section, we bring more empirical analysis to the data correlation that affects the error of statistical estimation.
Here we denote the estimation errors at l th layer as E ( l ) µ = || µ ( l ) s − µ ( l ) ||
2 and E ( l ) σ = ||
σ ( l ) s − σ ( l ) || 2 , where µ ( l ) s & σ ( l ) s are the estimated mean & variance from the sampled data (including the created data in VDN) while µ ( l ) & σ ( l ) are the ground truth from the vanilla BN for the whole batch.
The analysis is conducted on ResNet-56 over CIFAR-10.
The estimation errors of all layers are recorded throughout the ﬁrst training epoch.
Fig.and Fig.
8 present the distribution of estimation errors for all layers and the inter-layer correlation between estimation errors, respectively.
In Fig.
7, FS demonstrates the least estimation error within each layer which implies its better convergence. The estimation error of VDN seems similar to BS and NS here, but we should note that it uses much lower sampling ratio of 0.8% compared to others
[Figure]
of 3.1%.
From Fig.
[Figure] it can be seen that BS presents obviously less inter-layer correlation than NS, which is consistent with previous experimental results that BS can converge better than NS even though they have similar estimation error as shown in Fig.
For FS and VDN, although it looks like they present averagely higher correlations, there exist negative corrections which effectively improve the model accuracy.
Moreover, FS produces better accuracy than NS and BS since its selected data come from all the samples with less correlation.
4. E FFICIENCY E VALUATION
After the normalization effectiveness evaluation, we will evaluate the execution efﬁciency which is the primary motivation.
Fig.
9 shows the BN speedup during training and overall training improvement.
[Figure]
In general, BS can gain higher acceleration ratio because it doesn’t incur the ﬁne-grained sampling within FMs like in FS and it doesn’t require the additional calculation and concatenation of the virtual samples like in VDN.
As for FS, it fails to achieve speedup on CIFAR-10 due to the small image size that makes the reduction of operations unable to cover the sampling overhead.
The proposed approaches can obtain up to 2x BN acceleration and 21.8% overall training acceleration.
Table 4 gives more results on ResNet-18 using single approach and DenseNet-121 using joint approach.
On ResNet-18, we perform much faster training compared with two recent methods for BN simpliﬁcation (Wu et al., 2018; Banner et al., 2018).
On ResNet18 we can achieve up to 16.5% overall training speedup under BS; on very deep networks with more BN layers, such as DenseNet-121, the speedup is more signiﬁcant that reaches 23.8% under “BS+VDN” joint approach. “FS+VDN” is a little bit slower than “BS+VDN” since the latter one has a more regular execution pattern as aforementioned. Nonetheless, on a very deep model, we still recommend the “FS+VDN” version because it can preserve the accuracy better. The relationship be-
tween sampling ratio and overall training speedup is represented in Fig. 10 which illustrates that 1) BS & FS can still achieve considerable speedup with a moderate sampling ratio; 2) BS can achieve more signiﬁcant acceleration than FS, for its more regular execution pattern. It’s worth noting that, our training speedup is practically obtained on modern GPUs without the support of specialized library that makes it easy-to-use.
Fig.
9 (c) reveals that the regular execution pattern can signiﬁcantly help us achieve practical speedup
[Table]
[Table]
R ELATED W ORK
BN has been applied in most state-of-art DNN models (He et al., 2016; Szegedy et al., 2017) since it was proposed. As aforementioned, BN standardizes the activation distribution to reduce the internal covariate shift.
Models with BN have been demonstrated to converge faster and generalize better (Ioffe & Szegedy, 2015; Morcos et al., 2018). Recently, a model called Decorrelated Batch
Normalization (DBN) was introduced which not only standardizes but also whitens the activations with ZCA whitening (Huang et al., 2018). Although DBN further improves the normalization performance, it introduces signiﬁcant extra computational cost.
Simplifying BN has been proposed to reduce BN’s computational complexity. For example, L1BN (Wu et al., 2018) and RBN (Banner et al., 2018) replace the original L 2 -norm variance with an L 1 -norm version and the range of activation values, respectively.
From another perspective, Selfnormalization uses the customized activation function (SELU) to automatically shift activation’s distribution (Klambauer et al., 2017).
However, as mentioned in Introduction, all of these methods fail to obtain a satisfactory balance between the effective normalization and computational cost, especially on large-scale modern models and datasets. Our work attempts to address this issue.
C ONCLUSION
Motivated by the importance but high cost of BN layer, we propose using few data to estimate the mean and variance for training acceleration.
The key challenge towards this goal is how to balance the normalization effectiveness with much less data for statistics’ estimation and the execution efﬁciency with irregular memory access.
To this end, we propose two categories of approach: sampling (BS/FS) or creating (VDN) few uncorrelated data, which can be used alone or jointly.
Speciﬁcally,
BS randomly selects few samples from each batch, FS randomly selects a small patch from each FM of all samples, and VDN generates few synthetic random samples. Then, multi-way strategies including intra-layer regularity, inter-layer randomness, and static execution graph are designed to reduce the data correlation and optimize the execution pattern in the meantime.
Comprehensive experiments evidence that the proposed approaches can achieve up to 21.7% overall training acceleration with negligible accuracy loss.
In addition, VDN can also be applied to the micro-BN scenario with advanced performance.
This paper preliminary proves the effectiveness and efﬁciency of BN using few data for statistics’ estimation.
We emphasize that the training speedup is practically achieved on modern GPUs, and we do not need any support of specialized libraries making it easy-to-use.
Developing specialized kernel optimization deserves further investigation for more aggressive execution beneﬁts.
A PPENDIX A I MPLEMENTATION A LGORITHMS
Notations .
We use the Conv layer for illustration, which occupies the major part of most modern networks (He et al., 2016; Huang et al., 2016).
The batched features can be viewed as a 4D tensor.
We use “ E 0 , 1 , 2 ” and “ V ar 0 , 1 , 2 ” to represent the operations that calculate the means and variances, respectively, where “0, 1, 2” denotes the dimensions for reduction.
Algorithm 1: NS/BS Algorithm
Data: input batch at layer l : B l ∈ R N × H l × W l × C l , l ∈ all layers; sampling size: n s ∈ (0 , N ]
Result: estimation of E [ B l ] & V ar
[ B l ] , l ∈ all layers beginfor ep ∈ all epochs do for l ∈ all layers do if BS: begin l = randint (0 , N − n s ) ; else NS: begin l =for it ∈ all iterations do for l ∈ all layers do
B s = B l [ begin l : begin l + n s − 1 , : , : , :]
E [ B l ] = E 0 , 1 , 2 [ B s ] , V ar
[ B l ] =
V ar 0 , 1 , 2 [ B s ]
Algorithm 2: Algorithm Data: input batch at layer l : B ∈ R N × H l × W l × C l , l ∈ all layers; sampling size: h ( l ) s ∈ (0 , H l ] & w ( l ) s ∈ (0 , W l ]
Result: estimation of E [ B l ] & V ar
[ B l ] , l ∈ all layers beginfor ep ∈ all epochs do for l ∈ all layers do begin ( l ) h = randint (0 , H l − h ( l ) s ) , begin ( l ) w = randint (0 , W l − w ( l ) s ) for it ∈ all iterations do for l ∈ all layers do
B s =
B [: , begin ( l ) h : begin
( l ) h
+ h ( l ) s − 1 , begin ( l ) w : begin ( l ) w + w ( l
) s − 1 , :]
[Equation]
Result: estimation of E [ B l ] & V ar
[ B l ] at layer l , l ∈ all layers begin
Calculate µ = E 0 , 1 , 2 [ D ] & σ 2 = V ar 0 , 1 , 2 [ D ] ofﬂine for ep ∈ all epochs do for it ∈ all iterations do
Create virtual samples V ∈ R n v × H 0 × W 0 × C 0 , V ∼ N ( µ, σ ) for l ∈ all layers do if l = 0 :
B l = [ V , B l ] , then feed B l into the network
B s = B l [0 : n v − 1 , : , : , :]
E [ B l ] = E 0 , 1 , 2 [ B s ] , V ar
[ B l ] =
V ar 0 , 1 , 2 [ B s ]
A PPENDIX B E XPERIMENTAL C ONFIGURATION
All the experiments on CIFAR-10 & CIFAR-100 are conducted on a single Nvidia Titan Xp GPU.
We use a weight decay of 0.0002 for all weight layers and all models are trained by 130 epochs.
The initial learning rate is set to 0.1 and it is decreased by 10x at 50, 80, 110 epoch.
During training, we adopt the “random ﬂip left & right” and all the input images are randomly cropped to 32 × 32 .
Each model is trained from scratch for 5 times in order to reduce random variation.
For ImageNet, We use 2 Nvidia Tesla V100 GPUs on DGX station for ResNet-18 and 3 for DenseNet-121.
We use a weight decay of 0.0001 for all weight layers and all models are trained by 100 epochs.
The initial learning rate is set to 0 .
1 / 256 × “ Gradient batch size ” and we decrease the learning rate by 10x at 30, 60, 80, 90 epoch.
During training, all the input images are augmented by random ﬂipping and cropped to 224 × 224 .
We evaluate the top-1 validation error on the validation set using the centered crop of each image.
To reduce random variation, we use the average of last 5 epochs to represent its ﬁnal error rate.
Besides, Winograd (Lavin & Gray, 2016) is applied in all models to speedup training.
A PPENDIX C A NALYSIS OF S PEEDUP
Compute Cycle and Memory Access .
Our proposed approaches can effectively speedup forward pass.
Under the condition that we use s (cid:28) m data to estimation the statistics for each feature, the total accumulation operations are signiﬁcantly reduced from m − 1 to s − 1 .
If using the adder tree optimization illustrated in Section 2.1, the tree’s depth can be reduced from log ( m ) to log ( s ) .
Thus, the theoretical compute speedup for the forward pass can reach log s ( m ) times.
For instance, if the FM size is 56 × 56 with batch size of 128 ( m = 56 × 56 × 128 ), under sampling ratio of 1/32, the compute speedup will be 36.7%.
The total memory access is reduced by m/s times.
For example, when the sampling ratio is 1/32, only 3.1% data need to be visited.
This also contributes a considerable part in the overall speedup.
Speedup in the Backward Pass .
The BN operations in the forward pass have been shown in equation (1)-(3).
Based on the derivative chain rule, we can get the corresponding operations in the backward pass as follows
[Equation]
[Equation] [Equation] Take the BN sampling as an example, the calculation of ∂l ∂x ( k ) i can be modiﬁed as
[Equation]
otherwise
[Equation]
while the others remain the same format.
Here S is the location set for sampled neurons.
For neurons outside S , they didn’t participate in the estimation of the mean and variance in the forward pass, so ∂E
[ x ( k ) ]
∂x ( k ) i and ∂V ar [ x ( k ) ]
∂x ( k ) i equal to zeros.
We should note that although we sampled only s data in the forward pass, the reduction operations in the backward pass still have m length since the neurons outside S also participate in the activation normalization using the estimated statistics from the sampled data.
Therefore, there is no theoretical speedup for the backward pass.
In our experiments, to assemble a dense and efﬁcient addition operation with
[Equation]
for all neurons, the mentioned zeros outside S are pre-padded.
However, it has great potential for memory space and access reduction since the ∂E [ x ( k ) ]
∂x ( k ) i and ∂V ar [ x ( k ) ]
∂x ( k ) i outside S are zeros, which can be leveraged by specialized devices.
Reason for only Speeding up Forward Pass . As shown in equation (1) & (5), although the forward pass only takes 13 reduction operations in BN, it requires nearly 23 time. The underlying reason is that the reduction operations in the backward pass can be parallelized while the second reduction operation in the forward pass used to calculate variance cannot operate until the mean is provided. Regular Execution Pattern .
To approach the theoretical speedup, we suggest using more regular sampling pattern such as the continuous samples in BS and intra-FM rectangular shape and inter-FM shared location in FS.
A regular sampling pattern can improve the cache usage by avoiding random access.
Moreover, the operation in the backward pass corresponding to the sampling operation in the forward pass is “padding” shown in Fig. 1(d).
Under a regular sampling pattern, the padding is block-wise that becomes much easier.
On the other side, intuitively, a static computational graph can be calculated faster and easier deployed on various platforms for (1) the static graph’s pipeline can be optimized once for all before execution, (2) popular deep learning frameworks like TensorFlow (Girija, 2016) are developed as a static computational graph.
For these reasons, we expect the random sampling is achieved through a static computational graph, which is obtained by updating the sampling indexes only per training epoch.
A PPENDIX D I NFLUENCE OF D ECAY R ATE FOR M OVING A VERAGE
During each validation step after certain training iterations, E [ x ( k ) ] & V ar [ x ( k ) ] are replaced with the recorded moving average, which is governed by
[Equation]
(7) where X stands for “ E ” or “ V ar ”, X ma [ x ( k ) ] denotes the moving average, it is the iteration number, and α is the decay rate.
Based on equation (7), wherein the variance of X ma [ x ( k ) ]
it equals to α 2 V ar [ X [ x ( k ) ] it ]+(1 − α ) 2 V ar [ X ma
[ x ( k ) ]
it − 1 ] , if we assume that (1) each estimated value shares the same variance V ar [ X [ x ( k ) ] it ] , (2) they are independent with each other, and (3) the iteration number goes sufﬁciently large, we will get
[Equation]
The above equation reveals that an appropriately smaller α might scale down the estimation error, thus produces better validation accuracy.
To verify this prediction, the experiments are conducted on ResNet-56 over CIFAR-10 and using BS1/128(0.78%) sampling.
As shown in Fig.
11, it’s obvious that there exists a best decay rate setting (here is 0.7) whereas the popular decay rate is 0.9.
The performance also decays when decay rate is smaller than 0.7, which is because a too small α may lose the capability to record the moving estimation, thus degrade the validation accuracy.
This is interesting because the decay rate is usually ignored by researchers, but the default value is probably not the best setting in our context. A PPENDIX E R EDUCTION O PERATIONS IN BN
To further demonstrate that reduction operations are the major bottleneck of BN, we build a pure BN network for proﬁling as shown in Fig.
The network’s input is a variable with shape of [128, 112, 112, 64] which is initialized following a standard normal distribution.
The network is trained for 100 iterations and the training time is recorded.
We overwrite the original BN’s codes and remove the reduction operations in both forward pass and backward pass for contrast test.
We use three different GPUs: K80,
Titan Xp, and Tesla V100 to improve the reliability.
The results are shown in Table 5.
We can see that on all the three GPUs, reduction operations take up to > 60% of the entire operation time in BN.
As a result, it’s solid to argue that the reduction operations are the bottleneck of BN.
[Table]
A PPENDIX F M ICRO -BN E XTENSION
Micro-BN aims to alleviate the diminishing of BN’s effectiveness when the amount of data in each
GPU node is too small to provide a reliable estimation of activation statistics.
Previous work can be classiﬁed to two categories: (1) Sync-BN (Zhang et al., 2018a) and (2) Local-BN (Ba et al., 2016; Wu & He, 2018; Ioffe, 2017; Wang et al., 2018).
The former addresses this problem by synchronizing the estimations from different GPUs at each layer, which induces signiﬁcant interGPU data dependency and slows down training process.
The latter solves this problem by either avoiding the use of batch dimension in the batched activation tensor for statistics or using additional information beyond current layer to calibrate the statistics.
In terms of BN’s efﬁciency preservation, we are facing a similar problem with micro-BN, thus our framework can be further extended to the micro-BN scenario.
In Sync-BN: (1) With FS , each GPU node executes the same patch sampling as normal FS; (2) With BS , we can randomly select the statistics from a fraction of GPUs rather than all nodes; (3) With VDN , the virtual samples can be fed into a single or few GPUs.
The ﬁrst one can just simplify the computational cost within each
GPU, while the last two further optimize the inter-GPU data dependency.
In Local-BN, since the available data for each GPU is already tiny, the BN sampling strategy will be invalid.
Fortunately, the VDN can still be effective by feeding virtue samples into each node.
Training curves in micro-BN scenario.
(ImageNet, ResNet 18)
Experiments .
The normalization in Sync-BN is based on the statistics from multiple nodes through synchronization, which is equivalent to that in Fig.
6 with large batch size for each node.
Therefore, to avoid repetition, here we just show the results on Local-BN with VDN optimization.
We let the overall batch size of 256 breaks down toworkers (each one has only 4 samples for local normalization).
We use “(gradient batch size, statistics batch size)” of (256, 4) to denote the conﬁguration (Wang et al., 2018).
A baseline of (256, 32) with BN and one previous work Group Normalization (GN) (Wu & He, 2018) are used for comparison.
As shown in Fig.
13, although the reduction of batch size will degrade the model accuracy, our VDN can achieve slightly better result (top-1 validation error rate: 30.88%) than GN (top-1 validation error rate: 30.96%), an advanced technique for this scenario with tiny batch size.
This promises the correct training of very large model so that each single GPU node can only accommodate several samples.
A PPENDIX G O RGANIZATION OF THE WHOLE PAPER .