C HANNEL E QUILIBRIUM N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Convolutional Neural Networks (CNNs) typically treat normalization methods such as batch normalization (BN) and rectiﬁed linear function-like activations ( e.g.
ReLU) as building blocks.
Previous work pointed out that learning feature channels with equal magnitudes is important for a CNN to achieve good generalization ability.
However, the above “Norm+ReLU-like” basic block often learns inhibited channels that has small magnitude ( i.e. contributes little to the feature representation), impeding both learning and generalization ability of CNNs.
This problem is seldom explored in the literature.
To mitigate the inhibited channels and encourage channels to contribute equally to the feature representation, we propose a new building block, Channel Equilibrium (CE), which is able to prevent inhibited channels in both experiments and theory.
CE has several appealing properties.
First, CE can be stacked after many different normalization methods such as BN and Group Normalization (GN), as well as integrated into many advanced
CNN architectures such as ResNet and MobileNet V2 to form a series of CE networks (CENets), outperforming existing network architectures.
Second, CE has an interesting connection with the Nash Equilibrium, a well-known solution of a non-cooperative game.
Third, extensive experiments show that CE achieves state-of-the-art results on various challenging benchmarks such as ImageNet and
COCO.
The models and codes will be released.
I NTRODUCTION
Normalization is an important technique for a wide range of tasks such as image classiﬁcation (Ioffe & Szegedy, 2015), object detection (He et al., 2017a;
Wu & He, 2018), and image generation (Miyato et al., 2018).
In recent years, a lot of work improved normalization methods, such as batch normalization (BN) (Ioffe & Szegedy, 2015), layer normalization (LN)
(Ba et al., 2016) and switchable normalization (SN) (Luo et al., 2018).
These methods are often used together with the ReLU-like activation functions such as ReLU (Glorot et al., 2011; Nair & Hinton, 2010), ELU (Clevert et al., 2015) and Leaky ReLU (LReLU) (Maas et al., 2013), making the “Norm+ReLU-like” module become one of the most widely-used building blocks of modern CNNs.
This work investigates and alleviates the inhibited channels emerged in the “Norm+ReLU-like” building block, which consists of a normalization layer and a ReLU-like activation function given by
[Equation]
where subscript n, c, i , and j denote indices of a sample, a channel, height and width of a feature channel respectively
For instance, y ncij indicates output value of the location ( i, j ) in the c -th channel of the n -th sample.
And ˜ x ncij and ¯ x ncij represent normalized channel features and standardized channel features respectively.
γ and β are two vectors of parameters, where each element re-scales and re-shifts the standardized features for each channel c .
Moreover, g ( · ) denotes a ReLU-like activation function.
As pointed out in (Morcos et al., 2018), a CNN that generalizes well would have its channels contributed equally in its feed-forward computations.
We term this desired property “channel equalization”.
However, a recent study disclosed a critical problem of the “Norm+ReLU-like” builiding block, known as “channel collapse”, where certain channels always produce small output values given any input.
For example, the lottery hypothesis (Frankle & Carbin, 2018) claimed that one over-parameterized CNN always contains unimportant channels that contribute little to the network’s prediction.
This paper shows that these unimportant channels are inhibited channels , which usually
BN
BN+CE
LN
LN+CE      70 70.
Inhibited
[Figure]
Ratio Top 1 Error
[Figure]
ELU
CE+ELU
LReLU
CE+LReLU     56.
[Figure]
Top
[Figure]
associated with small values of γ in Eqn.(1) or small values of the channel features (In this paper inhibited channel is deﬁned as channel whose all feature values are less than 1 e − 2 ).
For example, as shown in Fig.1(a,b), the inhibited channels exist in many “Norm+ReLU-like” basic block such as ‘LN+ReLU’, ‘BN+ELU’ and ‘BN+ReLU’.
This phenomenon has motivated many investigations to directly remove these inhibited channels, such as network slimming (Liu et al., 2017; Yu et al., 2018) and channel pruning (He et al., 2017b).
However, disequilibrium among channels caused by the inhibited channels would do harm to generalization ability of the network and simply removing the inhibited channels that are inactive during training does not help improve learning capacity.
Instead of removing the inhibited channels, we present an alternative perspective by proposing a novel building block, named Channel Equilibrium (CE), to recover and equalize the inhibited channels by encouraging channels to contribute equally in the feature representation learning process, thus enhancing the representation and generalization of CNNs.
To this end, a key observation from
Eqn.(1) is that the dependency (covariance) matrix of all the feature channels after normalization is scaled by γγ T .
Let this covariance matrix be Σ .
We will see that by applying a decorrelation operator, i.e. Σ − 12 , we can not only effectively eliminate the correlations between computations of channel features and the magnitude of γ , but also equalize the channels’ magnitudes (Barlow et al., 1961; Bengio & Bergstra, 2009).
This operator enables all the channels to play an equal role in the computations of a CNN, improving its generalization ability.
For example, as shown in Fig.1, the
VGGNet (Simonyan & Zisserman, 2014) equipped with CE is able to effectively prevent inhibited channels and achieves channel equalization in various of “Norm+RelU-like” blocks, consistently improving their recognition performance.
The main contributions of this work are three-fold.
First, we introduce an efﬁcient and effective building block, Channel Equilibrium (CE), which encourages channel equalization and enhances representation learning of CNNs.
Second, CE blocks can be stacked after common normalizers and plugged into various advanced architectures, consistently improving their performance by a large margin.
For example, CE can be integrated into ResNet and MobileNet V2, forming a series of
CE-Networks by replacing the ordinary ‘BN-ReLU’ block by ‘BN-CE-ReLU’ block, which merely introduces subtle extra computational complexity.
As a result, the CE-ResNet50 and CE-MobileNet
V2 outperform their counterparts by 1 . 7 % and 2 .
1 % top-1 accuracy with nearly the same FLOPs.
We also show that combining CE with synchronization across GPUs increases the AP metric on the
MS-COCO dataset to 42 . 0 , surpassing its counterpart by 3 . 4 .
Third, the learned equalized feature representation of CENet can be better transferred to many other tasks like object detection and segmentation.
R ELATED W ORK
Channel equalization.
Channel equalization indicates that channels in a layer of CNNs contribute equally to network’s computation.
The success of the two most commonly used regularization techniques, i.e. BN (Ioffe & Szegedy, 2015) and Dropout (Srivastava et al., 2014), is attributed to channel or neuron equalization.
For example, Mianjy et al.
(2018) showed that Dropout makes the norm of incoming/outgoing weight vectors of all the hidden nodes equal, indicating a kind of equalization between neurons.
Moreover, Morcos et al.
(2018) pointed out that BN implicitly discourages single direction reliance, indicating that equalizing different channels is able to enhance the generalization of learned feature representation.
Note that the squeeze-and-excitation (SE) network (Hu et al., 2018) is a pioneer work that explicitly modeling interdependencies among channels by investigating network design.
However, SE selectively emphasizes informative channels and suppresses less useful ones.
In contrast, the proposed CE block encourages all the channels to play an equal role in network’s computation, which, as will be shown, can be linked with Nash Equilibrium.
More related work on sparsity in ReLU and normalization methods are provided in Sec.
A of Appendix.
M ETHOD
In this section, we ﬁrst review the normalization method and then introduce the proposed Channel
Equilibrium (CE) block.
CE contains two complementary branches, i.e. batch decorrelation (BD) and adaptive instance inverse (AII).
We show how BD and AII beneﬁt from each other through parameter γ and how CE is linked with Nash Equilibrium.
Notations.
For CNNs, we use x ∈ R N × C × H × W to represent the feature in a layer, speciﬁcally, x ncij denotes a pixel ( i, j ) in the c -th channel of the n -th sample.
Sometimes, we ignore the subscript ‘ n ’ and denote it as x cij for clarity of notation.
x nij ∈ R C is obtained by stacking elements in all channels of x ncij into a column vector.
Diag( · ) returns a matrix with the given diagonal and zero offdiagonal entries, and diag( · ) extracts the diagonal of the given matrix.
γ, β ∈ R C are normalization parameters.
3. O VERVIEW OF N ORMALIZATION
Normalization is usually employed after convolution layers to stabilize the training of CNNs.
Given a hidden feature x ∈ R N × C × H × W , a normalizer ﬁrst standardizes it to ¯ x , and then maps it to ˜ x by an afﬁne transformation, as written by
[Equation]
where s ∈ Ω = { IN , BN , LN , · · · } indicates a normalizer and µ s , σ s are the mean and standard deviation of the given normalizer.
For simplicity, (cid:15) in original formulation is omitted (Ioffe & Szegedy, 2015).
From Eqn.(2), we claim that normalization would lead to an unequal feature representation in a channel basis, based on the fact that common-used normalizers like IN and BN are performed channel-wisely.
It is known that importance of channels are quantiﬁed by the learned parameters γ with its magnitude since channel features are scaled by γ in a channel basis.
Previous work (Frankle & Carbin, 2018; Mehta et al., 2019) revealed that some inhibited channels emerge when the associated γ or feature map gets small (Mehta et al., 2019; Lu et al., 2019).
Obviously, the inhibited channels would cause disequilibrium among channels, resulting in limited generalization ability.
To alleviate such a disequilibrium, Channel Equilibrium (CE) block is proposed in the following section.
3. C HANNEL E QUILIBRIUM (CE) B LOCK
A Channel Equilibrium (CE) block is a computational unit which aims to equalize feature representation capacity among channels.
To this end, decorrelation method is adopted.
Different from previous methods (Huang et al., 2018; 2019) decorrelating features by a single batch estimated covariance matrix Σ , the proposed method brings in an adaptive instance variance, S n , on the diagonal of the covariance matrix Σ , considering that channel dependency is speciﬁc to each input (Hu et al., 2018), as formulated in the following,
[Equation]
where the subscript n is the sample index, λ ∈ (0 , 1) is a trainable ratio used to switch between batch and instance statistics, F : R C → R C is a transformation conditioned on the current input ˜ x and σ 2 (˜ x n ) computes instance variance of ˜ x n within each channel.
On the issue of channel disequilibrium, CE block works by decorrelating feature maps after normalization using D − 12 n .
Further, the Jensen inequality for matrix functions (Peˇcari´c, 1996) can be employed to obtain a relaxed decorrelation operator D − 12 n :
[Equation]
where A (cid:22) B indicates B − A is semi-deﬁnite positive.
We introduce this relaxation for the following two reasons.
(1) Computation reduction.
It allows less computational cost for each training step since the relaxed form only needs to calculate the inverse of square root Σ − 12 once, and the other branch Diag( S n ) − 12 is easy to compute.
(2) Inference acceleration.
Σ − 12 is a moving-average statistic in inference which can be absorbed into previous layer, therefore, enabling fast inference.
Note that Eqn.(4) transforms the combination of covariance and adaptive instance variance into the combination of their inverse square roots.
In the following, we refer Σ − 12 in Eqn.(4) as batch decorrelation (BD) and refer [Diag( S n )] − 12 as adaptive instance inverse (AII).
The former decorrelates channels by a batch covariance, while the latter adjusts the extend of inverse for each channel and instance in an adaptive manner.
Integrating both of them yields the forward representation of CE block:
[Equation]
where p nij ∈ R C denotes the output of CE, as illustrated in Fig.2(b).
Note that CE is performed after the normalization layer, BN is taken as an example to introduce these two branches in the following sections.
3.2. B ATCH D ECORRELATION (BD)
Although a lot of previous work (Huang et al., 2018; 2019; Pan et al., 2019) has investigated whitening method using covariance matrix, all of them are applied after the convolution layer.
Thus, inhibited channels still exist since whitened channel features are also scaled by γ .
Instead, decorrelation in CE is applied after the normalization layer to equalize magnitude of all channels.
Consider a tensor ˜ x after a BN layer, it can be reshaped as ˜ x ∈ R C
× M and M = N · H · W .
Then the covariance matrix Σ of ˜ x can be written as (details are presented in Sec.
B of Appendix)
[Equation]
(6) where ¯ x is a standardized feature with zero mean and unit variance and (cid:12) indicates elementwise multiplication.
Eqn.(6) implies that the covariance matrix Σ of ˜ x can be decomposed into two parts.
The ﬁrst part depends on normalization parameter γ and the second part becomes correlation matrix of ˜ x .
It is observed that Σ ij , which represents dependency between i -th channel and j -th channel, is scaled by γ i
γ j after BN is applied.
The Batch Decorrelation (BD) branch requires computing Σ − 12 , which is usually related to eigendecomposition or SVD and involves heavy computation (Huang et al., 2018).
Instead, here we adopt an efﬁcient approach, i.e., Newton’s Iteration to obtain Σ − 12 (Bini et al., 2005; Higham, 1986).
Given covariance matrix Σ , Newton’s Iteration calculates Σ − 12 by the following iterations:
[Equation]
(7) where T is the iteration number ( T = 3 in our experiments).
Note that the convergence of Eqn.(7) is guaranteed if (cid:107) Σ (cid:107) 2 <
(Bini et al., 2005).
To this end, Σ is normalized as Σ / tr(Σ) where tr( · ) is the trace operator (Huang et al., 2019).
In this way, the normalized covariance matrix is written as Σ = γγ T (cid:107) γ (cid:107) 22 (cid:12) 1 M ¯ x ¯ x T .
To sum up, the batch decorrelation branch ﬁrstly calculates a normalized covariance matrix and then applies Newton’s Iteration to obtain its inverse square root, reducing lots of computational cost compared with SVD decomposition in the training stage.
Furthermore, BD branch can be merged into convolutional layers in the inference stage, which adds extra computation marginally.
3.2. A DAPTIVE INSTANCE I NVERSE (AII)
Channel dependencies are speciﬁc to each sample.
Consequently, a conditional decorrelation is desired for each sample.
The adaptive instance inverse (AII) branch only uses diagonal entries to model channel dependencies, as shown in Eqn.(3).
Since a diagonal matrix can be inverted easily, this approach can avoid the computation of Eqn.(4).
To construct the AII branch, we analyze its input (the output of a BN layer), which is formulated as ˜ x ncij = γ c ¯ x ncij + β c .
The input of AII is the instance variance of each channel (details are provided in Appendix Sec. B),
[Equation] [Equation]
where σ 2IN and σ 2BN represent the variances in IN and BN respectively.
The ratio of them measures the relative ﬂuctuation of how much the instance statistic are deviated from the batch-estimated statistic.
Similar to Eqn.(6), the input of AII is also scaled by γ 2 c .
The AII branch takes σ 2 nc as input and computes an adaptive instance inverse, i.e. [Diag( S n )] − 12 .
It needs to satisfy two requirements.
First, as is desired, dependencies among channels should be embedded in [Diag( S n )] − 12 for each sample.
Second, the output of AII should have the same philosophy as inverse square root of variance or covariance in BD branch.
To achieve this, a reparameterization trick is employed to generate adaptive instance inverse.
Let s be an estimate of variance, the AII branch can be reparameterized as below,
[Equation]
where δ 1 and δ 2 are ReLU and sigmoid activation function respectively, W 1 ∈
R Cr × C and W 2 ∈
R C ×
Cr and r is reduction ratio, s ∈ R denotes variance of all elements in ˜ x , which is a batch statistic in training and is obtained using moving average for inference.
˜ F ( σ 2 n ) ∈ (0 , 1) C is treated as a gating mechanism in order to control the strength of instance inverse for each channel.
Following the best practice (Hu et al.
; Cao et al.
in characterizing channel relationships, ˜ F is expressed by a bottleneck architecture that is able to model channel dependencies and limit model complexity.
Layer normalization (LN) is used inside the bottleneck transform (before ReLU) to ease optimization.
It is seen from Eqn.(9) that s − 12 represents the quantity of inverse square root of variance and ˜ F ( σ 2 n ) regulates the extend of variance inverse.
˜ F maps the instance variance to a set of channel weights.
In this sense, the AII branch intrinsically introduces channel dependencies conditioned on each input.
3. D ISCUSSIONS
Instantiations.
Our CE block can be integrated into various advanced architectures, such as ResNet,
VGGNet, ShufﬂeNet V2 and MobileNet V2, by inserting it in ’norm+ReLU-like’ building block.
The CE block is described in Fig.2(b).
As discussed earlier, CE processes incoming features after the normalization layer by combining two branches, i.e. batch decorrelation (BD) and adaptive instance inverse (AII).
Compared with SE block in Fig.2(a), the proposed CE block combines both instance and batch statistics, and it can consequently model dependencies among channels better.
A series of CENets can be constructed by integrating CE block into various advanced CNN architectures.
For example, we consider the residual networks (ResNet).
The core unit of the ResNet is the residual block that consists of ‘ 1 × 1 ’, ‘ 3 × 3 ’ and ‘ 1 × 1 ’ convolution layers, sequentially.
Since the
CE block is expected to help channel equalization, it would beneﬁt from larger number of channels.
Therefore, we employ CE block in the last ‘ 1 × 1 ’ convolution layer by plugging the CE module before ReLU non-linearity, as shown in Fig.2(c).
As for CE-MobileNet V2, since the last ‘ 1 × 1 ’ convolution layer in the bottleneck is not followed by a ReLU activation, we insert CE in the ‘ 3 × 3 ’ convolution layer that also has a largest number of channels in the bottleneck.
Following similar strategies, CE is further integrated into ShufﬂeNet V2 to construct CE-ShufﬂeNet V2.
We provide extensive experiments evaluating all these CENets in Sec.4 and computational details in training and inference in Sec.
F of Appendix.
[Figure]
Equivalent γ .
Here we show how BD and AII beneﬁt from each other through parameter γ .
First, we disclose the mechanism in preventing inhibited channels behind the BD branch.
Previous work (He et al., 2017b; Yu et al., 2018; Frankle & Carbin, 2018) revealed that γ in BN can be used to prune less unimportant channels, implying that the representational power of feature map largely depends on the magnitude of γ .
Combining Eqn.(4) and Eqn.(5), the output of BD can be expressed as p BD nij =
Diag(Σ − 12 γ )¯ x nij + Σ − 12 β .
Comparing with Eqn.(2), an equivalent γ can be deﬁned as ˆ γ =
Σ − 12 γ for BD branch.
The proposition 1 shows that BD explicitly increases the magnitude of ˆ γ in a feed-forward way, encouraging all channels to contribute to the feature learning process.
The representational power is thus boosted in a channel basis.
We provide the proof of proposition 1 in
Sec.
C of Appendix.
Furthermore, the original γ in BN is also implicitly enlarged.
It can be seen from Eqn.(6), a sufﬁcient small γ c can cause degradation of the covariance matrix and then the convergence of Newton’s iteration (Bini et al., 2005) cannot be guaranteed.
As a result, once the network converges, γ c is not supposed to degrade.
This will in turn bring many beneﬁts to AII branch.
Eqn.(8) shows that the input of AII is proportional to γ , meaning that the features fed into AII branch are enlarged as γ increases.
In this way, a bottleneck architecture in AII can learn more compact global information and model channel dependencies better.
Proposition 1.
Let Σ be covariance matrix of feature maps after batch normalization.
Assume that
Σ k = Σ − 12 , ∀ k
= 2 , 3 , · · · , T , then (cid:107) ˆ γ (cid:107) 1 > (cid:107) γ (cid:107) 1 .
Especially, we have | ˆ γ i | >
| γ i | .
Connection with Nash Equilibrium.
We show an interesting connection between the proposed
CE block and the well-known Nash Equilibrium in game theory (Leshem & Zehavi, 2009).
To be speciﬁc, we bring novel insights on normalization from an optimization perspective.
Suppose each channel obtains its output by maximizing capacity available to itself under some constraints.
Especially, we restrict that each channel has a maximum budget and all the outputs are non-negative.
Further, if we consider dependencies among channels, the channels are thought to play a noncooperative game, named Gaussian interference game which admits a unique Nash Equilibrium solution (Laufer et al., 2006).
In Sec.
D of Appendix, we present the detailed construction of Gaussian interference game in the context of CNNs.
It is worth noting that when all the outputs are activated (larger than 0), this Nash Equilibrium solution has an explicit expression.
Under some mild approximations, it can be shown that the explicit Nash Equilibrium solution can surprisingly match the representation of CE in Eqn.(5).
It shows that decorrelating features after normalization layer can be connected with Nash Equilibrium, implying that the proposed CE block indeed encourages every channel to contribute to the network’s computation.
We present detailed explanations about the connection between CE and Nash Equilibrium in Sec.
D of Appendix.
[Table] [Table]
EXPERIMENTS
We evaluate our methods on two basic vision tasks, image classiﬁcation on ImageNet and object detection/segmentation on COCO, where we demonstrate the effectiveness of the CE block.
4. I MAGE C LASSIFICATION ON I MAGE N ET
We ﬁrst evaluate CE on the ImageNet benchmark.
The training details are illustrated in Sec.
F of
Appendix.
Performance comparison on ResNet.
We evaluate on representative residual network structures including ResNet18, ResNet50 and ResNet101.
The CE-ResNet is then compared with baseline (plain
ResNet) and SE-ResNet.
For fair comparisons, we use publicly available code and re-implement baseline models and SE modules with their respective best settings in a uniﬁed Pytorch framework.
To save computation, the CE blocks are selectively inserted into the last normalization layer of each residual block.
Speciﬁcally, for ResNet18, we plug the CE block into each residual block.
For
ResNet50, CE is inserted into all residual blocks except for those layers with 2048 channels.
For
ResNet101, the CE blocks are employed in the ﬁrst seven residual blocks.
As shown in Table 1, our proposed CE outperforms the BN baseline and SE block by a large margin with little increase of GFLOPs.
Concretely, CE-ResNet18, CE-ResNet50 and CE-ResNet101 obtain top-1 accuracy increase of 1 . 5% , 1 .
7% and 1 . 0% compared with the corresponding plain ResNet architectures.
The CE-ResNet50 even outperforms the plain ResNet101 (78.0).
We plot training and validation loss during the training process for ResNet50, SE-ResNet50 and CE-ResNet50 in
Sec.
E of Appendix.
We also analyze the complexity of BN, SE, and CE in terms of GFLOPs, GPU and CPU running time.
We evaluate the inference time 1 with a mini-batch of 32.
In term of GFLOPs, the CEResNet18, CE-ResNet50, CE-ResNet101 has only 0 .
242% and 0 .
241% relative increase in GFLOPs compared with plain ResNet.
Additionally, the CPU and GPU inference time of CENet is nearly the same with SENet.
Performance comparison on light-weight networks .
We further verify the effectiveness of our proposed CE in two representative light-weight networks, MobileNet V2 and ShufﬂeNet V2.
The results of comparison are listed in Table 2.
It is seen that CE blocks bring conspicuous improvements
[Figure]
[Table]
[Figure]
weight decays.
It is observed that the networks trained with the proposed BD and CE were consistently and substantially more robust to the increasing strength of weight decay than those trained with single BN.
in performance at a minimal increase in computational burden on mobile settings.
For MobileNet
V2, we see that CE blocks even improves top-1 accuracy of baseline by 2 . 1% .
Other Normalizers .
In addition to BN, CE is also effective for other normalization technologies, since inhibited channels emerges in many well-known normalizers as shown in Fig.1.
To prove this, we conduct experiments using ResNet-50 under different normalizers including batch normalization (BN), group normalization (GN), instance normalization (IN), and layer normalization (LN).
For these experiments, we stack CE block after the above normalizers to see whether CE helps other normalization methods.
As shown in Table 3, our CE generalize well over different normalization technology, improving the performance by 0.6-1.8 top-1 accuracy.
4. A NALYSIS OF CE
In this section, we ﬁrst demonstrate that CE is able to equalize the importance of all channels and then analyze the effects of BD and AII branches separately on CIFAR10 and ImageNet datasets.
More discussions about CE are provided in Sec.
E of Appendix.
CE is effective in channel equalization.
In Fig.1, we have demonstrated that CE is able to alleviate inhibited channels, which is a necessary condition of channel equalization.
Here, we further show that the ability that CE can prevent inhibited channels is robust to a wide range of strength of weight decay.
As shown in Fig.3(c,d), CE prevents inhibited channels and retains higher performance under different strengths of weight decay.
Next, we verify whether CE can help channel equalization by an ablation approach used in Morcos et al.
Typically, the importance of a single channel to the network’s computation can be measured by the relative performance drop once that channel is removed (clamping activity a feature map to zero).
In this regard, the more reliant a network is on a small set of channels, the more quickly the accuracy will drop if those channels are ablated.
On the contrary, if the importance of channels to the network’s computation are more equal, the accuracy will drop more gently.
With this powerful technique, we see how ResNet50 and MobileNet V2 with CE blocks respond to cumulative random ablation of channels.
We plot the ablation ratio versus the topaccuracy in Fig.3(a,b).
As we can see, our CE block is able to resist the cumulative random ablation of channels on both ResNet50 and MobileNet V2, showing that CE can effectively equalize the importance of channels.
For example, the top-1 accuracy of our CE-ResNet50 is 1 .
7 higher
[Table]
than the original ResNet50 if no channels are ablated, but when 70% channels are ablated, CEResNet50 still obtain 23 .
0 top-1 accuracy, while the original ResNet50 gets only 4 .
6 top-1 accuracy.
[Figure]
BD is able to mitigate
Here we investigate this property experimentally with a single BD branch.
The inhibited channels ratio are measured by the percentage of feature maps whose values are less than 1e-2.
Fig.3(d) shows inhibited channels ratio under a wide range of weight decay for BN, BD and CE.
It is observed that the top-1 accuracy of VGGNet with BN drops signiﬁcantly as the weight decay increases, but BD can reduce accuracy drop.
For example, when the weight decay is 1e-3, the top-1 accuracy of BD is only 0.1 higher than BN, but when the weight decay reaches to 1e-2, BD is 4.95 higher.
Moreover, the inhibited channels ratio of CE is even lower than BD while the top-1 accuracy is higher, which can demonstrate that CE can strengthen the effect of alleviation of inhibited channels compared with single BD.
AII helps CE learn preciser feature representation.
First, as discussed in Sec.3.3, AII beneﬁts from BD such that the features fed into AII branch are more informative.
To see this, we train
ResNet50 with a single AII or CE branch, termed AII-ResNet50 or CE-ResNet50.
We do principal component analysis (PCA) on the inputs of AII branch in AII-ResNet50 and the counterpart in CEResNet50 and plot the box chart of principal components.
As is shown in Fig.4, the input of AII branch in CE-ResNet50 gets much lower means and variances, meaning that the input feature has more valid basis and thus more informative.
From the Grad-CAM visualization provided in Sec.
E of
Appendix, we ﬁnd that AII helps CE learn preciser feature representation.
BD and AII are complementary .
Here, we verify that BD and AII are complementary to each other.
We train plain ResNet50, BD-ResNet50, AII-ResNet50, and CE-ResNet50 for comparison.
The top-1 accuracy is reported in Table 4.
It is observed that the BD-ResNet50 and AII-ResNet are 0.4 and 0.7 higher than the plain ResNet-50 respectively.
However, when they are combined, the top-1 accuracy improves by 1.7, higher than combined accuracy increase (1.1), which demonstrates that they beneﬁt from each other.
4. O BJECT D ETECTION AND I NSTANCE S EGMENTATION ON COCO
We assess the generalization of our CE block on detection/segmentation track using the COCO dataset ( Lin et al.
We train our model on the union of 80k training images and 35k validation images and report the performance on the mini-val 5k images.
Mask-RCNN is used as the base detection/segmentation framework.
The standard COCO metrics of Average Precision (AP) for bounding box detection (APbb) and instance segmentation (APm) is used to evaluate our methods.
In addition, we adopt two common training settings for our models, (1) freezing the vanilla batch normalization and channel equilibrium layer and (2) updating parameters with the synchronized version.
For vanilla BN and CE layers, all the gamma, beta parameters, and the tracked running
[Table]
statistics are frozen.
In contrast, for the synchronized version, the running mean and variance for batch normalization and the covariance for CE layers are computed across multiple GPUs.
The gamma and beta parameters are updated during training while ˜ F and λ are frozen to prevent overﬁtting.
We use MMDetection training framework with ResNet50/ResNet101 as basic backbones and all the hyper-parameters are the same as Chen et al.
Table 5 shows the detection and segmentation results.
The results show that compared with vanilla BN, our CE block can consistently improve the performance.
For example, our ﬁne-tuned CE-ResNet50 is 2.2 AP higher in detection and 2.7 AP higher in segmentation.
For the sync BD version, CE-ResNet50 gets 42.0 AP in detection and 37.5 AP in segmentation, which is the best performance for ResNet50 to the best of our knowledge.
To sum up, these experiments demonstrate the generalization ability of CE blocks in other tasks.
C ONCLUSION
In this paper, we presented a novel network block, termed as Channel Equilibrium (CE).
The CE block conditionally decorrelates feature maps after normalization layer by switching between batch decorrelation branch and adaptive instance inverse branch.
We show that CE is able to explicitly alleviate the inhibited channels and help channel equalization, enhancing the representational power of a neural network in a feed-forward way.
Speciﬁcally, CE can be stacked between the normalization layer and the ReLU function, making it ﬂexible to be integrated into many advanced CNN architectures.
The superiority of CE blocks has been demonstrated on the task of image classiﬁcation and instance segmentation.
We hope that the analysis of channel equalization in CE could bring a new perspective for future work in architecture design
Appendix A M ORE RELATED WORK
Sparsity in ReLU.
An attractive property of ReLU (Sun et al., 2015; Nair & Hinton, 2010) is sparsity, which brings potential advantages such as information disentangling and linear separability.
However, Lu et al.
(2019) and Mehta et al.
(2019) pointed out that some ReLU neurons may become inactive and output 0 values for any input.
Previous work tackled this issue by designing new activation functions, such as ELU (Clevert et al., 2015) and Leaky ReLU (Maas et al., 2013).
Recently, Lu et al.
(2019) also tried to solve this problem by modifying initialization scheme.
Different from these work, we focus on explicitly preventing inhibited channel in a feed-forward way by the proposed CE blocks.
Normalization and decorrelation.
There are many practices on normalizer development, such as
Batch Normalization (BN) (Ioffe & Szegedy, 2015), Group normalization (GN)
(Wu & He, 2018) and Switchable Normalization (Luo et al., 2018).
A normalization scheme is typically applied after a convolution layer and contains two stages: standardization and rescaling.
Another type of normalization methods not only standardizes but also decorrelates features, like DBN (Huang et al., 2018), IterNorm (Huang et al., 2019) and switchable whitening (Pan et al., 2019).
Despite their success in performance improvement, little is explored about relation between those methods and inhibited channels.
Fig.1 shows that inhibited channels emerges in VGGNet where ‘BN+ReLU’ or ‘LN+ReLU’ are used.
Unlike previous decorrelated normalizations where decorrelation operation is applied after a convolution layer, our CE explicitly decorrelates features after normalization.
B C OMPUTATION DETAILS IN ’BN-CE-R E LU’ BLOCK
As discussed before, CE processes incoming features after normalization layer by combining two branches, i.e. batch decorrelation and adaptive instance inverse.
The former computes a covariance matrix and the latter calculates instance variance.
We now take ’BN-CE-ReLU’ block as an example to show the computation details of statistics in it.
Given a tensor x ∈ R N × C × H × W , the mean and variance in IN (Ulyanov et al., 2016) are calculated as:
[Equation]
Hence, we have µ IN , σ 2IN ∈ R N × C .
Then, the statistics in BN can be reformulated as follows:
[Equation] [Equation]
Then, we have µ BN = E [ µ IN ] and σ 2BN
=
E
[ σ 2IN ] + D[ µ IN ] , where E [ · ] and D[ · ] denote expectation and variance operators over N samples.
Further, the input of AII is instance variance of features after BN, which can be calculated as follows:
[Equation]
At last, the output of BN is ˜ x ncij =
γ c ¯ x ncij + β c , then the entry in c-th row and d-th column of covariance matrix Σ of ˜ x is calculated as follows:
[Equation]
where ρ cd is the element in c-th row and j-th column of correlation matrix of ¯ x .
Thus, we can write Σ into vector form: Σ = γγ T (cid:12) 1 M ¯ x ¯ x T if we reshape ˜ x to ˜ x ∈ R C × M and M = N · H · W .
C P ROOF OF PROPOSITION
Proposition 1.
Let Σ be covariance matrix of feature maps after batch normalization.
Assume that
Σ k = Σ − 12 , ∀ k
= 2 , 3 , · · · , T , then (cid:107) ˆ γ (cid:107) 1 > (cid:107) γ (cid:107) 1 .
Especially, we have | ˆ γ i | > | γ i |
Proof.
Since Σ k = Σ − 12 , ∀ k = 2 , 3 , · · · , T , we have Σ k γ = 12 Σ k − 1 (3 I − Σ 2 k − 1 Σ) γ
=
Σ
k − 1 γ .
Therefore, we only need to show (cid:107) ˆ γ (cid:107)
= (cid:107) Σ T γ (cid:107) 1 = · · · = (cid:107) Σ 2 γ (cid:107) 1 > (cid:107) γ (cid:107) 1 .
Now, we show that for k = 2 we have (cid:13)(cid:13) 12 (3 I − Σ) γ (cid:13)(cid:13) 1 > (cid:107) γ (cid:107) 1 .
From Eqn.(6), we know that Σ = γγ T (cid:107) γ (cid:107) 22 (cid:12) ρ where ρ is the correlation matrix of ˜ x and − 1 ≤ ρ ij ≤ 1 , ∀ i, j ∈
[ C ] .
Then, we have
[Equation]
Note that | 3 − (cid:80) Cj
[Equation]
ij
[Equation]
22 |
[Equation]
3 − |
[Equation]
Cj γ 2 j ρ ij
[Equation]
γ
[Equation]
22 | ≥ 3 − (cid:80) Cj γ 2 j (cid:107) γ (cid:107)
=
2 , where the last equality holds iff ρ ij = 1 , ∀ i, j ∈
[ C ] .
However, this is not the case in practice.
Hence we have
[Equation] [Equation]
Therefore, we have (cid:107) ˆ γ (cid:107) 1 > (cid:107) γ (cid:107) 1 .
Here completes the proof.
D C ONNECTION BETWEEN CE BLOCK AND N ASH E QUILIBRIUM
We ﬁrst introduce the deﬁnition of Gaussian interference game in context of CNN and then build the connection between a CE block and Nash Equilibrium.
For clarity of notation, we omit the subscript n for a concrete sample.
Let the channels 1 , 2 , · · · , C operate over H × W pixels.
Assume that the C channels have dependencies G = { g cd ( i, j ) } C,Cc,d =1 .
Each pixel is characterized by a power gain h cij ≥ 0 and channel noise strength σ c > 0 .
In context of normalization, we suppose h cij =
¯ x cij + δ where ¯ x cij is standardized pixel in Eqn.(2) and δ is sufﬁciently large to guarantee a non-negative power gain.
Assume that c-th channel is allowed to transmit a total power of P c and we have (cid:80) H,Wi,j =1 p cij
= P c .
Besides, each channel can transmit a power vector p c
= ( p c 11 , · · · , p cHW ) .
Since normalization layer is often followed by a ReLU activation, we restrict p cij ≥ 0 .
What we want to maximize the capacity transmitted over the c-th channel, ∀ c ∈
[ C ] , then the maximization problem is given by:
[Equation]
where C c is the capacity available to the c-th channel given power distributions p 1 , p 2 , · · · , p C .
In game theory, C channels and solution space of { p cij } C,H,Wc,i,j =1 together with pay-off vector C =
( C 1 , C 2 , · · · , C C ) form a Gaussian interference game G .
Different from basic settings in G , here we do not restrict dependencies g cd to (0 , 1) .
It is known that G has a unique Nash Equilibrium point whose deﬁnition is given as below,
Deﬁnition 1.
An C-tuple of strategies ( p 1 , p 2 , · · · , p C ) for channels 1 , 2 , · · · , C respectively is called a Nash equilibrium iff for all c and for all p ( p a strategy for channel c )
[Equation]
) i.e., given that all other channels d (cid:54) =
c use strategies p d , channel c best response is p c .
Since
C 1 , C 2 , · · · , C C are concave in p 1 , p 2 , · · · , p C respectively, KKT conditions imply the following theorem.
Theorem 1.
Given pay-off in Eqn.(17), ( p ∗ 1 , · · · , p ∗ C ) is a Nash equilibrium point if and only if there exist v 0 = ( v 10 , · · · , v C 0 )
(Lagrange multiplier) such that for all i ∈
[ H ] and j ∈ [ W ] ,
[Equation]
Proof.
The Lagrangian corresponding to minimization of − C c subject to the equality constraint and non-negative constraints on p cij is given by
L c = −
[Equation]
Differentiating the Lagrangian with respect to p cij and equating the derivative to zero, we obtain
[Equation]
Now, using the complementary slackness condition v cij 1 p cij =
and v cij 1 ≥ 0 , we obtain condition (19).
This completes the proof.
By Theorem 1, the unique Nash Equilibrium point can be explicitly written as follows when p ∗ cij >
[Equation]
[Figure]
where p ∗ ij , h ij , σ ∈ R C and v 0 ∈ R C are Lagrangian multipliers corresponding to equality constraints.
Note that a approximation can be made using Taylor expansion as follow: − σ c h cij =
σ c (2 + h cij + O
((1 + h cij ) 2 )) .
Thus, a linear proxy to Eqn.(22) can be written as
[Equation] [Equation]
Let G =
[ D n ] 12 , γ = σ and β =
Diag( v 0 ) − 1 diag( G )
+ (2 + δ ) σ , Eqn.(23) can surprisingly match
CE unit in Eqn.(5), implying that the proposed CE block indeed performs a mechanism on channel equalization.
In Gaussian interference game, σ is known and v 0 can be determined when budget
P c ’s are given.
However, γ and β are learned by SGD in deep neural networks.
E E XPERIMENTS E. T RAINING AND VALIDATION CURVES
We plot training and validation loss during the training process for ResNet50, SE-ResNet50 and
CE-ResNet50 in Fig.5.
We can observe that CE-ResNet50 consistently have lower training and validation errors over the whole training period, indicating that CE improves both learning capacity and generalization ability.
E. M ORE DISCUSSION ABOUT CE
As discussed in related work in Sec.
A, many methods have been proposed to improve normalizers and ReLU activation.
The ablation approach in Morcos et al.
(2018) is used to see whether and how these methods help channel equalization.
We demonstrate the effectiveness of CE by answering the following questions.
Do other ReLU-like activation functions help channel equalization?
Two representative improvements on ReLU function, i.e. ELU (Clevert et al., 2015) and LReLU (Maas et al., 2013), are employed to see whether other ReLU-like activation functions can help channel equalization.
We plot the cumulative ablation curve that depicts ablation ratio versus the top-1 accuracy on CIFAR10 dataset in Fig.6(a).
The baseline curve is ’BN+ReLU’.
As we can see, the top-1 accuracy curve of ’BN+LReLU’ drops more gently, implying that LReLU helps channel equalization.
But ’ELU+ReLU’ has worse cumulative ablation curve than ’BN+ReLU’.
By contrast, the proposed CE block improves the recognition performance of ’BN+ReLU’ (higher top-1 accuracy) and promotes channel equalization most (the most gentle cumulative ablation curve).
Do the adaptive normalizers help channel equalization?
We experiment on a representative adaptive normalization method (i.e. SN), to see whether it helps channel equalization.
SN learns to
[Figure]
(a)
[Figure]
T o p 1 A cc
BN
CE
SN (b)
[Figure]
select an appropriate normalizer from IN, BN and LN for each channel.
The cumulative ablation curves are plotted on ImageNet dataset with ResNet-50 under blocks of ’BN+ReLU’, ‘SN+ReLU’ and ’BN+CE+ReLU’.
As shown in Fig.6(b), SN even does damage to channel equalization when it is used to replace BN.
However, ’BN+CE+ReLU’ shows the most gentle cumulative ablation curve, indicating the effectiveness of CE block in channel equalization.
Compared with SN, ResNetwith CE block also achieves better top-1 accuracy (78.3 vs 76.9), showing that channel equalization is important for block design in a CNN.
Integration strategy of CE block.
We put CE in different position of a bottleneck in ResNet50, which consists of three ”Conv-BN-ReLU” basic blocks.
The channel of the third block is 4 times than that of the second one.
We compare the performance of CE-ResNet50 by putting CE in the second block (CE2-ResNet50) or the third block (CE3-ResNet50).
As shown in Table 6, the topaccuracy of CE3-ResNet-50 outperforms CE2-ResNet50 by 0.4, which indicates that our CE block beneﬁts from larger number of channels.
E. G RAD - CAM VISUALIZATION
We claim that AII learns adaptive inverse of variance for each channel in a self-attention manner.
Fed into more informative input, AII is expected to make the network respond to different inputs in a highly class-speciﬁc manner.
In this way, it helps CE learn preciser feature representation.
To verify this, we employ an off-the-shelf tool to visualize the class activation map (CAM)
Selvaraju et al.
We use ResNet50, BD-ResNet50, and CE-ResNet50 trained on ImageNet for comparison.
As shown in Fig.7, the heat maps extracted from CAM for CE-ResNet50 have more coverage on the
[Figure]
object region and less coverage on the background region.
It shows that the AII branch helps CE learn preciser information from the images.
F TRAINING AND INFERENCE
Moving average in inference .
Unlike previous methods in manual architecture design that do not depend on batch estimated statistics, the proposed CE block requires computing the inverse square root of a batch covariance matrix Σ and a global variance scale s in each training step.
To make the output depend only on the input, deterministically in inference, we use the moving average to calculate the population estimate of ˆΣ − 12 and ˆ s − 12 by following the below updating rules:
[Equation]
24) where s and Σ are the variance scale and covariance calculated within each mini-batch during training, and m denotes the momentum of moving average.
It is worth noting that since ˆ Σ − 1 2 is ﬁxed during inference, the BD branch does not introduce extra costs in memory or computation except for a simple linear transformation ( ˆΣ − 12 ˜ x ).
Model and computational complexity .
The main computation of our CE includes calculating the covariance and inverse square root of it in the BD branch and computing two FC layers in the AII branch.
We see that there is a lot of space to reduce computational cost of CE.
For BD branch, given an internal feature x ∈ R N × C × H × W , the cost of calculating a covariance matrix is 2 NHWC 2 , which is comparable to the cost of convolution operation.
A pooling operation can be employed to downsample featuremap for too large H and W .
In this way, the complexity can be reduced to 2 NHWC 2 /k 2 + CHW where k is kernel size of the window for pooling.
Further, we can use group-wise whitening to improve efﬁciency, reducing the cost of computing Σ − 12 from TC 3 to
TCg 2 ( g is group size).
For AII branch, we focus on the additional parameters introduced by two FC layers.
In fact, the reduction ratio r can be appropriately chosen to balance model complexity and representational power.
Besides, the majority of these parameters come from the ﬁnal block of the network.
For example, a single AII in the ﬁnal block of ResNet-50 has 2 ∗ 2048 2 /r parameters.
In practice, the CE blocks in the ﬁnal stages of networks are removed to reduce additional parameters.
We provide the measurement of computational burden and Flops in Table 1.
ResNet Training Setting .
All networks are trained using 8 GPUs with a mini-batch of 32 per GPU.
We train all the architectures from scratch for 100 epochs using stochastic gradient descent (SGD) with momentum 0 .
9 and weight decay 1e-4.
The base learning rate is set to 0 .
and is multiplied by 0.1 after 30 , 60 and 90 epochs.
Besides, the covariance matrix in BD branch is calculated within each GPU.
Since the computation of covariance matrix involves heavy computation when the size of feature map is large, a 2 × 2 maximum pooling is adopted to down-sample the feature map after the ﬁrst batch normalization layer.
Like (Huang et al., 2019), we also use group-wise decorrelation with group size 16 across the network to improve the efﬁciency in the BD branch.
By default, the reduction ratio r in AII branch is set to 4 .
MobileNet V2 training Setting .
All networks are trained using 8 GPUs with a mini-batch of 32 per
GPU for 150 epochs with cosine learning rate.
The base learning rate is set to 0.05 and the weight decay is 4e-5.
ShufﬂeNet V2 training Setting .
All networks are trained using 8 GPUs with a mini-batch ofper GPU for 240 epochs with poly learning rate.
The base learning rate is set to 0.5 and weight decay is 4e-5.
We also adopt warmup and label smoothing tricks.
VGG networks on CIFAR10 training setting .
For CIFAR10, we train VGG networks with a batch size of 256 on a single GPU for 160 epochs.
The initial learning rate is 0.1 and is decreased bytimes every 60 epochs.
The inhibited channel ratios in Fig.
1 and Fig.3(c) is measured by the average ratio for the ﬁrst 6-th layers since the bottom layers can extract rich feature representation and the sparsity is not desired.
For inference drop experiments in Fig.1(c), we randomly drop channels in the third layer with different dropout ratio.
For each ratio, we run the experiment 5 times and average the top 1 accuracy.
Mask-RCNN training setting in COCO .
We ﬁne-tune the ImageNet pretrained model in COCO for 24 epoch with base learning rate 0.02 and multiply it by 0.1 after 16 and 22 epochs.
All the models are trained using 8 GPUs with a mini-batch of 2 images.
The basic backbone structure is adopted from the ResNet50/ResNet101 trained on ImageNet.