S ELECTIVE C ONVOLUTIONAL U NITS : I MPROVING CNN S VIA C HANNEL S ELECTIVITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Bottleneck structures with identity (e.g., residual) connection are now emerg ing popular paradigms for designing deep convolutional neural networks CNN) , for processing large scale features efﬁciently
In this paper , we focus on the information -preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity , i.e ., redistributing its computations to important channels.
In particular, we propose
Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efﬁciency of various modern CNNs with bottlenecks.
During training, SCU gradually learns the channel-selectivity on-the-ﬂy via the alternative usage of (a) pruning unimportant channels, and (b) rewiring the pruned parameters to important channels.
The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it.
Our experimental results demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures
I NTRODUCTION
Nowadays, convolutional neural networks (CNNs) have become one of the most effective approaches in various ﬁelds of artiﬁcial intelligence.
With a growing interest of CNNs, there has been a lot of works on designing more advanced CNN architectures (Szegedy et al., 2015; Simonyan &
Zisserman, 2014; Ioffe & Szegedy, 2015).
In particular, the simple idea of adding identity connection in ResNet (He et al., 2016a) has enabled breakthroughs in this direction, as it allows to train substantially deeper/wider networks than before by alleviating existed optimization difﬁculties in previous CNNs.
Recent CNNs can scale over a thousand of layers (He et al., 2016b) or channels (Huang et al., 2017b) without much overﬁtting, and most of these “giant” models consider identity connections in various ways (Xie et al., 2017;
Huang et al., 2017b; Chen et al., 2017).
However, as CNN models grow rapidly, deploying them in the real-world becomes increasingly difﬁcult due to computing resource constraints.
This has motivated the recent literature such as network pruning (Han et al., 2015;
He et al., 2017; Liu et al., 2017; Neklyudov et al., 2017), weight quantization (Rastegari et al., 2016; Courbariaux & Bengio, 2016; Chen et al., 2018), adaptive networks (Teerapittayanon et al., 2016;
Figurnov et al., 2017; Bolukbasi et al., 2017; Huang et al., 2018), and resource-efﬁcient architectures (Huang et al., 2017a; Sandler et al., 2018; Ma et al., 2018).
For designing a resource-efﬁcient CNN architecture, it is important to process succinct representations of large-scale channels.
To this end, the identity connections are useful since they allow to reduce the representation dimension to a large extent while “preserving” information from the previous layer.
Such bottleneck architectures are now widely used in modern CNNs such as ResNet (He et al., 2016a) and DenseNet (Huang et al., 2017b) for parameter efﬁciency, and many state-of-the-art mobile-targeted architectures such as SqueezeNet (Iandola et al., 2016), ShufﬂeNet (Zhang et al., 2017b; Ma et al., 2018), MoblileNet (Howard et al., 2017; Sandler et al., 2018), and CondenseNet (Huang et al., 2017a) commonly address the importance of designing efﬁcient bottlenecks.
Contribution.
In this paper, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit for efﬁcient utilization of parameters in particular as a bottleneck upon identity connection.
At a high-level, SCU performs a convolutional operation to transform a given input.
The main goal of SCU, however, is rather to re-distribute their computations only to selected channels
[Figure]
of importance, instead of processing the entire input naively.
To this end, SCU has two special operations: (a) de-allocate unnecessary input channels ( dealloc ), and (b) re-allocate the obstructed channels to other channels of importance ( realloc ) (see Figure 1a).
They are performed without damaging the network output (i.e., function-preserving operations), and therefore one can call them safely at any time during training.
Consequently, training SCU is a process that increases the efﬁciency of CNN by iteratively pruning or rewiring its parameters on-the-ﬂy along with learning them.
In some sense, it is similar to how hippocampus in human brain learn, where new neurons are generated daily, and rewired into the existing network while maintaining them via neuronal apoptosis or pruning (Sahay et al., 2011a;b).
We combine several new ideas to tackle technical challenges for such on-demand, efﬁcient trainable SCU.
First, we propose expected channel damage score (ECDS), a novel metric of channel importance that is used as the criterion to select channels for dealloc or realloc .
Compared to other popular magnitude-based metrics (Li et al., 2016; Liu et al., 2017; Neklyudov et al., 2017),
ECDS allows capturing not only low-magnitude channels but also channels of low-contribution under the input distribution.
Second, we impose channel-wise spatial shifting bias when a channel is reallocated, providing much diversity in the input distribution.
It also has an effect of enlarging the convolutional kernel of SCU.
Finally, we place a channel-wise scaling layer inside SCU with sparsity-inducing regularization, which also promotes dealloc (and consequently realloc as well), without further overhead in inference and training.
We evaluate the effectiveness of SCU by applying it to several modern CNN models including
ResNet (He et al., 2016a), DenseNet (Huang et al., 2017b), and ResNeXt (Xie et al., 2017), on various classiﬁcation datasets.
Our experimental results consistently show that SCU improves the efﬁciency of bottlenecks both in model size and classiﬁcation accuracy.
For example, SCU reduces the error rates of DenseNet-40 model (without any post-processing) by using even less parameters: 6 . 57% → 5 .
95% and 29 .
97% → 28 .
64% on CIFAR-10/100 datasets, respectively.
We also apply
SCU to a mobile-targeted CondenseNet (Huang et al., 2017a) model, and further improve its efﬁciency: it even outperforms NASNet-C (Zoph et al., 2018), an architecture searched with 500 GPUs for 4 days, while our model is constructed with minimal efforts automatically via SCU.
There have been signiﬁcant interests in the literature on discovering which parameters to be pruned during training of neural networks, e.g., see the literature of network sparsity learning (Wen et al., 2016; Lebedev & Lempitsky, 2016; Scardapane et al., 2017; Molchanov et al., 2017; Neklyudov et al., 2017; Louizos et al., 2017; 2018; Dai et al., 2018).
On the other hand, the progress is, arguably, slower for how to rewire the pruned parameters of a given model to maximize its utility.
Han et al.
(2016) proposed Dense-Sparse-Dense (DSD), a multi-step training ﬂow applicable for a wide range of DNNs showing that re-training with re-initializing the pruned parameters can improve the performance of the original network.
Dynamic network surgery (Guo et al., 2016), on the other hand, proposed a methodology of splicing the pruned connections so that mis-pruned ones can be recovered, yielding a better compression performance.
In this paper, we propose a new way of rewiring for parameter efﬁciency, i.e., rewiring for channel-selectivity, and a new architectural framework that enables both pruning and rewiring in a single pass of training without any postprocessing or re-training (as like human brain learning).
Under our framework, one can easily set a targeted trade-off between model compression and accuracy improvement depending on her purpose, simply by adjusting the calling policy of dealloc and realloc .
We believe that our work sheds a new direction on the important problem of training neural networks efﬁciently.
S ELECTIVE CONVOLUTIONAL UNITS
In this section, we describe Selective Convolutional Unit (SCU), a generic architectural unit for bottleneck CNN architectures.
The overall structure of SCU is described in Section 2.1 and 2.2.
In
Section 2.3, we introduce a metric deciding channel-selectivity in SCU.
We present in Section 2.
how to handle a network including SCUs in training and inference.
O VERVIEW
Bottleneck structures in modern CNNs.
We ﬁrst consider a residual function deﬁned in ResNet (He et al., 2016a) which has an identity mapping: for a given input random variable X ∈ R
I × H × W
( H and W are the height and width of each channel, respectively, and I is the number of channels or feature maps) and a non-linear function F , the output of a residual function is written by Y =
X + F ( X ) .
This function has been commonly used as a building block for designing recent deep CNN models, in a form that F is modeled by a shallow CNN.
However, depending on how F is designed, computing F ( X ) can be expensive when I is large.
For tackling the issue, bottleneck structure is a prominent approach, that is, to model F by F (cid:48) ◦ R by placing a bottleneck R that ﬁrstly maps X into a lower dimension of I (cid:48) <
I features.
This approach, in essence, requires the identity connection, for avoiding information loss from X to Y .
Namely, the identity connection enables a layer to save redundant computation (or parameters) for just “keeping” information from the input.
Bottleneck structures can be used other than ResNet as well, as long as the identity connection exists.
Recent architectures including DenseNet (Huang et al., 2017b), PyramidNet (Han et al., 2017) and DPN (Chen et al., 2017) develop this idea with using a different aggregation function W instead of addition in ResNet, e.g., W ( X , X (cid:48) )
=
[ X , X (cid:48) ]
(channel-wise concatenation) for DenseNet.
Designing R - F (cid:48) - W is now a common way of handling large features.
Channel-selectivity for efﬁcient bottlenecks.
Although placing a bottleneck R reduces much computation of the main function F (cid:48) , we point out the majority of modern CNNs currently use inefﬁcient design of R itself, so that even the computation of R often dominates the remaining.
In
ResNet and DenseNet models, for example, bottlenecks are designed using a pointwise convolution with a batch normalization layer (BN) (Ioffe & Szegedy, 2015) and ReLU (Nair & Hinton, 2010):
[Equation] where Conv 1 × 1 I → I (cid:48) denotes a pointwise convolution that maps I features into I (cid:48) features, i.e., its parameters can be represented by a I × I (cid:48) matrix. This means that the parameters of R grows linearly on I , and it can be much larger than F (cid:48) if I (cid:29) I (cid:48) . For example, in case of DenseNetBC-190 (Huang et al., 2017b), 70% of the total parameters are devoted for modeling R , which is inefﬁcient as the expressivity of a pointwise convolution is somewhat limited.
In this paper, we attempt to improve the efﬁciency of R in two ways: (a) reducing the parameters in Conv 1 × 1 I → I (cid:48) by channel pruning, and (b) improving its expressivity by using the pruned parameters again. This motivates our goal to learn both channel-selectivity and parameters jointly. Overall architecture of SCU. SCU is designed to learn the channel-selectivity via dynamic pruning and rewiring of channels during training.
In this paper, we focus on putting SCU as a bottleneck R , and show that the channel-selectivity of SCU improves its parameter efﬁciency.
Our intuition is that (a) the information-preserving nature of identity connection brings optimization beneﬁts if neurons in its structure are dynamically pruned during training, and (b) such pruning can be particularly effective on bottlenecks as their outputs are in a much lower dimension compared to the input. Nevertheless, we believe that our ideas on SCU are not limited to the bottleneck structures, as the concept of channel-selectivity can be generalized to other structures.
At a high level, SCU follows the bottleneck structure from (1), but for two additional layers: Channel Distributor (CD) and Noise Controller (NC) whose details are presented in Section 2.2.
We model a non-linear function SCU : R I × H × W
→ R I (cid:48) × H × W as follows (see Figure 1b):
[Equation]
SCU has two special operations which control its input channels to process: (a) channel deallocation ( dealloc ), which obstructs unnecessary channels from being used in future computations, and (b) channel re-allocation ( realloc ), which allocates more parameters to important, non-obstructed channels by copying them into the obstructed areas.
We design those operations to be function preserving , i.e. they do not change the original function of the unit, so that can be called at anytime during training without damage.
Repeating dealloc and realloc alternatively during training translates the original input to what has only a few important channels, potentially duplicated multiple times.
Namely, the parameters originally allocated to handle the entire input now operate on its important subset.
On the way of designing the operations of function preserving, we propose Expected Channel Damage Score (ECDS) that leads to an efﬁcient, safe way to capture unimportant channels by measuring how much the output of SCU changes on average (w.r.t.
data distribution) after removing each channel.
The details of ECDS are in Section 2.3.
2. D ESIGN OF SCU: CD AND NC
Channel Distributor (CD) is the principal mechanism of SCU and is placed at the beginning of the unit.
The role of CD is to “rebuild” the input, so that unnecessary channels can be discarded, and important channels are copied to be emphasized.
In essence, we implement this function by re-indexing and blocking the input channel-wise: CD( X ) i := g i · X π i with an index pointer π i ∈ { 1 , 2 , · · · , I } , a gate variable g i ∈ { 0 , 1 } for i = 1 , 2 , · · · , I . Here, we notice that CD( X ) may contain a channel copied multiple times, i.e., multiple π i ’s can have the same value. Since SCU has different parameters for each channel, setting multiple π i ’s has an effect of allocating more parameters to better process the channel pointed by π i .
We found that, however, it is hard to take advantage of the newly allocated parameters by simply copying a channel due to symmetry, i.e., the parameters for each channel usually degenerates. Due to this, we consider spatial shifting biases b i = ( b hi , b wi ) ∈ Rfor each channel, as illustrated in Figure 2. This trick can provide the copied channels much diversity in input distributions (and hence relaxing degeneracy), in a way that it is effective for the convolutional layer in SCU: it enlarges the convolutional kernel from 1 × 1 for the re-allocated channels only.
To summerize, CD( X ) i takes (a) the channel which π i is pointing, in the spatially shifted form with bias b i , or (b) 0 if the gate g i is closed.
Formally, CD can be represented by ( π , g , b ) =
( π i , g i , b i ) Ii =1 .
CD( X ) has the same size to X , and deﬁned as follows:
[Equation]
Here, shift( X, b h , b w ) denotes the “shifting” operation along spatial dimensions of X .
[-In other words, CD( X ) i takes (a) the channel which π i is pointing, in the spatially shifted form with bias b i , or (b) 
-]
For each pixel location ( i, j ) in X , we deﬁne shift( X, b h , b w ) i,j as:
[Equation]
using a bilinear interpolation kernel.
This formulation allows b h and b w to be continuous real values, thereby to be learned via gradient-based methods with other parameters jointly.
Noise Controller (NC) is a component for more effective training of SCU.
As SCU continuously performs channel pruning via dealloc during training, the efﬁciency of SCU depends on which regularization is used. The key role of NC is to induce the training of SCU to get more channel-wise sparsity, so that more channels can be de-allocated safely. Formally, NC is a channel-wise re-scaling layer: NC( X ) :
= X (cid:12) θ , 1 where θ =
( θ i ) Ii =1 are parameters to be learned. For the channel-wise sparsity, we impose sparsity-inducing regularization speciﬁcally on θ .
Although any sparsity-inducing regularization can be used for θ (Liu et al., 2017; Wen et al., 2016), in this paper we adopt the Bayesian pruning approach proposed by Neklyudov et al.
(2017) 2 for two reasons: (a) it is easy to incorporate into training process, and (b) we found that noise incurred from
Bayesian parameters helps to recover damage from channel pruning. In general, a Bayesian scheme regards each parameter θ as a random variable with prior p ( θ ) .
Updating the posterior p ( θ |D ) from data D often leads the model to have much sparsity, if p ( θ ) is set to induce sparsity, e.g., by log-uniform prior (Kingma et al., 2015). Meanwhile, p ( θ |D ) is usually approximated with a simpler model q φ ( θ ) , where φ are parameters to be learned.
In case of NC, we regard each scaling parameter as a random variable, so that they become channel-wise multiplicative noises on input.
We follow
Neklyudov et al.
(2017) for the design choices on q φ ( θ ) and p ( θ ) , by fully-factorized log-normal distribution q φ ( θ ) =
(cid:81)
i LogN( θ i | µ i , σ 2 i ) and log-uniform prior p ( θ ) =
(cid:81)
i LogU ( θ i )
∝ 1 θ i .
2. M ETRIC FOR C HANNEL -S ELECTIVITY : ECDS
Consider an input random variable X
=
( X i ∈ R H × W ) Ii =1 of I features.
From now on, we denote a SCU by S =
( W NC , W CD , W BN , W Conv ) , where each denotes the parameters in NC, CD, BN, and
Conv, respectively.
Here, W BN = ( γ i , β i ) Ii =1 , and W Conv =
( W Conv i ∈
R I (cid:48) × 1 × 1 ) Ii =1 .
Expected channel damage score (ECDS) aims for measuring E
[SCU( X ; S ) − SCU( X ; S − i )] ∈
R I (cid:48) × H × W , where S − i denotes a SCU identical to S but g i = 0 .
In other words, it is the expected amount of changes in outputs when S i is “damaged” or “pruned”.
The primary goal of this criteria is to make dealloc to be function preserving.
We deﬁne ECDS( S )
i by the (Euclidean) norm of the averaged values of E
[SCU( X ; S ) − SCU( X ; S − i )] over the spatial dimensions:
[Equation]
Notice that the above deﬁnition requires a marginalization over random variable X .
One can estimate it via Monte Carlo sampling using training data, but this is computationally too expensive compared to other popular magnitude-based metrics (Li et al., 2016; Liu et al., 2017; Neklyudov et al., 2017).
Instead, we utilize the BN layer inside SCU, to infer the current input distribution of each channel at any time of training.
This trick enables to approximate ECDS( S )
i by a closed formula of S i , avoiding expensive computations of SCU( X ; · ) , as in what follows.
Consider a hidden neuron x following BN and ReLU, i.e., y = ReLU(BN( x )) , and suppose one wants to estimate E
[ y ] without sampling.
To this end, we exploit the fact that BN already “accumulates” its input statistics continuously during training.
Under assuming that BN( x ) ∼ N ( β, γ 2 ) where γ and β are the scaling and shifting parameter in BN, respectively, it is elementary to check:
[Equation]
where φ N and Φ N denote the p.d.f.
and the c.d.f.
of the standard normal distribution, respectively.
The assumption is quite reasonable during training BN as each mini-batch is exactly normalized before applying the scaling and shifting inside BN.
The idea is directly extended to obtain a closed form formula of ECDS( S )
i under some assumptions, as stated in the following proposition.
Proposition 1.
Assume BN(CD
( X ; W CD ); W BN )
i,h,w ∼ N ( β i , γ 2 i ) for all i, h, w .[-
-]
Then, it holds
ECDS( S ) i
= g i · (cid:18) | γ i | φ N (cid:18) β
i | γ i | (cid:19)
+ β i Φ N (cid:18) β i | γ i | (cid:19)(cid:19) (cid:124) (cid:123)(cid:122) (cid:125) (a) · E [ θ i ] (cid:124)(cid:123)(cid:122)(cid:125)
(b) · (cid:13) (cid:13) W Conv i (cid:13) (cid:13) (cid:124) (cid:123)(cid:122) (cid:125) (c) , for all i.
The proof of the above proposition is given in Appendix D. In essence, there are three main terms in the formula: (a) a term that measures how much the input channel is active, (b) how much the
NC ampliﬁes the input, and (c) the total magnitude of weights in the convolutional layer.
Therefore, it allows a way to capture not only low-magnitude channels but also channels of low-contribution under the input distribution (see Section 3.2 for comparisons with other metrics).
2. T RAINING AND INFERENCE PROCEDURES
Consider a CNN model p
( Y | X , Θ ) employing SCU, where Θ denotes the collection of model parameters.
For easier explanation, we rewrite Θ by ( V , W ) :
V consists of ( π , g ) in CDs, and W is the remaining ones.
Given dataset D = { ( x n , y n ) } Nn =1 , ( V , W ) is trained via alternating two phases: (a) training W via stochastic gradient descent (SGD), and (b) updating V via dealloc or realloc .
The overall training process is mainly driven by (a), and the usage of (b) is optional.
In (a), we use stochastic variational inference (Kingma & Welling, 2013) in order to incorporate stochasticity incurred from NC, so that SCU can learn its Bayesian parameters in NC jointly with the others via SGD.
On the other hand, in (b), dealloc and realloc are called on demand during training depending on the purpose.
For example, one may decide to call dealloc only throughout the training to obtain a highly compressed model, or one could use realloc as well to utilize more model parameters.
Once (b) is called, (a) is temporally paused and V are updated.
Training via stochastic variational inference.
We can safely ignore the effect of V during training of W , since they remain ﬁxed. Recall that, each noise θ from a NC is assumed to follow q φ ( θ ) = LogN( θ | µ, σ 2 ) . Then, θ can be “re-parametrized” with a noise ε from the standard normal distribution as follows: θ = f ( φ = ( µ, σ ) , ε ) =
exp ( µ + σ · ε ) , where ε ∼ N (0 , 1 2 ) .
Stochastic variational inference (Kingma & Welling, 2013) allows a minibatch-based stochastic gradient method for θ , in such case that θ can be re-parametrized with an non-parametric noise.
The ﬁnal loss we minimize for a minibatch { ( x i k , y i k ) } Mk =1 becomes (see Appendix F for more details): [Equation] where ε i k = ( ε i k ,u ) | φ | u =1 is a sampled vector from the fully-factorized standard normal distribution.
Channel de-allocation and re-
allocation.
Consider a SCU S
=
( W NC , W CD , W BN , W Conv ) .
The main role of dealloc and realloc is to update W CD in S that are not trained directly via SGD.
They are performed as follows: select slices to operate by thresholding ECDS( S ) , and update S from the selected channels.
More formally, when dealloc is called, S i ’s where ECDS( S )
i < T l for a ﬁxed threshold T l are selected, and g i ’s in W CD are set by 0.
If one chooses small T l , this operation does not hurt the original function.
On the other hand, realloc selects channels by collecting S i where ECDS( S ) i > T h , for another threshold T h .
Each of the selected channels can be re-allocated only if there is a closed channel in S .
If there does not exist a enough space, channels with higher ECDS have priority to be selected.
A single re-allocation of a channel S i to a closed channel S j consists of several steps: (i) open S j by g j ← 1 , (ii) copy W NC j , W BN j ← W NC i , W BN i
(iii) set W Conv j ← (iv) re-initialize the shifting bias b j , and (v) set π j ← π
i This procedure is function-preserving, due to (iii).
After training a SCU S , one can safely remove S i ’s that are closed, to yield a compact unit. Then,
CDs are now operated by “selecting” channels rather than by obstructing, thereby the subsequent layers play with smaller dimensions.
Hence, at the end, SCU is trained to select only a subset of the input for performing the bottleneck operation.
For NC, on the other hand, one can still use it for inference, but efﬁcient inference can be performed by replacing each noise θ i by constant E
[ θ i ] , following the well-known approximation used in many dropout-like techniques (Hinton et al., 2012).
E XPERIMENTAL RESULTS
In our experiments, we apply SCU to several well-known CNN architectures that uses bottlenecks, and perform experiments on CIFAR-10/100 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) classiﬁcation datasets.
The more details on our experimental setups, e.g., datasets, training details, and conﬁgurations of SCU, are given in Appendix G.
3. I MPROVED CNN MODELS WITH SCU
Improving existing CNNs with SCU.
We consider models using ResNet (He et al., 2016a),
DenseNet (Huang et al., 2017b) and ResNeXt (Xie et al., 2017) architectures.
In general, every model we used in this paper forms a stack of multiple bottlenecks, where the deﬁnition of each bottleneck differs depending on its architecture except that it can be commonly expressed by RF (cid:48) - W (the details are given in Table 5 in the appendix).
We compare the existing models with the corresponding new ones in which the bottlenecks are replaced by SCU.
For each SCU-based model, we consider three cases: (a) neither dealloc nor realloc is used during training, (b) only dealloc is used, and (c) both dealloc and realloc are used.
We measure the total number of parameters in bottlenecks, and error rates.
[Table] [Table] [Table] Table 1 compares the existing CNN models with the corresponding ones using SCU, on CIFAR10/100.
The results consistently demonstrate that SCU improves the original models, showing their effectiveness in different ways.
When only dealloc is used, the model tends to be trained with minimizing their parameter to use.
Using realloc , SCU now can utilize the de-allocated parameters to improve their accuracy aggressively.
Note that SCU can improve the accuracy of the original model even neither dealloc nor realloc is used.
This gain is from the regularization effect of stochastic NC, acting a dropout-like layer.
We also emphasize that one can set a targeted trade-off between compression of SCU and accuracy improvement depending on her purpose, simply by adjusting the calling policy of dealloc and realloc .
For example, in case of
DenseNet-100 model on CIFAR-10, one can easily trade-off between reductions in (compression, error) =
( − 51 .
4% , − 1 .
and ( − 18 .
2% , − 8 .
In overall, SCU-based models achieve both model compression and accuracy improvement under all tested architectures.
Table 2 shows the results on ImageNet, which are consistent to those on CIFAR-10/100.
Notice that reducing parameters and error simultaneously is much more non-trivial in the case of ImageNet, e.g., reducing error 23 .
6% → 23 . 0% requires to add 51 more layers to ResNet-101 (i.e., ResNet-152), as reported in the ofﬁcial repository of ResNet (He et al., 2016c).
Designing efﬁcient CNNs with SCU.
We also demonstrate that SCU can be used to design a totally new efﬁcient architecture.
Recall that, in this paper, SCU focus on the bottlenecks inside the overall
E rr o r r a t e s ( % ) 5.
D +R +R+S (a)
M1M 0% 3% 6% 9% 12%
ECDS0.
(b)
Error rates (%) # de-allocated with ECDS without ECDS 6.
M
M M M
M (c) structure. The other parts, F (cid:48) or W , are other orthogonal design choices. To improve the efﬁciency of the parts, we adopt some components from CondenseNet (Huang et al., 2017a), which is one of the state-of-the-art architectures in terms of computational efﬁciency, designed for mobile devices.
Although we do not adopt their main component, i.e., learned group convolution (LGC) as it also targets for the bottleneck as like SCU, we can still utilize other components of CondenseNet: increasing growth rate (IGR) (doubles the growth rate of DenseNet for every N blocks starting from 8) and the use of group convolution for F (cid:48) . Namely, we construct a new model, coined CondenseNet-SCU by adopting IGR and GC upon a DenseNet-182 model with SCU. We replace each 3 × 3 convolution for
F (cid:48) by a group convolution of 4 groups. We train this model using dealloc only to maximize the computational efﬁciency. In Table 3, we compare our model with state-of-the-art level CNNs, including ResNet-1001 (He et al., 2016b), WRN-28-10 (Zagoruyko & Komodakis, 2016), NASNet-C
(Zoph et al., 2018), and the original CondenseNet-182.
As one can observe, our model shows better efﬁciency compared to the corresponding CondenseNet, suggesting the effectiveness of SCU over
LGC.
Somewhat interestingly, ours even outperforms NASNet-C that is an architecture searched over thousands of candidates, in both model compression and accuracy improvement.
We ﬁnally remark that CondenseNet-SCU-182 model presented in Table 3 originally has 6.29M parameters in total before training, devoting 5.89M for bottlenecks, i.e., it is about 93.7% of the total number of parameters.
This is indeed an example in that reducing overhead from bottlenecks is important for better efﬁciency, which is addressed by SCU.
3. A BLATION STUDY
We also perform numerous ablation studies on the proposed SCU, investigating the effect of the key components: CD, NC, and ECDS.
For evaluation, we use the DenseNet-SCU-40 model (DenseNet40 using SCU) trained for CIFAR-10.
We also follow the training details described in Appendix G.
Spatial shifting and re-allocation.
We propose spatial shifting as a trick in realloc procedure to provide diversity in input distributions.
To evaluate its effect, we compare three DenseNet-SCUmodels with different conﬁgurations of SCU: (D) only dealloc during training, (+R) realloc together but without spatial shifting, and (+R+S) further with the shifting.
Figure 3a shows that +R does not improve the model performance much compared to D, despite +R+S outperforms both of them.
This suggests that copying a channel naively is not enough to fully utilize the rewired parameters, and spatial shifting is an effective way to overcome the issue. Model Metric NC
M
ECDS (cid:86)
M ECDS (cid:88) M SNR <(cid:86)
M
SNR < 2 .(cid:86)
M (cid:96) 2 < 0 .(cid:86)
Sparsity-inducing effect of NC.
We place NC in SCU to encourage more sparse channels.
To verify such an effect, we consider DenseNetSCU-40 model (say M1) and its variant removing NC from SCU (say M2).
We ﬁrst train M1 and M2 calling neither dealloc nor realloc , and compare them how the ECDS of each channel is distributed.
Figure 3b shows that M1 tends to have ECDS closer to zero, i.e., more channels will be de-allocated than M2.
Next, we train these models using dealloc , to conﬁrm that NC indeed leads to more deallocation.
The left panel of Figure 3c shows that the number of de-allocated channels of M1 is relatively larger than that of M2, which is the desired effect of NC.
Note that M1 also outperforms
M2 on error rates, which is an additional advantage of NC from its stochastic regularization effect.
Effectiveness of ECDS.
Nevertheless, remark that M2 in Figure 3c already de-allocates many channels, which suggests that SBP (used in NC) is not crucial for efﬁcient de-allocation.
Rather, the efﬁciency mainly comes from ECDS.
To prove this claim, we evaluate three variants of M1 which use different de-allocation policies than ECDS <
T l : (a) SNR < 1 (thresholding the signal-to-noise ratio of NC in each channel by 1, proposed by the original SBP; M3), (b) SNR < 2 . 3 (M4) and (c) (cid:96) 2 < 0 .
25 (thresholding (cid:107) W Conv i (cid:107) 2 ; M5).
We train them using only dealloc , and compare the performances with the proposed model (M1).
The right panel of Figure 3c shows the results of the three variants.
First, we found that the M3 could not de-allocate any channel in our setting (this is because we prune a network on-the-ﬂy during training, while the original SBP only did it after training).
When we de-allocate competitive numbers of channels against M1 by tuning thresholds of others (M4 and M5), the error rates are much worse than that of M1.
These observations conﬁrm that ECDS is a more effective de-allocation policy than other magnitude-based metrics.
C ONCLUSION
We demonstrate that CNNs of large-scale features can be trained effectively via channel-selectivity, primarily focusing on bottleneck architectures.
The proposed ideas on channel-selectivity, however, would be applicable other than the bottlenecks, which we believe is an interesting future research direction. We also expect that channel-selectivity has a potential to be used for other tasks as well, e.g., interpretability (Selvaraju et al., 2017), robustness (Goodfellow et al., 2014), and memorization (Zhang et al., 2017a).
A S TOCHASTIC VARIATIONAL INFERENCE FOR B AYESIAN MODELS
Consider a probabilistic model p ( Y | X , θ ) between two random variables X and Y , and suppose one wants to infer θ from a dataset D
= { ( x n , y n ) } Nn =1 consisting N i.i.d.
samples from the distribution of ( X , Y ) .
In Bayesian inference , θ is regarded as a random variable, under assuming some prior knowledge in terms of a prior distribution p ( θ ) .
The dataset D is then used to update the posterior belief on θ , namely p ( θ |D ) =
p ( D| θ ) p ( θ ) /p
( D ) from the Bayes rule.
In many cases, however, computing p ( θ |D ) through Bayes rule is intractable since it requires to compute intractable integrals.
To address the issue, variational inference approximates p ( θ |D ) by another parametric distribution q φ ( θ ) , and tries to minimize the KL-divergence D KL ( q φ ( θ ) (cid:107) p ( θ |D )) between q φ ( θ ) and p ( θ |D ) .
Instead of directly minimizing it, one typically maximizes the variational lower bound L ( φ ) , due to the following:
[Equation]
where
[Equation]
In case of complex models, however, expectations in (9) are still intractable.
Kingma & Welling (2013) proposed an unbiased minibatch-based Monte Carlo estimator for them, which can be used when q φ ( θ ) is representable by θ = f
( φ , ε ) with a non-parametric noise ε ∼ p ( ε ) .
For a minibatch { ( x i k , y i k ) } Mk =1 of size M , one can obtain
L D ( φ ) :=
[Equation]
Now we can solve optimize L ( φ ) by stochastic gradient ascent methods, if f is differentiable.
For a model having non-Bayesian parameters, say W , we can still apply the above approach by maximizing
[Equation]
where φ and W can be jointly optimized under L SGVB ( φ , W ) (cid:39) L ( φ , W ) .
B S TRUCTURED B AYESIAN PRUNING
Structured Bayesian pruning (SBP)
(Neklyudov et al., 2017) is a good example to show how stochastic variational inference can be incorporated into deep neural networks.
The SBP framework assumes X to be an object of I features, that is, X =
( X i ) Ii =1 .
For example, X ∈ R I × H × W can be a convolutional input consisting I channels, of the form X =
( X i ∈ R W × H ) Ii =1 where W and H denote the width and the height of each channel, respectively.
It considers a dropout-like layer with a noise vector θ =
( θ i ) Ii =1 ∼ p noise ( θ ) , which outputs X (cid:12) θ of the same size as X .
Here, θ is treated as a random vector, and the posterior p ( θ |D ) is approximated by a fully-factorized truncated log-normal distribution q φ ( θ ) :
[Equation]
where 1 [a , b] denotes the indicator function for the inveral [a , b] .
Meanwhile, the prior p ( θ ) is often chosen by a fully-factorized log-uniform distribution, e.g., Sparse
Variational Dropout (Molchanov et al., 2017), and SBP use the truncated version:
[Equation]
The reason why they use truncations for q φ ( θ ) and p ( θ ) is to prevent D KL ( q φ ( θ ) (cid:107) p ( θ )) to be improper.
Previous works (Kingma et al., 2015; Molchanov et al., 2017) ignore this issue by implicitly regarding them as truncated distributions on a broad interval, but SBP treats this issue explicitly.
Note that, each θ i ∼ q φ ( θ i ) =
LogN( θ i | µ i , σ 2 i ) in the noise vector θ can be re-parametrized with a non-parametric uniform noise ε i ∼ U ( ε | 0 , 1) by:
[Equation]
where α i = a − µ i σ i , β i =
b − µ i σ i , and Φ denotes the cumulative distribution function of the standard normal distribution.
Now one can optimize φ = ( µ , σ ) jointly with the weights W of a given neural network via stochastic variational inference described in Section A. Unlike Molchanov et al.
SBP regards W as a non-Bayesian parameter, and the ﬁnal loss L SBP to optimize becomes
[Equation]
θ )) .
Here, the KL-divergence term is scaled by α to compensate the trade-off between sparsity and accuracy.
In practice, SBP starts from a pre-trained model, and re-trains it using the above loss.
Due to the sparsity-inducing behavior of log-uniform prior, θ is forced to become more noisy troughout the re-training.
Neurons with θ of signal-to-noise ratio (SNR) below 1 are selected, and removed after the re-training:
[Equation]
C B AYESIAN PRUNING AND IDENTITY CONNECTIONS
SCU requires “training-time removal” of input channels for the channel de-allocation and reallocation to work.
But usually, this process should be done carefully since it can make the optimization much difﬁcult and put the network into a bad local minima.
In particular, it occurs if we select channels to remove too aggressively.
It is known that this issue becomes more pronounced in
Bayesian neural networks (Sønderby et al., 2016; Molchanov et al., 2017; Neklyudov et al., 2017;
Louizos et al., 2017), such as SBP we use in this paper.
Recall the variational lower bound objective in (12), for Bayesian parameters φ and non-Bayesian W .
If the gradient of the ﬁrst term (cid:80) N n =1 E q φ [log p ( y n | x n , θ , W )] on the right-hand side does not contribute much on ∇ φ L ( φ , W ) , then φ will be optimized mostly by −∇ φ D KL ( q φ ( θ ) (cid:107) p ( θ )) , that is, to follow the prior p ( θ ) .
Unfortunately, in practice, we usually observe this phenomena at the early stage of training, when W are randomly initialized.
In that case then, q φ ( θ ) will become p ( θ ) too fast because of the “uncertain”
W , thereby many channels will be pruned forever, in SBP for example.
This problem is usually dealt with in one of two ways: (a) using a pre-trained network as a starting point of W (Molchanov et al., 2017; Neklyudov et al., 2017), and (b) a “warm-up” strategy, where the KL-divergence term is rescaled by β that increases linearly from 0 to 1 during training (Sønderby et al., 2016;
Louizos et al., 2017).
In this paper, however, neither methods are used, but instead we have found that the problem can be much eased with identity connections, as it can eliminate a possible cause of the optimization difﬁculty from removing channels: optimization difﬁculty from losing information as an input passes through a deep network.
The presence of identity connection implies that the information of an input will be fully preserved even in the case when all the parameters in a layer are pruned.
This may not be true in models without identity, for example, in VGGNet (Simonyan & Zisserman, 2014), one can see that the information of an input will be completely lost if any of the layers removes its entire channels.
This suggests us that identity connections can be advantageous not only for scaling up the network architectures, but also for reducing the size of them.
D P ROOF OF P ROPOSITION
One can assume that S i is open, i.e. g i = 1 , otherwise ECDS( S ) i = 0 by deﬁnition since S = S − i .
Say Y :=
BN(CD( X ; W CD ); W BN ) , and let Conv 1 × 1 , ( i ) I
→ I (cid:48) :
R 1 × H × W
→ R I (cid:48) × H × W be the i -th slice of Conv 1 × 1 I → I (cid:48) , which can be deﬁned by the convolution with W Conv i so that Conv
[Equation]
I
[Equation]
I
[Equation]
; W Conv
[Equation]
=
[Equation] [Equation]
W Conv
[Equation]
Then, we have:
[Equation]
Now, check that ECDS( S i ) becomes:
[Equation]
By the assumption that Y i,h,w ∼ N ( β i , γ 2 i ) for all h, w , we get:
[Equation]
i,h,w
[Equation]
max
[Equation]
i
[Equation]
h
[Equation]
w
[Equation]
· θ i , and
[Equation]
where φ N and Φ N denote the probability distribution function and the cumulative distribution function of the standard normal distribution.
Therefore, the desired formula for ECDS( S i ) can be obtained by using the linearity of expectation:
[Equation]
(cid:48)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
This completes the proof of Proposition 1.
E E MPIRICAL SUPPORTS ON THE ASSUMPTION OF P ROPOSITION
To validate whether the assumption BN(CD( X ; W CD ); W BN ) i,h,w ∼ N ( β i , γ 2 i ) holds in modern
CNNs, we ﬁrst observe that, once we ignore the effects from spatial shifting, 5 a necessary condition of the assumption is that ( X i, : , : ) are identically distributed normal for a given channel X i .
This is because BN and CD do not change the “shape” of pixel-wise distributions of X i .
From this observation, we conduct a set of experiments focusing on a randomly chosen hidden layer in a
DenseNet-40 model.
We analyze the empirical distribution of the hidden activation incoming to the layer calculated from CIFAR-10 test dataset.
Since the data consists of 10,000 samples, we get an hidden activation X test ∈ R 10000 ×
C × 32 × 32 , 6 where C denotes the number of channels of the input.
−−   (a) Non-boundary pixels −−   (b) Boundary pixels     − 10 − 8 − 6 − 4 − 2 (c) Initialized randomly 0.
− 1. − 1.
− 0.
(d) Trained to converge
Now, suppose that we focus on a speciﬁc channel in X test , say X test c ∈ R 10000 ×
32 × 32 .
Notice that if our assumption is perfectly satisﬁed, then all the slices X test c,h,w ∈ R 10000 will represent a ﬁxed normal distribution for any h, w .
Interestingly, by analyzing empirical distributions of X test c,h,w for varying h and w , we found that: (a) for a ﬁxed c , most of the empirical distributions from X test c,h,w have unimodal shapes, except for the pixels in boundaries of the channel (Figure 4a, 4b), and (b) for a large portion of c ∈ { 1 , · · · , C } , the empirical means and variances of X test c,h,w ’s are concentrated in a cluster (Figure 4c, 4d).
These observations suggest us that the assumption of Proposition 1 can be reasonable except for the boundaries.
We also emphasize that these trends we found are also appeared even when the model is not trained at all (Figure 4c), that is, all the weights are randomly initialized, which implies that these properties are not “learned”, but come from a structural property of CNN, e.g. equivariance on translation, or the central limit theorem.
This observation provides us another support why the
ECDS formula stated in Proposition 1 is valid at any time during training.
F T RAINING SCU VIA STOCHASTIC VARIATIONAL INFERENCE
From Θ = ( V , W ) : V consists of ( π , g ) in CDs, we further rewrite W by ( W NC , W C ) :
W NC the parameters in NCs, and W C is the remaining ones.
One can safely ignore the effect of V during training of ( W NC , W C ) , since they remain ﬁxed.
Recall that each noise θ from a NC is assumed to follow LogN( θ | µ, σ 2 ) .
They can be re-written with a noise ε from the standard normal distribution, i.e., θ = f
(( µ, σ ) , ε ) =
exp ( µ + σ · ε ) , where ε ∼ N (0 , 1 2 ) .
In such case that each noise θ from NC can be “re-parametrized” with an non-parametric noise and the corresponding parameters φ = ( µ, σ ) , we can then use stochastic variational inference (Kingma & Welling, 2013) for the optimization of ( W NC , W C ) with a minibatch-based stochastic gradient method (see Appendix A for more details).
Then, the ﬁnal loss we minimize for a minibatch { ( x i k , y i k ) } Mk =1 becomes:
[Equation]
where ε i k
= ( ε i k ,u ) | φ | u =1 is a sampled vector from the fully-factorized standard normal distribution, and D KL ( ·(cid:107)· ) denotes the KL-divergence.
Although not shown in (24), an extra regularization term
R ( W C ) can be added to the loss for the non-Bayesian parameters W C , e.g., weight decays.
In fact, in our case, i.e. q φ ( θ ) =
LogN( θ | µ, σ 2 ) and p ( θ ) =
LogU( θ ) , D KL ( q φ ( θ ) (cid:107) p ( θ )) becomes improper:
[Equation]
where
[Equation]
As we explain in Appendix B, SBP bypasses this issue by using truncated distributions on a compact interval [ a, b ] for q φ ( θ ) and p ( θ ) .
We found that, however, this treatment also imposes extra computational overheads on several parts of training process, such as on sampling noises and computing
D KL ( q φ ( θ ) (cid:107) p ( θ )) .
These overheads are non-negligible on large models like ResNet or DenseNet, which we are mainly focusing on.
Therefore, unlike SBP, here we do not take truncations on q φ ( θ ) and p ( θ ) due to practical consideration, assuming an approximated form between the truncated distributions of q φ ( θ ) and p ( θ ) on a large interval.
Then we can replace each D KL ( q φ ( θ ) (cid:107) p ( θ )) in (24) by − log σ for optimization.
In other words, each noise θ in NC is regularized to a larger variance, i.e., the more “noisy”.
We observed that this approximation does not harm much on the performance of SCU.
Nevertheless, one should be careful that q φ ( θ ) and p ( θ ) should not be assumed as the un-truncated forms itself, but instead as approximated forms of truncated distributions on a large interval, not to make the problem ill-posed.
As used in SBP, if they are truncated, the KL-divergence becomes:
[Equation]
[Equation]
where α = a − µσ , β = b − µσ , φ ( · ) and Φ( · ) are the p.d.f.
and c.d.f.
of the standard normal distribution, respectively.
G E XPERIMENT SETUPS
Datasets.
We perform our experiments extensively on CIFAR-10 and CIFAR-100 (Krizhevsky, 2009) classiﬁcation datasets.
CIFAR-10/100 contains 60,000 RGB images of size 32 × 32 pixels, 50,000 for training and 10,000 for test.
Each image in the two datasets is corresponded to one ofand 100 classes, respectively, and the number of data is set evenly for each class.
We use a common scheme for data-augmentation (Srivastava et al., 2015;
Lin et al., 2013; He et al., 2016a; Huang et al., 2017b).
ImageNet classiﬁcation dataset, on the other hand, consists of 1.2 million training images and 50,000 validation images, which are labeled with 1,000 classes.
We follow (Huang et al., 2017a;
He et al., 2016a) for preprocessing of data in training and inference time.
Training details.
All models in our experiments is trained by stochastic gradient descent (SGD) method, with Nesterov momentum of weight 0.9 without dampening.
We use a cosine shape learning rate schedule (Loshchilov & Hutter, 2016), i.e., decreasing the learning rate gradually from 0 .to 0 throughout the training.
We set the weight decay 10 − 4 by for non-Bayesian parameters of each model.
We train each CIFAR model for 300 epochs with mini-batch size 64 following Huang et al.
(2017b), except for the “DenseNet-BC-190+ mixup ” models as they are trained for 200 epochs following the original setting (Zhang et al., 2018).
For ImageNet models, on the other hand, we train for 120 epochs with mini-batch size 256.
Conﬁgurations of SCU.
When a SCU S =
( W NC , W CD , W BN , W Conv ) is employed in a model, we initialize W NC =
( µ , σ ) by ( 0 , e − 3 ) , and W CD =
( π i , g i , b i ) Ii =1 by ( i, 1 , 0 ) Ii =1 .
Initializations of W BN and W Conv may differ depending on models, and we follow the initialization scheme of the given model.
In our experiments, we follow a pre-deﬁned calling policy when dealloc and realloc will be called throughout training.
If dealloc is used, it is called at the end of each epoch of training.
On the other hand, if realloc is used, it start to be called after 10% of the training is done, called for every 3 epochs, and stopped in 50% of training is done.
The thresholds for dealloc and realloc , i.e. T l and T h , is set by 0.0025 and 0.05, respectively, except for
CondenseNet-SCU-182 (Table 3), in which T l is adjusted by 0.001 for an effective comparison with the baseline.
For all the CIFAR-10/100 models, we re-initialize b i by a random sample from [ − 1 . 5 , 1 . 5] × [ − 1 . 5 , 1 . 5] pixels uniformly whenever a channel slice S i is re-open via realloc process.
We set the weight decay on each b i to 10 − 5 separately from the other parameters.
For the ImageNet results (Table 2), however, we did not jointly train b for faster training.
Instead, each b i is set ﬁxed unless it is re-initialized via realloc .
In this case, we sampled a point from [ − 2 . 5 , 2 . 5] × [ − 2 .
pixels uniformly for the re-initialization.
We found that this simple reallocation scheme can also improve the efﬁciency of SCU.
Models.
In general, every model we used here forms a stack of multiple bottlenecks, where the definition of each bottleneck differs depending on its architecture (see Table 5).
Each stack is separated into three (CIFAR-10/100) or four (ImageNet) stages by average pooling layers of kernel 2 × 2 to perform down-sampling.
Each of the stages consists N bottleneck blocks, and we report which N is used for all the tested models in Table 6.
The whole stack of each model follows a global average pooling layer (Lin et al., 2013) and a fully connected layer, and followed by single convolutional layer (See Table 7).
There exist some minor differences between the resulting models and the original papers (He et al., 2016a;
Huang et al., 2017b; Xie et al., 2017).
In ResNet and ResNeXt models, we place an explicit 2 × 2 average pooling layer for down-sampling, instead of using convolutional layer of stride 2.
Also, we use a simple zero-padding scheme for doubling the number of channels between stages.
In case of DenseNet, on the other hand, our DenseNet models are different from
DenseNet-BC proposed by Huang et al. (2017b), in a sense that we do not place a 1 × 1 convolutional layer between stages (which is referred as the “compression” layer in the original DenseNet).
Nevertheless, we observed that the models we used are trained as well as the originals.
[Table]
(cid:48) ]
[Table] [Table]