M ODEL C OMPRESSION VIA H YPER -S TRUCTURE N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a novel channel pruning method to solve the problem of compression and acceleration of Convolutional Neural Networks (CNNs).
Previous channel pruning methods usually ignore the relationships between channels and layers.
Many of them parameterize each channel independently by using gates or similar concepts.
To ﬁll this gap, a hyper-structure network is proposed to generate the architecture of the main network.
Like the existing hypernet, our hyperstructure network can be optimized by regular backpropagation.
Moreover, we use a regularization term to specify the computational resource of the compact network.
Usually, FLOPs is used as the criterion of computational resource.
However, if
FLOPs is used in the regularization, it may over penalize early layers.
To address this issue, we further introduce learnable layer-wise scaling factors to balance the gradients from different terms, and they can be optimized by hyper-gradient descent.
Extensive experimental results on CIFAR-10 and ImageNet show that our method is competitive with state-of-the-art methods.
I NTRODUCTION
Convolutional Neural Networks (CNNs) have accomplished great success in many machine learning and computer vision tasks (Krizhevsky et al., 2012; Redmon et al., 2016;
Ren et al., 2015; Simonyan & Zisserman, 2014a; Bojarski et al., 2016).
To deal with real world applications, recently, the design of CNNs becomes more and more complicated in terms of width, depth, etc.
(Krizhevsky et al., 2012;
Simonyan & Zisserman, 2014b; He et al., 2016; Huang et al., 2017).
Although these complex CNNs can attain better performance on benchmark tasks, their computational and storage costs increase dramatically.
As a result, a typical application based on CNNs can easily exhaust an embedded or mobile device due to its enormous costs.
Given such costs, the application can hardly be deployed on resource-limited platforms.
To tackle these problems, many methods (Han et al., 2015b;a) have been devoted to compressing the original large CNNs into compact models.
Among these methods, weight pruning and structural pruning are two popular directions.
Unlike weight pruning or sparsiﬁcation, structural pruning, especially channel pruning, is an effective way to truncate the computational cost of a model because it does not require any post-processing steps to achieve actual acceleration and compression.
Many existing works (Liu et al., 2017; Ye et al., 2018; Huang & Wang, 2018;
Kim et al., 2020; You et al., 2019) try to solve the problem of structure pruning by applying gates or similar concepts on channels of a layer.
Although these ideas have achieved many successes in channel pruning, there are some potential problems.
Usually, each gate has its own parameter, but parameters from different gates do not have dependence.
As a result, they can hardly learn inter-channel or inter-layer relationships.
Due to the same reason, the slimmed models from these methods could overlook the information between different channels and layers, potentially bringing sub-optimal model compression results.
To address these challenges, we propose a novel channel pruning method inspired by hypernet (Ha et al., 2016).
In hypernet, they propose to use a hyper network to generate the weights for another network, while the hypernet can be optimized through backpropagation.
We extend a hypernet to a hyper-structure network to generate an architecture vector for a CNN instead of weights.
Each architecture vector corresponds to a sub-network from the main (original) network.
By doing so, the inter-channel and inter-layer relationships can be captured by our hyper-structure network.
Besides the hyper-structure network, we also introduce a regularization term to control the computational budget of a sub-network.
Recent model compression methods focus on pruning computational
FLOPs instead of parameters.
The problem of applying FLOPs regularization is that the gradients of the regularization will heavily penalize early layers which can be regarded as a bias towards latter layers.
Such a bias will restrict the potential search space of sub-networks.
To make our hyper-structure network explore more possible structures, we further introduce layer-wise scaling factors to balance the gradients from different losses for each layer.
These factors can be optimized by hyper-gradient descent.
Our contributions are summarized as follows:
1) Inspired by hypernet, we propose to use a hyper-structure network for model compression to capture inter-channel and inter-layer relationships.
Similar to hypernet, the proposed hyper-structure network can be optimized by regular backpropagation.
Gradients from FLOPs regularization are biased toward latter layers, which truncate the potential search space of a sub-network.
To balance the gradients from different terms, layerwise scaling factors are introduced for each layer.
These scaling factors can be optimized through hyper-gradient descent with trivial additional costs.
3) Extensive experiments on CIFAR-10 and ImageNet show that our method can outperform both conventional channel pruning methods and AutoML based pruning methods on ResNet and MobileNetV2.
R ELATED W ORKS 2. M ODEL C OMPRESSION
Recently, model compression has drawn a lot of attention from the community.
Among all model compression methods, weight pruning and structural pruning are two popular directions.
Weight pruning eliminates redundant connections without assumptions on the structures of weights.
Weight pruning methods can achieve a very high compression rate while they need specially designed sparse matrix libraries to achieve acceleration and compression.
As one of the early works, Han et al. (2015b) proposes to use L 1 or L 2 magnitude as the criterion to prune weights and connections.
SNIP (Lee et al., 2019) updates the importance of each weight by using gradients from loss function.
Weights with lower importance will be pruned.
Lottery ticket hypothesis (Frankle & Carbin, 2019) assumes there exist high-performance sub-networks within the large network at initialization time.
They then retrain the sub-network with the same initialization.
In rethinking network pruning (Liu et al., 2019b), they challenging the typical model compression process (training, pruning, ﬁne-tuning), and argue that ﬁne-tuning is not necessary.
Instead, they show that training the compressed model from scratch with random initialization can obtain better results.
One of the previous works (Li et al., 2017) in structural pruning uses the sum of the absolute value of kernel weights as the criterion for ﬁlter pruning.
Instead of directly pruning ﬁlters based on magnitude, structural sparsity learning (Wen et al., 2016) is proposed to prune redundant structures with Group Lasso regularization.
On top of structural sparsity, GrOWL regularization is applied to make similar structures share the same weights (Zhang et al., 2018).
One of the problems when using
Group Lasso is that weights with small values could still be important, and it’s difﬁcult for structures under Group Lasso regularization to achieve exact zero values.
As a result, Louizos et al.
(2018) propose to use explicit L 0 regularization to make weights within structures have exact zero values.
Besides using the magnitude of structure weights as a criterion, other methods utilize the scaling factor of batchnorm to achieve structure pruning, since batchnorm (Ioffe & Szegedy, 2015) is widely used in recent neural network designs (He et al., 2016; Huang et al., 2017).
A straightforward way to achieve channel pruning is to make the scaling factor of batchnorm to be sparse (Liu et al., 2017).
If the scaling factor of a channel fell below a certain threshold, then the channel will be removed.
The scaling factor can also be regarded as the gate parameter of a channel.
Methods related to this concept include (Ye et al., 2018;
Huang & Wang, 2018;
Kim et al., 2020; You et al., 2019).
Though it has achieved many successes in channel pruning, using gates can not capture the relationships between channels and across layers.
Besides using gates, Collaborative channel pruning (Peng et al., 2019) try to prune channels by using Taylor expansion.
Our method is also related to Automatic
Model Compression(AMC)
(He et al., 2018b).
In AMC, they use policy gradient to update the policy
GRU
GRU
GRU 𝑎𝑎𝑎 𝐿 [ 1… 1 ] [ 0… 1 ] [ 1… 1 ]
Sub-network Architecture
Number of Output Channels
N u m b e r
o f
I npu t
C h a nn e l s
Remained channels
Layer i
Feature map
Layer i+ 𝒎𝒊𝒏 𝜽 𝑳 + 𝝀𝑹
Hyper
[Figure]
Structure Network
Convolutional Neural Network
W e i g h t
T e n s o r
Generate Architecture Vector
Optimize HSN by evaluating the sub-network network, which potentially provides both inter-channel and inter-layer information.
However, the high variance of policy gradient makes it less efﬁcient and effective compared to our method.
In this paper, we focus on channel pruning, since it provides a natural way to reduce computation and memory costs.
Besides weight and channel pruning methods, there are works from other perspectives, including bayesian pruning (Molchanov et al., 2017; Neklyudov et al., 2017), weight quantization (Courbariaux et al., 2015; Rastegari et al., 2016), and knowledge distillation (Hinton et al., 2015).
2. H YPERNET
Hypernet (Ha et al., 2016) was introduced to generate weights for a network by using a hyper network.
Hyper networks have been applied to many machine learning tasks.
von Oswald et al.
(2020) uses a hyper network to generate weights based on task identity to combat catastrophic forgetting in continual learning.
MetaPruning (Liu et al., 2019a) utilizes a hyper network to generate weights when performing evolutionary algorithm.
SMASH (Brock et al., 2018) is a neural architecture search method that can predict the weights of a network given its architecture.
GPN (Zhang et al., 2019) extends the idea of SMASH and can be used on any directed acyclic graph.
Other applications include
Bayesian neural networks (Krueger et al., 2017), multi-task learning (Pan et al., 2018), generative models (Suarez, 2017) and so on.
Different from original hyper network, the proposed hyper-structure network aims to generate the architecture of a sub-network.
P ROPOSED M ETHOD 3. N OTATIONS
To better describe our proposed approach, necessary notations are introduced ﬁrst.
In a CNN, the feature map of i th layer can be represented by F i ∈ (cid:60) C
i × W
i × H
i , i = 1 , . . .
, L , where C i is the number of channels, W i and H i are height and width of the current feature map, L is the number of layers.
The mini-batch dimension of feature maps is ignored to simplify notations.
sigmoid ( · ) is the sigmoid function.
round ( · ) rounds inputs to nearest integers.
3. H YPER -S TRUCTURE N ETWORK
In the context of channel pruning, we need to decide whether a channel should be pruned or not.
We can use 0 or 1 to depict the removal or keep of a channel.
Consequently, the architecture of a sub-network can be represented as a concatenated vector (containing 0 or 1) from all layers.
Our goal is then to use a neural network to generate this vector to represent the corresponding sub-network.
[Figure]
For i th layer, the following output vector is generated:
[Equation]
where HSN is our proposed hyper-structure network composed of gated recurrent unit (GRU) (Cho et al., 2014) and dense layers, a i is a ﬁxed random vector generated from a uniform distribution
U (0 , 1) , and Θ is the parameter of HSN.
The detailed setup of HSN can be found in Appendix C.
In short, GRU is used to capture sequential relationships between layers, and dense layers are capable of capturing inter-channel relationships.
Note that a i is a constant vector during training, if a i is randomly sampled, it will make learning more difﬁcult and result in sub-optimal performance.
Now we have the output o i , we need to convert it to a 0-1 vector to evaluate the sub-network.
The binarization process can be demonstrated by the following equations:
[Equation]
where g follows Gumbel distribution: g ∼ Gumbel (0 , 1) , v i is the architecture vector of i th layer, and τ is the temperature hyper-parameter.
Since the round operation is not differentiable, we use straight through estimator (STE) (Bengio et al., 2013) to enable gradient calculation: ∂ J ∂z i =
∂
J ∂v i .
This process can be summarized as using ST Gumbel-Softmax (Jang et al., 2016) with ﬁxed temperature to approximate Bernoulli distribution.
The idea of HSN can also be viewed as mapping from constant vectors { a i } Li =1 to the architecture of a sub-network.
When we evaluate a sub-network, the feature map of i th layer is modiﬁed as follows:
[Equation]
where (cid:12) is element-wise multiplication, ˆ v i is the expanded version of v i , and ˆ v i has the same size of
F i .
The feature map F i is from the output of Conv-Bn-Relu block.
The overall loss function is:
[Equation]
where v = ( v 1 , . .
. , v L )
, T ( v ) is the current FLOPs decided by the architecture vector v , T total is the total FLOPs of the original model, p ∈ (0 , 1] is a predeﬁned parameter deciding the remaining fraction of FLOPs, λ is the hyper-parameter controlling the strength of FLOPs regularization, f ( x ; W , v ) is the CNN parameterized by W and the sub-network structure is determined by architecture vector v , L is the cross entropy loss function and R is the regularization term for FLOPs, and Θ again is the parameters of HSN.
The regularization term R used in this paper is R ( T ( v ) , pT total ) =
log( | T ( v ) − pT total | + 1) .
3. L AYER -W ISE S CALING
The FLOPs regularization considered in Eq.
4 will heavily penalize layers with a larger amount of
FLOPs (early layers for most architectures).
Consequently, the resulting architecture from the original
FLOPs regularization will have a larger pruning rate at early layers.
The alternative architectures with similar FLOPs could be omitted.
This phenomenon is also demonstrated in Fig.
Further analysis is provided in Appendix B.
To alleviate the problem caused by original FLOPs regularization, we introduce layer-wise scaling factors to dynamically balance gradients from the regularization term R and the loss term L .
Only endreturn the gradients in dense layers are balanced since GRU is shared by all layers.
The gradients w.r.t the parameters of i th dense layer can be written in the following equation:
[Equation] [Equation]
where θ i is the parameter of i th dense layer, α i is the layer-wise scaling factor for i th layer.
If no layer-wise scaling is applied, α i = 1 .
α i can be regarded as a balancing factor between ∂ R ∂θ i and ∂ L ∂θ
i .
α i only appears in gradient calculation, as a result, it can not be directly optimized.
To optimize α i , we follow similar deriving process from (Baydin et al., 2018).
We ﬁrst deﬁne the update rule θ ti = u
( θ t − 1 i , α ti ) and it can be applied to any optimization algorithms.
For example, under stochastic gradient descent, u ( θ t − 1 i ,
[Equation]
Ideally, our goal is to update α i so that the corresponding architecture can obtain lower loss value with loss function J .
To do so, we want to min α
i J ( u ( θ t − 1 i , α ti )) before update θ i .
For simplicity, the expectation is omitted.
The hyper-gradient with respect to α i can be calculated by:
[Equation]
Given the hyper-gradient of α i , it can be updated by regular gradient descent method.
In experiments, the update rule is ADAM optimizer (Kingma & Ba, 2014), and the detail derivation of ∂J [Equation] u [Equation]
θ
[Equation]
1 i [Equation] α ti )) ∂α i for ADAM optimizer is described in Appendix E.
3. M ODEL C OMPRESSION VIA H YPER -S TRUCTURE N ETWORK
In Fig. 6, we provide the ﬂowchart of HSN. The overall algorithm of model compression via hyperstructure network is shown in Alg.
As shown in Alg. 1, our method can prune any pre-trained CNNs without modiﬁcations.
It should be emphasized again that the gradient of GRU is not affected by α i , which is simply ∂ L ∂θ GRU + λ ∂ R ∂θ GRU , and θ GRU is the parameter for GRU.
Moreover, HSN does not need a whole dataset for training, and a small fraction of the dataset is enough, which makes the training of HSN quite efﬁcient.
After the training of HSN, we then use HSN to generate an architecture vector v , and prune the model according to this vector.
Also, note that there is certain randomness (Eq. 2 approximates Bernoulli distribution) when generating v , but we ﬁnd that there is no need to generate the vector multiple times, and average them or conduct majority vote.
When generating the vector multiple times, most parts of vectors are the same, the different parts are trivial and do not have impacts on the ﬁnal performance.
[Table]
E XPERIMENTAL R ESULTS 4. I MPLEMENTATION D ETAILS
Similar to many model compression works, CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009) are used to evaluate the performance of our method.
Our method requires one hyper-parameter p to control the FLOPs budget.
The detailed choices of p are listed in Appendix
F.
For CIFAR-10, we compare with other methods on ResNet-56 and MobileNetV2.
For ImageNet, we select ResNet-34, ResNet-50, ResNet-101 and MobileNetV2 as our target models.
The reason we choose these models is because that ResNet (He et al., 2016) and MobileNetV2 (Sandler et al., 2018) are much harder to prune than earlier models like AlexNet (Krizhevsky et al., 2012) and
VGG (Simonyan & Zisserman, 2014b).
λ decides the regularization strength in our method.
We choose λ
= 4 in all CIFAR-10 experiments and λ = 8 for all ImageNet experiments.
For CIFAR-10 models, we train ResNet-56 from scratch following the pytorch examples.
After pruning, we ﬁnetune the model for 160 epochs using SGD with a start learning rate 0 .
1 , weight decay 0 . 0001 and momentum 0 .
8 , the learning rate is multiplied by 0 .
1 at epoch 80 and 120 .
For
ImageNet models, we directly use the pre-trained models released from pytorch (Paszke et al., 2017; 2019).
After pruning, we ﬁnetune the model for 100 epochs using SGD with a start learning rate 0 .
01 , weight decay 0 . 0001 and momentum 0 .
9 , and the learning rate is scaled by 0 .
1 at epoch 30 ,and 90 .
For MobileNetV2 on ImageNet, we choose weight decay as 0 .
which is the same with the original paper (Sandler et al., 2018).
For the training process of HSN, we use ADAM (Kingma & Ba, 2014) optimizer with a constant learning rate 0 .
001 and train HSN for 200 epochs.
τ in Eq.
2 is set as 0.4.
The β for α i is chosen as 0 . 01 , and α i is updated as shown in Alg. 1.
To build dataset D HSN , we random sample 2 , 500 and 10 , 000 samples for for CIFAR-10 and ImageNet separately.
In the experiments, we found that a stand-alone validation set is not necessary, all samples in D HSN come from the original training set.
All codes in this paper are implemented with pytorch (Paszke et al., 2017; 2019).
The experiments are conducted on a machine with 4 Nvidia Tesla P40 GPUs.
4. CIFAR-10 R ESULTS
In Tab. 1, we present the comparison results on CIFAR-10 dataset.
Our method is abbreviated as
MCH ( M odel C ompression via H yper Structure Network) in the experiment section.
For ResNet56, our method can prune 50% of FLOPs while obtain 0 .
24% performance gain in accuracy.
On
MobileNetV2, our method can obtain 0 .
38% gain in accuracy.
Compared to all other methods, our method can achieve the best results.
Our method can outperform the second best method (DCP-Adapt) by 0 .
23% on ResNet-56.
On MobileNetV2, our method can outperform the the second best method by 0 .
16% while pruning 14% more FLOPs.
For both models, our method performs much better than early methods (He et al., 2017; 2018b; Li et al., 2017;
He et al., 2018a).
Our method can outperform [Table]
CCP by 0 .
32% in terms of ∆ -Acc, which demonstrate that learning both inter-layer and inter-channel relationships are better than only considering inter-channel relationships.
4. I MAGE N ET R ESULTS
In Tab. 2, the results on ImageNet are presented, Top-1/Top-5 accuracy after pruning are presented.
Most of our comparison methods comes from recently published papers including IE (Molchanov et al., 2019), FPGM (He et al., 2019), GAL (Lin et al., 2019), CCP (Peng et al., 2019), MetaPruning (Liu et al., 2019a), GBN (You et al., 2019),
Hinge (Li et al., 2020), abd HRank (Lin et al., 2020).
ResNet-34 .
Our proposed MCH can prune 44 .
0% of FLOPs with 0 .
45% and 0 .
27% performance loss on Top-1 and Top-5 accuracy.
Such a result is better than any other method.
Proposed MCH performs similarly compared to IE (Molchanov et al., 2019) in Top-1 and ∆ Top-1 accuracy ( 72 .
85% / − 0 .
45% vs. 72 .
83% / − 0 .
43% ), while our method can prune almost 20% more FLOPs.
Given similar
FLOPs pruning rate, our method achieves better results compared to FPGM (He et al., 2019)
( − 0 .
45% / − 0 .
27% vs. − 1 .
29% / − 0 .
54% for ∆ Top-1/ ∆ Top-5).
Besides IE and FPGM, The margin between our method and rest methods are even larger.
ResNet-50 .
ResNet-50 is a very popular model for evaluating model compression methods.
With such intense competition, our method can still achieve the best Top-1/Top-5 and ∆ Top-1/ ∆ Topresults.
The second best method in terms of Top-1 accuracy is MetaPruning (Liu et al., 2019a), which can achieve 75 .
40% Top-1 result after pruning.
Our method outperforms MetaPruning by 0 .
20% in Top-1 accuracy while our method can prune 5% more FLOPs.
MetaPruning utilizes hypernet to generate weights when evaluating sub-networks, however, such a design paradigm prohibits
MetaPruning to be directly used on pre-trained models.
The weights inherited from the pre-trained model might be one of the reasons why our method can outperform MetaPruning.
GBN (You et al., 2019) obtains the second-best ∆ Top-1 accuracy, however, the accuracy after pruning is quite low compared to other methods.
Our method can outperform GBN by 0 .
42% in Top-1 accuracy.
Besides
GBN and MetaPruning, our method can outperform two recent methods
HRank (Lin et al., 2020) and
Hinge (Li et al., 2020) by 0 .
62% to 0 .
90% on Top-1 accuracy.
ResNet-101 .
For ResNet-101, our method can increase the performance of the baseline model by 0 .
21% and 0 .
25% on Top-1 and Top-5 accuracy, while removes 56% of FLOPs.
The second best [Figure] [Figure] method FPGM (He et al., 2019) can maintain the performance and reducing 41% of FLOPs.
In short, compared to FPGM, our method can obtain performance gain while pruning 15% more FLOPs.
MobileNetV2 .
On MobileNetV2, we mainly compare with AMC (He et al., 2018b) and MetaPruning (Liu et al., 2019a).
Both of them can be regarded as representative works for AutoML related model compression methods (AMC uses reinforcement learning; MetaPruning uses evolutionary algorithm and hypernet).
Our method can achieve 71 .
54% Top-1 accuracy while pruning around 30% of FLOPs, which is 0 .
34% and 0 .
74% higher than MetaPruning and AMC.
These results show that our method can outperform AutoML based methods.
In summary, our method can outperform these comparison methods and achieve the state-of-the-art performance.
These experimental results also indicate that inter-channel and inter-layer relationships should be considered when designing model compression methods.
4. E FFECTS OF L AYER - WISE S CALING
We further study the impact of λ and layer-wise scaling (LWS) when training HSN on CIFAR-10.
In Fig.
3 (a,b), we can see that changing λ does not have a large impact on the ﬁnal performance of a sub-network, and our method is not sensitive to it.
One possible reason is that α i adapts to λ when using ADAM optimizer. In general, we do not spend too much time on tuning λ .
In Fig.(c,d), it shows that using LWS can improve the ﬁnal performance of a sub-network and obtain lower loss.
Moreover, early layers usually have a larger preserved rate with LWS as shown in Fig 4, indicating that alternative sub-network architectures can be discovered from LWS.
Without LWS, the ﬁnal performance of ResNet-56 will decrease 0 .
19% , achieves 93 .
04% ﬁnal accuracy on CIFAR-10.
Similar observations hold for MobileNetV2 ( 94 .
45% ﬁnal accuracy and the relative gap is 0 .
These observations show that LWS indeed helps the training of the HSN. 4. D ETAILED A NALYSIS
In this section, we provide detailed analysis to answer the following questions: (1) why we use ﬁxed inputs for a i ?
(2) Can we replace HSN with dense layers?
(3) Does LWS work for different learning rate settings?
(4) Does LWS still work for other optimization methods?
[Figure]
To answer the ﬁrst question, we examine three different settings: learnable inputs, ﬁxed inputs, and randomly generated inputs from the uniform distribution.
From Fig.
5 (a,b), we observe that ﬁxed inputs have similar performance to learned inputs, and both of them outperform random inputs.
The idea of using ﬁxed inputs is that we want to project the optimal sub-network to ﬁxed vectors in the input space, which is generally simple (compared to learned inputs) and easy to train (compared to random inputs).
The above results justify why we use ﬁxed inputs.
To verify the effectiveness of different components of HSN, we use three different settings: vanilla
HSN, HSN only with dense layers and gates (deﬁnition is given in Appendix).
From Fig.
5 (c,d), it can be shown that HSN has the best performance, which again shows that we should not separately treat each channel or each layer.
In Fig.
5 (e,f), we plot training curves for different learning rates with or without LWS.
It can be seen that LWS can lead to better performance, given different learning rates.
Finally, in Fig 5 (g,h), we examine whether LWS is still useful given two additional optimizer: SGD and LARS (You et al., 2017).
LARS applies layer-wise learning rates on overall gradients, which can be complementary to
LWS.
When applying LWS on these two methods, it still improves performance.
SGD is not a good choice when the optimization involves discrete values, as suggested by the previous study (Alizadeh et al., 2019).
C ONCLUSION
In this paper, we proposed a hyper-structure network for model compression to capture inter-channel and inter-layer relationships.
An architecture vector can be generated from HSN to select a subnetwork from the original model.
At the same time, we evaluated this sub-network by using classiﬁcation and resource losses.
The HSN can be updated by the gradients from them.
Moreover, we also identiﬁed the problem of FLOPs constraint (bias towards latter layers), which limits the ﬁnal search space of HSN.
To solve it, we further proposed layer-wise scaling to balance the gradients.
With the aforementioned novel techniques, our method can achieve state-of-the-arts performance on
ImageNet with four different architectures.
A V ISUALIZATION OF P RUNED A RCHITECTURES
In Fig 6, we visualize the pruned architecture for ResNet-50 and MobileNetV2.
B B IAS OF FLOP S R EGULARIZATION
We brieﬂy discuss two types of FLOPs regularization used in our paper and trainable gate (TG)
(Kim et al., 2020).
First, we provide the speciﬁc deﬁnition of T ( v i ) (FLOPs of i th layer):
[Equation]
where G i is the number of groups in a convolution layer, K i is the kernel size, 1 is a all one vector, and 1 T v i is the number of perversed channels in i th layer.
With T ( v i ) , T ( v ) =
(cid:80) Li =1 T ( v i ) .
In TG, they simply use mean square error (MSE) as the regularization term, and in their paper
R MSE ( T ( v ) , pT total ) =
( T ( v ) − pT total ) 2 .
The gradients w.r.t v
i is:
[Equation]
For the regularization used in our method: R ( T ( v ) , pT total ) = log
[Equation]
v ) − pT total | +1) , the gradients w.r.t v
i is:
[Equation]
[Equation]
For both regularization functions, the ratio between the gradients w.r.t v i of two layers k, j is
[Equation]
[Figure]
[Equation]
− 1 G l
W j H j .
Take ResNet-50 as an example, let j, k be the middle layers of a bottleneck block, and we random initialize HSN.
If j is in the ﬁrst block, and k is in the last block, then
K k = K j = 3 , W j = H j = 56 , W k = H k
= 7 , 1 T v j − 1 ≈ 0 .
5 × 64 (due to random initialization), 1 T v k − 1 ≈ 0 .
5 × 512 , ﬁnally, ∂T ( v k ) ∂v k / ∂T ( v j ) ∂v j
≈ 3 × 3 × 256 × 7 × 7 3 × 3 × 32 × 56 × 56 ≈ 18 , which is not trivial.
When calculating the gradients w.r.t θ i , we have ∂ R ∂θ
i =
c R ∂T ( v i ) ∂v i
∂v i
∂θ i , all θ i share the same c R decided by the regularization function.
Without loss of generality, we assume the magnitude of ∂v
i ∂θ i is similar given different layers.
The assumption is based on the following derivation (to simplify derivation, we omit weight norm in dense layers):
[Equation]
where sigmoid ( x )(1 − sigmoid ( x )) ≤ 14 , and b
i is the input to i th dense layer, which is also the outputs of GRU.
Since all b i have the same shape, and weights in GRU are normalized, we can assume all b i have similar magnitude.
Since 14 τ b Ti is a upper bound of ∂v
i ∂θ i , similar assumptions can be made.
Following this assumption, the relative magnitude of gradients w.r.t θ j and θ k for layers j, k can be roughly represented by ∂T ( v k ) ∂v k / ∂T ( v j ) ∂v j .
After training for a while, the ratio might be smaller, however, it only indicates that early layers are more aggressively pruned.
Thus, when applying
FLOPs regularization, it penalizes early layers much heavier compared to latter layers.
One should also note that this is a general problem when using gradient based model compression methods with the FLOPs regularization.
It’s quite hard to circumvent calculating ∂T ( v i ) ∂v i as in
TG (Kim et al., 2020) and our paper.
[Figure]
[Figure]
C D ETAILED S ETUP OF H YPER - STRUCTURE N ETWORK
In Tab. 3, we present the architecture of HSN.
The forward calculation is:
[Equation]
where h i and b i are hidden states and outputs of GRU at step i ,
o i is the ﬁnal output of HSN.
GRU also requires hidden layer input at time-step 0 h 0 .
In the experiment, the h 0 is a all zero tensor.
As mentioned in Tab. 3, the dimension of a i is 64.
Since a i is a single input instead of a mini-batch, we cannot apply batchnorm.
To make the training more stable, we use weight norm (Salimans &
Kingma, 2016) on both GRU and dense layers.
Initially, we tried to use a huge dense layer (input size 64, output size C 1 + C 2 + · · · + C L ) as HSN.
However, we ﬁnd that the huge dense layer is hard to optimize and also parameter heavy.
To verify the strength of the proposed HSN, we can instead use a simpliﬁed setting to prune neural networks, which is shown as follows:
[Equation]
[Figure]
[Figure]
where the architecture vector is parameterized by ˆ θ i .
Under this setting, the parameter for each channel does not have relationships.
We use this setting to prune ResNet-56 and MobileNetV2 on
CIFAR-10, the results are shown in Fig.
From the ﬁgure, we can see that the performance and convergence speed of using HSN is much better.
Under high dimensional setting, like MobileNetV2, the simpliﬁed setting shown in Eq.
11 can not learn efﬁciently, which demonstrate that capturing inter-channel and inter-layer relationships are crucial for pruning deep neural networks.
D F ORWARD AND B ACKWARD P RUNING
Here, we refer forward pruning as start pruning from a random sub-network, and refer backward pruning as start pruning from the original large model.
Many model compression methods use backward pruning.
We also provide a simple way to extend our method to backward pruning.
When we binarize the output of HSN, we can add a constant c :
[Equation]
(12) where g ∼ Gumbel (0 , 1) , and the Gumbel (0 , 1) distribution can be sampled using inverse transform sampling by drawing u ∼ U (0 , 1) and computing g = − log( − log( u )) .
When the constant c is big enough, it will make v i become an all one vector, thus the sub-network produced by HSN will start from the original large CNN.
If we set c to 0, then it will start from a random sub-network.
In Fig.
8, we show the results of forward and backward pruning.
It can be seen that they can achieve similar sub-network performance, but the changes in regularization loss various dramatically.
E D ERIVATIVE OF H YPER -G RADIENT WITH ADAM O PTIMIZER
The update rule of ADAM for θ i is shown in Alg. 2, and it is:
[Equation]
Input :
[Equation]
: learning rate and decay rate for ADAM.
Initialize m 0 , n 0 , t =
Update rule at step t :
[Equation]
Then the derivation of ∂u ( θ t − 1 i ,α ti ) ∂α i is:
[Equation]
{++ (cid:15) ) 
+}
(1 − β t 2 ) (cid:9) , where ∂ ˆ m t
∂α
i
=
(1 − β 1 ) ∂
L ∂θt − 1 i 1 − β t and ∂ ˆ n t ∂α i =
2(1 − β 2 ) (cid:0) α ti ( ∂ L ∂θt − 1 i ) 2 + λ ∂ L ∂θt − 1 i ∂ R ∂θt − 1 i (cid:1) 1 − β t. Recall that when updating α t − 1 i to α ti , we have to compute:
[Equation]
[Equation]
We need α ti when updating θ t − 1 i to θ ti .
Thus at each update step, it requires an extra copy of θ t − 2 i and parameters of ADAM to compute ∂u ( θ t − 2 i ,α t − 1 i ) ∂α i .
The cost of the extra storage is trivial.
F C HOICE OF p G IVEN D IFFERENT D ATASETS AND A RCHITECTURES .
[Table]