Structured Pruning for Efﬁcient ConvNets via Incremental Regularization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Parameter pruning is a promising approach for CNN compression and accelerationby eliminating redundant model parameters with tolerable performance loss.
Despite its effectiveness, existing regularization-based parameter pruning methodsusually drive weights towards zero with large and constant regularization factors,which neglects the fact that the expressiveness of CNNs is fragile and needs a moregentle way of regularization for the networks to adapt during pruning.
To solve thisproblem, we propose a new regularization-based pruning method (named IncReg )to incrementally assign different regularization factors to different weight groupsbased on their relative importance, whose effectiveness is proved on popular CNNscompared with state-of-the-art methods
Introduction
Recently, deep Convolutional Neural Networks (CNNs) have made a remarkable success in computervision tasks by leveraging large-scale networks learning from big amount of data.
However, CNNsusually lead to massive computation and storage consumption, thus hindering their deployment onmobile and embedded devices.
To solve this problem, many research works focus on compressing thescale of CNNs.
Parameter pruning is a promising approach for CNN compression and acceleration,which aims at eliminating redundant model parameters at tolerable performance loss.
To avoidhardware-unfriendly irregular sparsity, structured pruning is proposed for CNN acceleration [1, 23].
In the im2col implementation [2, 4] of convolution, weight tensors are expanded into matrices, sothere are generally two kinds of structured sparsity, i.e. row sparsity (or ﬁlter-wise sparsity) andcolumn sparsity (or shape-wise sparsity) [25, 24].
There are mainly two categories of structured pruning.
One is importance-based methods, whichprune weights in groups based on some established importance criteria [18, 20, 24].
The other isregularization-based methods, which add group regularization terms to learn structured sparsity [25,17, 9].
Existing group regularization approaches mainly focus on the regularization form (e.g. Group
LASSO [26]) to learn structured sparsity, while ignoring the inﬂuence of regularization factor.
Inparticular, they tend to use a large and constant regularization factor for all weight groups in thenetwork [25, 17], which has two problems.
Firstly, this ‘one-size-ﬁt-all’ regularization scheme has ahidden assumption that all weights in different groups are equally important, which however does nothold true, since weights with larger magnitude tend to be more important than those with smallermagnitude.
Secondly, few works have noticed that the expressiveness of CNNs is so fragile duringpruning that it cannot withstand a large penalty term from beginning, especially for large pruningratios and compact networks (like ResNet [8]).
AFP [7] was proposed to solve the ﬁrst problem,while ignored the second one.
In this paper, we propose a new regularization-based method named
IncReg to incrementally learn structured sparsity.
Apart from pruning, there are also other kinds of methods for CNN acceleration, such as low-rankdecomposition [16, 28], quantization [3, 5, 19, 21], knowledge distillation [10] and architecturere-design [12, 11, 27].
Our method is orthogonal to these methods.
The Proposed Method
Given a conv kernel, modeled by a 4-D tensor W ( l ) ∈ R N ( l ) × C ( l ) × H ( l ) × W ( l ) , where N ( l ) , C ( l ) , H ( l )and W ( l ) are the dimension of the l th ( 1 ≤ l ≤ L ) weight tensor along the axis of ﬁlter, channel,height and width, respectively, our proposed objective function for regularization can be formulatedas: E ( W )
=
L ( W ) + λ 2 R ( W ) + (cid:80)
Ll =1 (cid:80) G ( l ) g
=1 λ ( l ) g 2 R ( W ( l ) g ) , where W denotes the collection of allweights in the CNN;
L ( W ) is the loss function for prediction; R ( W ) is non-structured regularizationon every weight, i.e. weight decay in this paper; R ( W ( l ) g ) is the structured sparsity regularizationterm on group g of layer l and G ( l ) is the number of weight groups in layer l .
In [17, 25], the authorsused the same λ g for all groups and adopted Group LASSO [26] for R ( W ( l ) g ) .
In this work, since weemphasize the key problem of group regularization lies in the regularization factor rather than theregularization form, we use the most common regularization form weight decay, for R ( W ( l ) g ) , butwe vary the regularization factors λ g for different weight groups and at different iterations.
Our method prunes all the conv layers simultaneously and independently.
For simplicity, we omitthe layer notation l for following description.
All the λ g ’s are initialized to zero.
At each iteration,λ g is increased by λ newg = max( λ g + ∆ λ g , 0) .
Like AFP [7], we agree that unimportant weightsshould be punished more, so we propose a decreasing piece-wise linear punishment function (Eqn.1, Fig.1) to determine ∆ λ g .
Note that the regularization increment is negative (i.e. reward actually)when ranking is above the threshold ranking RG , since above-the-threshold means these weights areexpected to stay in the end.
Regularization on these important weights is not only unnecessary butalso very harmful via our experiment conﬁrmation.
[Equation]
where R is the pre-assigned pruning ratio for a layer, G is the number of weight groups, A is ahyper-parameter in our method to describe the maximum penalty increment (set to half of the originalweight decay in default), r is the ranking obtained by sorting in ascending order based on a proposedimportance criterion, which is essentially an averaged ranking over time, deﬁned as 1 N (cid:80)
Nn =1 r n ,where r n is the ranking by L 1 -norm at n th iteration, N is the number of passed iterations.
Thisaveraging is adopted as smoothing for a more stable pruning process.
As training proceeds, the regularization factors of different weight groups increase gradually, whichwill push the weights towards zero little by little.
When the magnitude of a weight group is lowerthan some threshold ( 10 − 6 ), the weights are permanently removed from the network, thus leadingto increased structured sparsity.
When the sparsity of a layer reaches its pre-assigned pruning ratio R , that layer automatically stops structured regularization.
Finally, when all conv layers reach theirpre-assigned pruning ratios, pruning is over, followed by a retraining process to regain accuracy.
Experiments
3.1 Analysis with ConvNet on CIFAR-10
We ﬁrstly compare our proposed IncReg with other two group regularization methods, i.e. SSL [25]and AFP [7], with ConvNet [15] on CIFAR-10 [14], where both row sparsity and column sparsity areexplored.
Caffe [13] is used for all of our experiments.
Experimental results are shown in Tab.1.
Wecan see that IncReg consistently achieves higher speedups and accuracies than the other two constantregularization schemes.
Notably, even though AFP achieves similar performance as our methodunder relatively small speedup (about 4 . 5 × ), when the speedup ratio is large (about 8 × ∼ 10 × ), ourmethod outperforms AFP by a large margin.
We argue that this is because the incremental way ofregularization gives the network more time to adapt during pruning, which is especially important inface of large pruning ratios.
G
A
A r g
R=0.2 R=0.
R=0.6 R=0.
[Table]
[Table]
3.2 VGG-16 and ResNet-50 on ImageNet
We further evaluate our method with VGG-16 [22] ( 13 conv layers) and ResNet-50 [8] ( 53 convlayers) on ImageNet [6].
We download the open-sourced caffemodel as our pre-trained model,whose single-view top-5 accuracy on ImageNet validation dataset is 89 .
6% (VGG-16) and 91 . 2%(ResNet-50).
For VGG-16, following SPP [24], the proportion of remaining ratios of low layers( conv1_x to conv3_x ), middle layers ( conv4_x ) and high layers ( conv5_x ) are set to 1 : 1 . 5 : 2 , foreasy comparison.
For ResNet-50, constant pruning ratio 0 . 4 is adopted for all conv layers.
Pruning isconducted with batch size 64 and ﬁxed learning rate 0 . 0005 , followed by retraining with batch size256 .
Experimental results are shown in Tab.2.
On VGG-16, our method is slightly better than CP and
SPP, and outperforms FP by a signiﬁcant margin.
Notably, since we use the same pruning ratios as
SPP does, the only explanation for the performance improvement should be a better pruning processitself, guided by our incremental regularization scheme.
On ResNet-50, our method is signiﬁcantlybetter than CP and SPP, demonstrating the effectiveness of IncReg when pruning compact networks.
Moreover, to conﬁrm the actual speedup, we also evaluate our method with VGG-16 on CPU and
GPU.
The result is shown in Tab.3.
[Table]
[Table]
[Table]
[Table]
Conclusion
We propose a new structured pruning method based on an incremental way of regularization, whichhelps CNNs to transfer their expressiveness to the rest parts during pruning by increasing theregularization factors of unimportant weight groups little by little.
Our method is proved to becomparably effective on popular CNNs compared with state-of-the-art methods, especially in face oflarge pruning ratios and compact networks.