G LOBALLY S OFT F ILTER P RUNING F OR E FFICIENT C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper propose a cumulative saliency based Globally Soft Filter Pruning (GSFP) scheme to prune redundant ﬁlters of Convolutional Neural Networks (CNNs).
Speciﬁcally, the GSFP adopts a robust pruning method, which measures the global redundancy of the ﬁlter in the whole model by using the soft pruning strategy.
In addition, in the model recovery process after pruning, we use the cumulative saliency strategy to improve the accuracy of pruning.
GSFP has two advantages over previous works: (1) More accurate pruning guidance .
For a pre-trained CNN model, the saliency of the ﬁlter varies with different input data.
Therefore, accumulating the saliency of the ﬁlter over the entire data set can provide more accurate guidance for pruning.
On the other hand, pruning from a global perspective is more accurate than local pruning.
(2) More robust pruning strategy .
We propose a reasonable normalization formula to prevent certain layers of ﬁlters in the network from being completely clipped due to excessive pruning rate.
Experiment results show that GSFP is effective on many classic CNN architectures and different data sets.
Within my knowledge, GSFP is the ﬁrst algorithm to combine global and soft pruning strategies.
Notably, on MNIST and CIFAR10, it achieves a much higher compression ratio compared with prior work while maintaining the same test accuracy.
INTRODUCTION
In recent years, convolutional neural networks (CNNs) have developed rapidly and achieved remarkable performance, which is widely applied in the ﬁeld of computer vision, natural language processing, etc.
However, CNNs with high computational and storage volumes, in fact, are difﬁcult to deploy on embedded systems.
As a result, model compression for CNNs have received hot attention.
Pruning-based approachs aim to remove the unnecessary connections of the neural network.
parameter pruning which include non-structured pruning and structured pruning, reduces network complexity by directly reducing the scales of weight.
Non-structured pruning (Han et al.
(2015);Guo et al.
(2016)) randomly deletes redundant neurons reducing model scales greatly.
However, it is necessary to store additional coordinate information, resulting in a weak advantage in reducing compuational effort.
Structured pruning (Wen et al.
(2016);Mao et al.
(2017);Luo et al.
(2017);Zhu & Gupta (2018);He et al.
(2018)) takes into account both the amount of computation and storage required.
It performs regular model clipping based on ﬁlter, channel, and even a layer.
Since the model pruned has good regularity, it can signiﬁcantly reduce the scales of calculation and storage at the same time, which has recently received extensive attention.
Essentially, the work of this paper is based on the idea of ﬁlter pruning.
In terms of Filter pruning, the current algorithm can be divided into “Soft” and “Hard” according to the training strategy, or “Local” and “Global” according to the pruning range.
“Hard” is divided into two cases.
One is that each iteration performs a pruning operation, so the pruning is too frequent to allow enough time recovering the model capacity.
The second is to manually cut off the gradient update after pruning and then retraining(Luo et al.
(2017), Dong et al.
(2017), Li et al.
“Soft” means that each epoch performs a pruning operation, and the clipped ﬁlter can be updated during the retraining process(He et al.
“Local” means to set the pruning rate for each layer(He et al.
(2017), Mao et al.
(2017)), and “Global” means that only one overall pruning rate is set, and the pruning rate of each layer is obtained by dynamic adjustment(Lin et al.
Essentially, the work of this paper is based on the pruning strategy of global and soft.
Most of the previous works on ﬁlter pruning, especially with the global pruning strategy, is poorly robust.
That is, the distribution of pruning rate in each layer is extremely uneven.
The ﬁlters of some layers are cut off a lot, while some layers are hardly cut.
In extreme cases, when setting a too high pruning rate, the global pruning strategy will cause the ﬁlters of a certain layer removed completely, resulting in low accuracy.
For example, analyzing the pruning results of VGG-16 in Li et al.
(2017) did not trim the third to seventh convolutional layers, while the percentage of FLOPs in other convolutional layers was reduced by more than 50% on CIFAR-10, as shown in Table 3.
In Lin et al.
the overall VGG-16 pruning rate is about 50% on ImageNet, but the CONV3layer pruned FLOPs is as high as 98% .
In extreme cases, the pruning results of VGG-16 on CIFAR10 are shown in Table 1, both the GHFP and the GSFP methods produce a low accuracy setting the pruning rate of 61%, which is the result of the removal of certain layers.
Conversely, in Mao et al.
(2017) , the imbalanced pruning results is prevented by setting the pruning rate of each layer.
But this requires workers to have good prior knowledge and it takes a lot of time to perform repeated experiments.
In this work, we made efforts to improve the robustness of the global ﬁlter pruning algorithm.
We proposed a cumulative saliency based Globally Soft Filter Pruning (GSFP) scheme, which globally prunes the unsalient ﬁlters and has better robustness.
The contributions of this work are as follows:
• we experimentally compared the effects of four ﬁlter pruning strategies (global and local, hard and soft) on the results.
And we found that combining global pruning with soft pruning strategy will have better results.
• we proposed a reasonable normalized saliency formula to improve the robustness of global pruning method.
• By accumulating saliency score across all batches, we obtain more accurate pruning guidance, as we restore the model capacity during retraining process. [Table] [Figure]
The ﬂowchar of proposed GSFP scheme is shown in Figure 1.
In particular, we ﬁrst got the saliency score of each ﬁlter through the saliency formula and normalized the saliency score of each ﬁlter by dividing the average saliency score of each layer.
Then, accumulate the scores through the rebust saliency formula during each training iteration. After obtaining the cumulative saliency score, we sort the scores across all layers, and generate masks based on the prune rate to multiply the corresponding ﬁlters to complete the pruning operation. Finally, we iteratively tune the sparse network and dynamically update the ﬁlter saliency in a top-down manner.  MOTIVATION
In section
[Table]
we explain that the current ﬁlter pruning algorithm can be divided into “Soft” and “Hard”, or “Local” and “Global”.
In order to spy on the impact of different pruning strategies on the results
[Table]
we compared the combined effects of the above four pruning strategies.
As shown in Table
[Table]
We test in extreme sparsity levels (Pruned FLOPS 61% on VGG-16 in CIFAR-10).
“L” refers to “Local”, “G” refers to “Global”, “H” refers to “Hard”, “S” refers to “Soft”, “F” refers to “Filter” and “P” refers to “Pruning”.
Therefore, “LHFP” means Locally Hard Filter Pruning method, and so on.
The GHFP 1 method and the GHFP 2 method have the difference in the saliency score formula
[Table]
and
GHFP 2 uses a normalized score formula to achieve better robustness.
This also applies to GSFPand GSFP 2 .
GSFP 3 uses cumulative normalized saliency formula to achieve more accurate pruning guidance compared to GSFP
[Table] [Table]
[Figure]
The comparison results are shown in Table 1 and Figure 2(a).
We have found:
• soft pruning strategy is better than hard, because the soft method has enough time to recover model capacity after pruning.
This is
clearly
shown in Figure 2(a).
Correspondingly, soft method also takes longer training time than hard.
• global pruning has worse robustness.
As shown in Table 1, the accuracy drops rapidly under extreme sparsity on GHFP 1 and GSFP 1 compared with LHFP and LSFP.
Because the ﬁlters of entire layer is comletely cut off.
One of the reasons is that the size of the ﬁlter is not considered in the saliency formula.
Though certain normalization methods, the robustness of global pruning can be improved.
• when the algorithm has better robustness, the global pruning can reach higher accuracy than the local.
This can be obtained by comparing GHFP 2 and LHFP, GSFP 2 and LSFP.
In addition, we also found that cumulative saliency has more accurate pruning guidance.
By accumulating saliency score across batches, GSFP 3 has achieved the best performance.
METHOD 3. P RELIMINARIES
We will formally introduce the symbol and annotations in this section.
Consider a set of training examples D
= {X = { X 1 , X 2 , · · · , X N } , Y =
{ Y 1 , Y 2 , · · · , Y N }} , where X i and Y i represent an input and a target output, respectively.
We denote ﬁlters of the entire network as
W = (cid:110) W 11 , W 21 , · · · , W N L L (cid:111) ,
where (cid:8) W nl ∈ R C
l × K × K , 1 ≤ l ≤ L, 1 ≤ n ≤ N l (cid:9) .
C l denotes the number of channels for the l -th convolution layer; L denotes the number of layers; N l denotes the number of ﬁlters for the l -th convolution layer.
During the training, the gradient value of the parameter is deﬁned as G =
(cid:110)
G 11 , G 21 , · · · , G N L L (cid:111) .
The shapes of input tensor U and output tensor V are N l × H l × W l and C l +
×
H l +1 × W l +1 , respectively.
The saliency score corresponding to each ﬁlter as S =
(cid:110)
s 11 , s 21 , · · · , s N L L (cid:111) .
By sorting the calculated saliency scores, we can get the mask of each ﬁlter according to the global pruning rate P .
Noted that mask M = (cid:110) M 11 , M 21 , · · · , M N L L (cid:111) has two forms corresponding to ﬁlter mask and channel mask in Figure 1.
The parameters after pruning are deﬁned as W ∗ =
(cid:110) W 1 ∗ 1 , W 2 ∗ 1 , · · · , W N L ∗ L (cid:111) .
In the pruning process, we hope to replace the original network parameters with more sparse and regular parameters to achieve a trade-off between reducing the amount of calculation and not losing accuracy as L ( D|W ∗ ) =
L
[Equation]
D|W · M ) ≈
L ( D|W ) , where L ( · ) is a loss function for the pruned network.
The pruning problem translates into the following optimization problem:
[Equation]
where (cid:107)M(cid:107) 0 is the number of ﬁlter pruning form(all M internal elements are zero) in M .
In response to these problems, the key is to ﬁnd a suitable mask.
Therefore, we need to develop a set of rules to evaluate the importance of each ﬁlter.
3. G LOBALLY S OFT F ILTER P RUNING (GSFP)
Most of previous works(Li et al.
(2017);Luo et al.
; Lin et al.
(2018)) pruned the ﬁlter every training iteration.
This pruning strategy leads to a high rate of mis-pruning, and the model after pruning recovery time is short so that it will reduce the capacity.
In addition, the constructed saliency formula either does not take into account global redundancy or ignores normalization of each layer.
This work proposed a more reasonable saliency formula to make the pruning strategy more robust and improved the accuracy of pruning by accumulating saliency scores.
The details of GSFP is illustratively explained in Figure 1, which can be divided into the following three steps.
Rebust Saliency Formula.
Some works such as Li et al.
(2017) deﬁned the saliency formula as the sum of the absolute values of kernel weights.
He et al.
(2018) used the l 2 -norm to evaluate the importance of each ﬁlter.
Molchanov et al.
(2017) and Lin et al.
(2018) proposed that not only the weight but also the gradient value affecting the loss function, which is derived by Taylor expansion.
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
However when the global pruning strategy is adopted, the pruning result will be uneven if the weights between the different layers are very different.
Obviously, the extremely uneven pruning result is an important cause of poor pruning robustness.
We all know that ﬁlter sizes are often different in different convolution layers.
However, the size of the ﬁlter is not considered in the formula which is one of the reasons for the uneven pruning results.
So we have modiﬁed the l 2 -norm formula as follows:
[Equation]
In order to highlight the saliency of the ﬁlter for different input data, we use the gradient value in back propagation as the multiplier factor.
The speciﬁc deﬁnition is as follows:
[Equation]
Different convolutional layers have different parameter distributions, and they also accompany vanishing or exploding gradient problem during training.
Taking into account these two phenomena, we normalize the gradient factor and the weighting factor separately, and ﬁnally construct a complete saliency formula as follows:
[Equation]
Cumulative Saliency.
In order to improve the accuracy of pruning, we accumulate the saliency scores of the ﬁlters in each batch during the training process.
During the training stage, the saliency score is calculated while restoring the capacity of the model.
The saliency score continues to accumulate in each iteration until the entire data set is traversed and then prune based on the saliency score, as shown in Figure 1.
We deﬁne the cumulative signiﬁcance score as C = (cid:110) c 11 , c 21 , · · · , c N L L (cid:111) .
The process of saliency accumulation is described as:
[Equation]
where I represents the number of iterations within a training epoch.
This takes into account the saliency of the ﬁlter throughout the data set and provides more accurate guidance for subsequent pruning processing.
Filter Pruning.
In the ﬁlter pruning step, we use a global pruning strategy to fully consider the redundancy of each layer.
We only need to set a global pruning rate P .
The global saliency scores of all ﬁlters Indx is constructed, which are sorted by a descending order, i.e., Indx = sort ( (cid:110)
c
11 , c 21 , · · · , c N L L (cid:111) ) .
Then set the top- P (cid:80) Ll
=1 N l Indx corresponding ﬁlters to zero.
Note that if we set the value of selected N l P l ﬁlters to zero, the channel corresponding to the next layer of ﬁlters will be set zero simultaneously, i.e., (cid:8) W nl +1 ∈ R N l × (1 −P l ) × K × K , 1 ≤ n ≤ N l
+1 (cid:9) .
EXPERIMENTS
To evaluate the effectiveness of the GSFP scheme, CNN model pruning were performed on three databases MNIST, CIFAR-10, and ImageNet.
We implemented our ﬁlter pruning method in the
PyTorch framework.
To get the baseline accuracies for each network, we train each model from scratch on MNIST and CIFAR-10.
For the ImageNet dataset, pre-trained model from torchvision were used as the baseline model.
The same data argumentation strategies were used with PyTorch ofﬁcial examples (Paszke et al.
For retraining of ﬁlter pruning, we use a constant learning rate 0 . 01 and retrained 100 epochs on MNIST.
In CIFAR-10, we set the initial learning rate to 0 .
01 , multiply by 0 .
1 per 50 epoch, and retrained 150 epochs.
Finally, we retrained 100 epochs on
ImageNet.
The ﬁrst 70 epochs learning rates are set to 0 .
001 , and the last 30 epochs settings are 0 .
4. L E N ET ON MNIST
MNIST is a well-known database of handwritten digits which contains about 60 , 000 images for training and 10 , 000 for testing.
We perform experiments on a version of LeNet proposed in (LeCun et al.
(1998)), which consists of two convolutional and two fully-connected layers.
When applying
GSFP to LeNet, we ﬁrst need a pre-trained model and then pruning ﬁlter based on saliency score for each training epoch.
Table 2 summarizes the remained ﬁlters and channels, ﬂoating-point operations (FLOPs), and classiﬁcation accuracy.
Compared with the SSL method (Wen et al. (2016)), GSFP can prune more ﬁlters and channels resulting in a number of 3% reduction on computation complexity on average, while maintaining the same accuracy.
[Table]
[Figure]
To further analyze the signiﬁcance of the saliency score for pruning
[Figure]
we visualized the saliency scores for the four pruned models with different pruning rates in Figure 3.
In the ﬁgure, the 17th and 29th ﬁlters are critical ﬁlter.
We can see that as the pruning rate increases, the saliency of the critical ﬁlter becomes more and more prominent during the process of model retraining.
When we set the pruning rate to 70% , 80% , 85% and 90% , respectively, the corresponding convolution kernel of CONV1 layer remained is 7 , 6 , 5 , 3 in Table 2.
In Figure 3, we can clearly gauge the importance of each ﬁlter across the network.
When the pruning rate is set to 90% , CONV1 can only retainﬁlters after global measurement, but as can be seen from the ﬁgure 3 , there are 5 ﬁlters that are obviously saliency, we need drop two important ﬁlters, so the classiﬁcation accuracy drop about 0 .
Therefore, visual saliency scores can help us ﬁnd the most appropriate pruning rate.
[Table]
4. RESULT ON CIFAR-
For CIFAR-10 dataset, we have tested our GSFP on single-branch networks(VGG-16), multi-branch networks(ResNet-32 and ResNet-110) and compact networks(MobileNet and MobileNet-V2).
We use several difference pruning rates to select the best trade-off between model computation and classiﬁcation accuracy.
Table 4 summaries the pruning results and Figure 2(b) shows the training curve for different networks.
For example, in the case where the same classiﬁcation without loss of accuracy for single-branch networks VGG-16 (Simonyan & Zisserman (2014)), (Li et al.
pruning method reduced FLOPs by 34 .
2% , while GSFP reduced FLOPs by 61 .
The proposed scheme have achieved 2 × better results.
Note that VGG-16 on CIFAR-10 consists of 13 convolutional layers adding Batch Normalization and just 1 fully connected layer, in which the fully connected layers do not occupy large portions of parameters due to the small input size and less hidden units.
For single-branch networks we adopted a strategy of combining ﬁlter pruning with channel pruning.
Because the previous layer of ﬁlter pruning will affect the next layer of convolution ﬁlter channels.
Detailed pruning results are shown in Table 3.
Compared with the pruning results listed in Li et al.
(2017), the pruning results of the GSFP are more balanced in each layer, which is why the GSFP can pruning more, similar to the buckets effect.
For multi-branch networks such as ResNet-32 and ResNet-110 models
(He et al.
on CIFAR10, we need to consider the shortcut connection structure in the process of pruning.
Since there is a cross-layer connection for each residual block of ResNet, we only perform ﬁlter pruning for the ﬁrst layer convolution and the shortcut convolution layer of each block, and other layer ﬁlter pruning combined with channel pruning.
The proposed method reduced FLOPs by 50% on ResNet32 without losing the accuracy of the pre-trained model.
With the same recognition accuracy, the proposed sheme achieved 7% more pruning rates compared to He et al.
For the ResNet110, our scheme has a advantage of 12% on pruning rates compared with the best result of the state-of-the-art.
Moreover
[Table]
we have also pruned two compact networks MobileNet (Howard et al.
and MobileNetV2 (Sandler et al.
Compact network usually uses group convolution unit to reduce the amount of calculations and parameters, so we only conducted ﬁlter pruning for these networks.
After pruning
[Table]
it is found that MobileNet and MobileNetV2 have a lot of redundancy on the CIFAR10 dataset, the amount of calculation pruning can reach 72 .
57% and 69 .
68% , respectively
[Table] [Table]
[Figure]
4. R ES N ET ON I MAGE N ET
We also test our pruning scheme on the large scale ImageNet classiﬁcation task.
ResNet is currently the most widely used network structure, so in the experiment we chose ResNet-18 and ResNetfor pruning.
During the pruning process, since the ResNet cross-layer contains a lot of information, we do not pruned the shortcut convolution layer.
Table 5 shows the pruning results of ResNet on
Imagenet.
For ResNet-18, GSFP reduced the FLOPs by 47 .
06% , with a Top-1 accuracy drop 2 .
For ResNet-34, under almost the same loss of precision, the GSFP was reduced by 3.75% compared to the He et al.
In addition, when the pruning was further increased, the calculation amount was reduced by 9.6%, while the Top1 precision loss was only decreased by 0.5%.
Compared to
Dong et al.
(2017) , Li et al.
(2017) and He et al.
(2018), our proposed GSFP has a larger pruning space.
By analyzing the pruning results of convolution layers at different time points in Figure 4, we can see that the strategy of globally soft pruning can automatically adjust the pruning result and improve the robustness of the pruning process.
[Figure]
[Table]
CONCLUSIONS
In this work
[Table]
we propose a scheme GSFP on ﬁlter pruning for convolutional neural networks, which combines global pruning strategy and soft pruning strategy.
First, we designed the experiment and compared the previous work with four different pruning strategies (global and local, hard and soft).
In addition, it is proved by experiments that the global pruning strategy will reduce the pruning robustness and can be improved by a reasonable normalization formula.
Finally, the pruning accuracy can be effectively improved by accumulating the saliency of the ﬁlter across all batches in the model recovery training process after pruning.