A DAPTIVE A CTIVATION - BASED S TRUCTURED P RUNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Pruning is a promising approach to compress complex deep learning models in order to deploy them on resource-constrained edge devices.
However, many existing pruning solutions are based on unstructured pruning, which yield models that cannot efﬁciently run on commodity hardware, and require users to manually explore and tune the pruning process, which is time consuming and often leads to sub-optimal results.
To address these limitations, this paper presents an adaptive, activation-based, structured pruning approach to automatically and efﬁciently generate small, accurate, and hardware-efﬁcient models that meet user requirements.
First, it proposes iterative structured pruning using activation-based attention feature maps to effectively identify and prune unimportant ﬁlters.
Then, it proposes adaptive pruning policies for automatically meeting the pruning objectives of accuracy-critical, memory-constrained, and latency-sensitive tasks.
A comprehensive evaluation shows that the proposed method can substantially outperform the state-of-the-art structured pruning works on CIFAR-10 and ImageNet datasets.
For example, on ResNet-56 with CIFAR-10, without any accuracy drop, our method achieves the largest parameter reduction (79.11%), outperforming the related works by 22.81% to 66.07%, and the largest FLOPs reduction (70.13%), outperforming the related works by 14.13% to 26.53%.
I NTRODUCTION
Deep neural networks (DNNs) have substantial compute and memory requirements.
As deep learning becomes pervasive and moves towards edge devices, DNN deployment becomes harder because of the mistmatch between resource-hungry DNNs and resource-constrained edge devices.
DNN pruning is a promising approach (Li et al.
; Han et al.
; Molchanov et al.
; Theis et al.
; Renda et al.
(2020)), which identiﬁes the parameters (or weight elements) that do not contribute signiﬁcantly to the accuracy and prunes them from the network.
Recently, works based on the Lottery Ticket Hypothesis (LTH) have achieved great successes in creating smaller and more accurate models through iterative pruning with rewinding (Frankle & Carbin (2018)). However, LTH has only been shown to work successfully with unstructured pruning which, unfortunately leads to models with low sparsity and difﬁcult to accelerate on commodity hardware such as CPUs and GPUs (e.g., Hill et al. (2017) shows directly applying NVDIA cuSPARSE on unstructured pruned models can lead to 60 ˆ slowdown on GPU compared to dense kernels.)
Moreover, most pruning methods require users to explore and adjust multiple hyper-parameters, e.g., with LTH-based iterative pruning, users need to determine how many parameters to prune in each round.
Tuning the pruning process is time consuming and often leads to sub-optimal results.
We propose activation-based , adaptive , iterative structured pruning to ﬁnd the “winning ticket” models that are at the same time hardware efﬁcient and to automatically meet the users’ model accuracy, size, and speed requirements.
First, we propose an activation-based structured pruning method to identify and remove unimportant ﬁlters in an LTH-based iterative pruning (with rewinding) process.
Speciﬁcally, we properly deﬁne an attention mapping function that takes a 2D activation feature maps of a ﬁlter as input, and outputs a 1D value used to indicate the importance of the ﬁlter.
This approach is more effective than weight-value based ﬁlter pruning because activationbased attention values not only capture the features of inputs but also contain the information of convolution layers that act as feature detectors for prediction tasks.
We then integrate this attention- based method into the LTH-based iterative pruning framework to prune the ﬁlters in each round and ﬁnd the winning ticket that is small, accurate, and hardware-efﬁcient.
Second, we propose adaptive pruning that automatically optimizes the pruning process according to different user objectives.
For latency-sensitive scenarios like interactive virtual assistants, we propose FLOPs-guaranteed pruning to achieve the best accuracy given the maximum amount of compute FLOPs; For memory-limited environments like embedded systems, we propose modelsize-guaranteed pruning to achieve the best accuracy given the maximum amount of memory footprint; For accuracy-critical applications such as those on self-driving cars, we propose accuracyguaranteed pruning to create the most resource-efﬁcient model given the acceptable accuracy loss.
Aiming for different targets, our method adaptively controls the pruning aggressiveness by adjusting the global threshold used to prune ﬁlters.
Moreover, it considers the difference in each layer’s contributions to the model’s size and computational complexity and uses a per-layer threshold, calculated by dividing each layer’s remaining parameters or FLOPs by the entire model’s remaining parameters or FLOPs, to prune each layer with differentiated level of aggressiveness.
Our results outperform the related works signiﬁcantly in all cases targeting accuracy loss, parameters reduction, and FLOPs reduction.
For example, on ResNet-56 with CIFAR-10 dataset, without accuracy drop, our method achieves the largest parameter reduction (79.11%), outperforming the related works by 22.81% to 66.07%, and the largest FLOPs reduction (70.13%), outperforming the related works by 14.13% to 26.53%.
In addition, our method enables a pruned model the reach 0.6% or 1.08% higher accuracy than the original model but with only 30% or 50%of the original’s parameters. On ResNet-50 on ImageNet, for the same level of parameters and FLOPs reduction, our method achieves the smallest accuracy loss, lower than the related works by 0.08% to 3.21%; and for the same level of accuracy loss, our method reduces signiﬁcantly more parameters (6.45% to 29.61% higher than related works) and more FLOPs (0.82% to 17.2% higher than related works).  B ACKGROUND AND R ELATED W ORKS Unstructured vs. Structured Pruning.
Unstructured pruning (LeCun et al.
Han et al.
; Molchanov et al.
(2017)) is a ﬁne-grained approach that prunes individual unimportant elements in weight tensors.
It has less impact to model accuracy, compared to structured pruning, but unstructured pruned models are hard to accelerate on commodity hardware.
Structured pruning is a coarse-grained approach that prunes entire regular regions of weight tensors according to some rule-based heuristics, such as L1-norm (Li et al.
(2016)), average percentage of zero (Molchanov et al.
(2016)), and other information considering the relationship between neighboring layers (Theis et al.
; Lee et al.
Compared to unstructured pruning, it is more difﬁcult to prune a model without causing accuracy loss using structured pruning, because by removing entire regions, it might remove weight elements that are important to the ﬁnal accuracy (Li et al.
However, structured pruned models can be mapped easily to general-purpose hardware and accelerated directly with off-the-shelf hardware and libraries (He et al.
(2018b)).
One Shot vs. Iterative Pruning.
One-shot pruning prunes a pre-trained model and then retrains it once, whereas iterative pruning prunes and retrains the model in multiple rounds.
Both techniques can choose either structured or unstructured pruning techniques.
Recently, works based on the Lottery Ticket Hypothesis (LTH) have achieved great successes in creating smaller and more accurate models through iterative pruning with rewinding (Frankle & Carbin (2018)).
LTH posits that a dense randomly initialized network has a sub-network, termed as a winning ticket , which can achieve an accuracy comparable to the original network.
At the beginning of each pruning round, it rewinds the weights and/or learning rate of the sub-network to some early epoch of the training phase of the original model to reduce the distance between the sub-network and original model and increase the change of ﬁnding the winning ticket.
However, most of LTH-based works considered only unstructured pruning, e.g., Iterative Magnitude Pruning (IMP) (Frankle & Carbin (2018); Frankle et al.
(2019)), which, as discussed above, is hardware-inefﬁcient.
It is non-trivial to design an iterative pruning method with structured pruning.
To understand the state of iterative structured pruning, we experimented with IMP’s structured pruning counterpart—L1norm structured pruning (ILP) (Li et al.
(2016)) which removes entire ﬁlters depending on their L1norm value.
We observed that ILP cannot effectively prune a model while maintaining its accuracy, e.g., ILP can prune ResNet-50 by at most 11.5% of parameters when the maximum accuracy loss ´ q [Figure] k
[Figure]
k
[Figure] n [Figure] [Figure]
1i k
[Figure]
k i n i-
[Figure]
i k
i k
i n i-
[Figure]
X i w i-
[Figure]
h i-
[Figure]
n i
[Figure]
[Figure]
[Figure]
is limited to 1% on ImageNet.
Therefore, directly applying iterative pruning with existing weightmagnitude based structured pruning methods does not produce accurate pruned models.
In this paper, we study how to produce small , accurate , and hardware-efﬁcient models based on iterative structured pruning with rewinding .
Automatic Pruning.
For pruning to be useful in practice, it is important to automatically meet the pruning objectives for diverse ML applications and devices.
Most pruning methods require users to explore and adjust multiple hyper-parameters, e.g., with LTH-based iterative pruning, users need to determine how many parameters to prune in each round.
Tuning the pruning process is time consuming and often leads to sub-optimal results.
Therefore, we study how to automatically adapt the pruning process in order to meet the pruning objectives without user intervention .
Some works (Zoph et al.
; Cai et al.;
Ashok et al.
use reinforcement-learning algorithms to ﬁnd computationally efﬁcient architectures.
These works are in the area of Neural Architecture Search (NAS), among which the most recent is AutoML for Model Compression (AMC)
(He et al.
(2018b)).
AMC enables the model to arrive at the target speedup by limiting the action space (the sparsity ratio for each layer), and it ﬁnds out the limit of compression that offers no loss of accuracy by tweaking the reward function.
But this method has to explore over a large search space of all available layer-wise sparsity, which is time consuming when neural networks are large and datasets are complicated.
M ETHODOLOGY
Algorithm 1 illustrates the overall ﬂow of the proposed adaptive activation-based structured pruning.
To represent pruning of weights, we use a mask M r (cid:15) t 0 , 1 u d for each weight tensor W rt
(cid:15)R d , where r is the pruning round number and t is the training epoch.
Therefore, the pruned network at the end of training epoch T is represented by the element-wise product M r d W rT .
The ﬁrst three steps are to train the original model to completion, while saving the weights at Epoch k .
Steps 4–10 represent a pruning round.
Step 4 prunes the model (discussed in Section 3.1).
Step 5 (optional) and 6 perform rewinding.
Step 7 retrains the pruned model for the remaining T ´ k epochs.
Step 8 evaluates the pruned model according to the pruning target.
If the target is not met, Step 9 resets the weights to an earlier round.
Step 10 calculates the pruning threshold for the next pruning round following the adaptive pruning policy (discussed in Section 3.2).
3. A CTIVATION - BASED F ILTER P RUNING
To realize iterative structured pruning, one can start with the state-of-the-art structured pruning methods and apply it iteratively.
The widely used structured pruning method—L1-norm based structured pruning Renda et al.
(2020) removes ﬁlters with the lowest L1-norm values, whereas the most recent method—Polarization-based structured pruning Zhuang et al.
(2020) improves it by using a regularizer on scaling factors of ﬁlters and pruning ﬁlters whose scaling factors are below than a threshold.
These two methods both assume that the weight values of a ﬁlter can be used as an indicator about the importance of that ﬁlter, much like how LTH uses weight values in unstructured pruning.
However, we observe that weight-based structured pruning methods cannot produce accurate pruned models.
For example, to prune ResNet-56 on CIFAR-10 with no loss in top-1 accuracy, L1-norm based structured pruning can achieve at most only 1.15 ˆ model compression, and Polarization-norm based structured pruning can achieve at most only 1.89 ˆ inference speedup.
The reason is that, some ﬁlters, even though their weight values are small, can still produce useful non-zero activation values that are important for learning features during backpropagation.
That is, ﬁlters with small values may have large activations.
We propose that the activation values of ﬁlters are more effective in ﬁnding unimportant ﬁlters to prune.
Activations like ReLu enable non-linear operations, and enable convolutional layers to act as feature detectors.
If an activation value is small, then its corresponding feature detector is not important for prediction tasks.
So activation values, i.e., the intermediate output tensors after the non-linear activation, not only detect features of training dataset, but also contain the information of convolution layers that act as feature detectors for prediction tasks.
We present a visual motivation in Figure 1(a).
The ﬁgure shows the activation output of 16 ﬁlters of a convolution layer on one input image.
The ﬁrst image on the left is the original image, and the second image is the input features after data augmentation.
We observe that some ﬁlters extract image features with high activation patterns, e.g., the 6 th and 12 th ﬁlters.
In comparison, the activation outputs of some ﬁlters are close to zero, such as the 2 nd, 14 th, and
16 th.
Therefore, from visual inspection, removing ﬁlters with weak activation patterns is likely to have low impact on the ﬁnal accuracy of the pruned model.
There is a natural connection between our activation-based pruning approach and the related attention-based knowledge transfer works (Zagoruyko & Komodakis (2016)).
Attention is a mapping function used to calculate the statistics of each element in the activations over the channel dimension.
By minimizing the difference of the activation-based attention maps from intermediate layers between a teacher model and a student model, attention transfer enables the student to imitate the behaviour of the teacher.
Our proposed pruning method builds upon the same theory that activation-based attention is a good indicator of ﬁlters regarding their ability to capture features, and it addresses the new challenges in using activations to guide automatic structured pruning.
In the following, we describe how to prune ﬁlters based on its activation feature maps in each round of the iterative structured pruning process.
Figure 1(b) shows the inputs and outputs of a 2D convolution layer (referred to as conv2d), followed by an activation layer.
For the i th conv2d layer, let X i (cid:15)R n i ´
ˆ h
i ´ 1 ˆ w
i ´ 1 denote the input features, and F ij (cid:15)R n i ´
ˆ k
i ˆ
k
i be the j th ﬁlter, where h i ´
and w i ´ 1 are the height and width of the input features, respectively, n i ´ 1 is the number of input channels, n i is the number of output channels, and k i is the kernel size of the ﬁlter.
The activation of the j th ﬁlter F i j after ReLu mapping is therefore denoted by A i j (cid:15)R h i ˆ w i .
The attention mapping function takes a 2D activation A ij (cid:15)R h
i ˆ w
i of ﬁlter F ij as input, and outputs a 1D value which will be used as an indicator of the importance of ﬁlters.
We consider three forms of activation-based attention maps, where p ě 1 and a ik,l denotes every element of A ij : 1) Mean of attention values raised to the power of p : F mean p A i j q “ 1 h i ˆ w
i
[Equation]
of
[Equation]
tion values raised to the power of p : F max p
A ij q “ max l “ 1 ,h i ˆ w i ˇˇˇ a ik,l ˇˇˇ p ; and 3) Sum of attention values raised to the power of p : F sum p
A ij q “ ř h
i k “ 1 ř w i l “ 1 ˇˇˇ a ik,l ˇˇˇ p .
We choose F mean p A ij q with p equals to 1 as the indicator to identify and prune unimportant ﬁlters, and our method removes the ﬁlters whose attention value is lower than the pruning threshold.
See Section 5 for an ablation study on these choices.
3. A DAPTIVE I TERATIVE P RUNING
Our approach to pruning is to automatically and efﬁciently generate a pruned model that meets the users’ different objectives.
Automatic pruning means that users do not have to ﬁgure out how to conﬁgure the pruning process.
Efﬁcient pruning means that the pruning process should produce the user-desired model as quickly as possible.
Users’ pruning objectives can vary depending on the usage scenarios: 1) Accuracy-critical tasks, like those used by self-driving cars have stringent accuracy requirement, which is critical for safety, but do not have strict limits on their computing and storage usages; 2) Memory-constrained tasks, like those deployed on microcontrollers have very limited available memory to store the models but do not have strict accuracy requirement; and 3) Latency-sensitive tasks, like those employed by virtual assistants where timely responses are desirable but accuracy is not a hard constraint.
In order to achieve automatic and efﬁcient structured pruning, we propose three adaptive pruning policies to provide 1) Accuracy-guaranteed Adaptive Pruning which produces the most resourceefﬁcient model with the acceptable accuracy loss; 2) Memory-constrained Adaptive Pruning which generates the most accurate model within a given memory footprint; and 3) FLOPs-constrained
Adaptive Pruning which creates the most accurate model within a given computational intensity.
Speciﬁcally, our proposed adaptive pruning method automatically adjusts the global threshold ( T ), used in our iterative structured pruning algorithm (Algorithm 1) to quickly ﬁnd the model that meets the pruning objective.
Other objectives (e.g., limiting a model’s energy consumption) can also be supported by simply plugging in the relevant metrics into our adaptive pruning algorithm.
We take
Accuracy-guaranteed Adaptive Pruning, described in Algorithm 2, as an example to show the procedure of adaptive pruning.
Memory-constrained and FLOPs-constrained Adaptive Pruning algorithms are included in Appendix A.3 and A.4, respectively.
1: for the layer
i in a pruning round r do 2:
Calculate the number of parameters of unpruned ﬁlters of the current layer and the whole model, respectively: N i r r s “ n p
i ´ 1 q r r sˆ k
i r r
sˆ k
i r r sˆ n i r r s , N Total r r s “ ř i “ M i “ 1 n p
i ´ 1 q r r sˆ k
i r r
sˆ k
i r r sˆ n i r r s , where M is the number of convolution layers, and for other notations refer to Section 3.1.
Calculate the threshold T
i r r s of the current layer: T i r r s “ T r r s ˆ
N i
r r s N Total r r s .
For each ﬁlter F ij in F i , calculate its attention: F mean p A ij q “ 1 h i ˆ
w i
ř h
i k “ 1 ř w i l “ 1 ˇˇ a ik,l ˇˇ p .
Prune the ﬁlter if its attention is not larger than threshold T i r r s , i.e., set M ij “ 0 if F mean p A ij q ď T
i r r s ;
Otherwise, set M ij “6: end for
In Algorithm 2, the objective is to guarantee the accuracy loss while minimizing the model size (the algorithm for minimizing the model FLOPs is similar).
In the algorithm, T controls the aggressiveness of pruning, and λ determines the increment of T at each pruning round.
Pruning starts conservatively, with T initialized to 0 , so that only completely useless ﬁlters that cannot capture any features are pruned. After each round of pruning, if the model accuracy loss is below than the target accuracy loss, it is considered “acceptable”, and the algorithm increases the aggressiveness of pruning by incrementing T by λ . As pruning becomes increasingly aggressive, the accuracy eventually drops below the target accuracy at a certain round which is considered “unacceptable”. When this happens, our algorithm rolls back the model weights and pruning threshold to the last acceptable round where the accuracy loss is within the target, and restarts the pruning from there but more conservatively—it increases the threshold more slowly by cutting the λ value by half.
If this still does not lead to an acceptable round, the algorithm cuts λ by half again and restarts again.
If after several trials, the accuracy loss is still not acceptable, the algorithm rolls back even further and restarts from an earlier round.
The rationale behind this adaptive algorithm is that the aggressiveness of pruning should accelerate when the model is far from the pruning target, and decelerate when it is close to the target.
Eventually, the changes of model size converges (when the changes of the number of parameters is within 0.1% for several rounds) and the algorithm terminates.
3. L AYER - AWARE T HRESHOLD A DJUSTMENT
While adapting the global pruning threshold using the above discussed policies, our pruning method further considers the difference in each layer’s contribution to model size and complexity and use differentiated layer-speciﬁc thresholds to prune the layers.
As illustrated in Figure 1(b), in terms of the contribution to model size, the number of parameters of layer i can be estimated as N i “ n p i ´
1 q ˆ k
i ˆ k i ˆ n i ; in terms of the contribution to computational complexity, the number of FLOPs of layer i can be estimated as 2 ˆ h
i ˆ
w i ˆ N i .
A layer that contributes more to the model’s size or FLOPs is more likely to have redundant ﬁlters that can be pruned without affecting the model’s accuracy.
Therefore, to effectively prune a model while maintaining its accuracy, we need to treat each layer differently at each round of the iterative pruning process based on its current contributions to the model size and complexity.
Speciﬁcally, our adaptive pruning method calculates a weight for each layer based on its contribution and then uses this weight to adjust the current global threshold and derive a local threshold for the layer.
If the goal is to reduce model size, the weight is calculated as each layer’s number of parameters divided by the model’s total number of parameters; if the goal is to reduce model computational complexity, the weight is calculated as each layer’s
FLOPs divided by the model’s total FLOPs.
These layer-speciﬁc thresholds are then used to prune the layers in the current pruning round.
Assuming the goal is to reduce model size, the procedure in a pruning round r for the i th conv2d layer is shown in Algorithm 3 (the algorithm for reducing model FLOPs is similar).
For pruning, we introduce a mask M ij (cid:15) t 0 , {+1 u n i ´ 
 ˆ k i ˆ+}
k i for each ﬁlter F ij . The effective weights of the convolution layer after pruning is the element-wise product of the ﬁlter and mask— F ij d M ij .
For a ﬁlter, all values of this mask is either 0 or 1, thus either keeping or pruning the entire ﬁlter.
[Table]
E VALUATION 4. CIFAR-10 R ESULTS
Table 1 lists the results from CIFAR-10 [-(Krizhevsky et al. (2009)); and 2) ResNet-
 (He et al.-]on ResNet-56, ResNet-50, VGG-16, VGG-
[Table]
and MobileNetV2.
We compare our method with the state-of-the-art works, including SCOP (Tang et al.
, HRank (Lin et al.
, ILP (Renda et al.
(2020)), PPR (Zhuang et al.
, RFR (He et al. (2017)), GAL (Lin et al.
, DCP (Zhuang et al.
, GBN (You et al.
CP (He et al. (2017)), SFP (He et al.
(2018a))
, FPGM (He et al.
, DeepHoyer ( DH ) (Yang et al.
, AMC (He et al. (2018b)), PFS (Wang et al. (2020)), SCP (Kang & Han (2020)),
VCNNP (Zhao et al.
, Hinge (Li et al.
, GDP (Guo et al.
, Eigendamage (Wang et al. , NN Slimming (Liu et al. , MDP (Guo et al. (2020)), and DMC (Gao et al.
Note that a negative accuracy drop means that the pruned model achieves better accu- [Table] racy than its unpruned baseline model.
“Ours-1” and “Ours-2” denote the cases where the pruning objective is set to minimize model size and FLOPs, respectively, while meeting the accuracy target.
In all cases targeting accuracy, model size [Table] and compute intensity
[Table] our method signiﬁcantly outperforms the recent related works. For example, on ResNet-56, without accuracy drop, our method achieves the largest parameter reduction (79.11%), outperforming the related works by 22.81% to 66.07%, and the largest FLOPs reduction (70.13%), outperforming the related works by 14.13% to 26.53%.
With 70% of parameter reduction, our method achieves the smallest accuracy loss (-0.6%), outperforming the related works by 0.59% to 2.98%.
Also, with 75% of FLOPs reduction, our method achieves the smallest accuracy loss (0.3%), outperforming the related works by 0.87% to 2.24%.
Note that the proposed method also produces a pruned model that reaches 0.6% or 1.08% higher accuracy than the original model but with only 30% or 50%, respectively, of the original’s parameters.
Such small and accurate models are useful for many real-world applications.
On VGG-16, without accuracy drop, our method achieves the largest parameter reduction (72.85%), outperforming the related works by 6.62% to 33.78%; With 70% of parameter reduction, the accuracy loss achieved by our method is lower than GDP by 0.19%.
On VGG-19, with no accuracy loss, our method achieves the largest parameter reduction (85.99%) which outperforms the related works by 5.92% to 7.81%, and the largest FLOPs reduction (56.22%) which outperforms the related works by 13.57% to 19.09%; With 85% of FLOPs reduction, the accuracy loss achieved by our method is lower than EigenDamage by 0.07%.
On MobileNet, without accuracy drop, our method achieves the largest parameter reduction (60.16%), outperforming GDP and MDP by 13.94% and 31.45%, respectively; With 40% of parameter reduction, the accuracy loss achieved by our method is lower than SCOP and DMC by 1.04% and 0.54%, respectively.
The results validate that activation-based pruning is more effective than weight magnitude based pruning in ﬁnding unimportant ﬁlters, as discussed in Section 3.1.
For example, on ResNet-56, with 1% of accuracy loss, the amount of parameter reduction achieved by our method is higher than that of ILP (Renda et al.
by 34.51%; and with 75% FLOPs reduction, the accuracy achieved by our method is higher than PPR (Zhuang et al.
(2020)) by 0.87%.
The results conﬁrm that our method can achieve better results that the state-of-the-art automatic pruning methods, AMC (He et al.
(2018b)).
For example, when targeting at about 55% of FLOPs reduction on ResNet-56, our method achieves 1.53% higher accuracy than AMC; and when targeting at about 60% of parameters reduction on ResNet-50, our method achieves 0.84% higher accuracy than AMC.
Our activation-based adaptive pruning can directly target unimportant ﬁlters in the model, whereas AMC has to use reinforcement learning to explore the whole network.
4. I MAGE N ET R ESULTS
Table 2 shows the results of of ResNet-50 on ImageNet dataset and VGG-19 on Tiny-ImageNet dataset, comparing our method with PFB (Liebenwein et al.
, HRank (Lin et al.
[Table]
[Table]
ILP (Renda et al.
, Sparse Structure Selection ( SSS ) (Huang & Wang (2018)),
NN Slimming (Liu et al. (2017)), and Eigendamage (Wang et al.
“Ours-1” and “Ours-2” denote the cases where the pruning objective is set to minimize the number of parameters and FLOPs, respectively, while meeting the accuracy target.
On ImageNet, for the same level of accuracy loss, our method reduces signiﬁcantly more parameters (6.45% to 29.61% higher than the related works) and more FLOPs (0.82% to 17.2% higher than the related works).
For the same level of parameters or FLOPs reduction, our method achieves the smallest accuracy loss, lower than the related works by 0.08% to 3.21%.
On Tiny-ImageNet, for the same level of parameters reduction, our method achieves signiﬁcantly lower accuracy loss than NN
Slimming by 11%; For the same level of accuracy loss, our method achieves higher parameter reduction (72.58% vs 61.87%), and higher FLOPs reduction (78.97% vs 66.21%) than EigenDamage.
A BLATION S TUDY
The effect of attention mapping functions. Table 3 shows the Top-1 accuracy loss of the VGGpruned by one-shot pruning with different types of attention mapping functions on CIFAR-10. The parameter reduction is 57.73%, and the accuracy of the original model is 93.63% (5 trails).
First, we analyze three forms of activation-based attention maps with p equal to 1 (discussed in Section 3.1), noted as Attention Mean ( p “ 1 ),
Attention Sum ( p “ 1 ), Attention Max ( p “ 1 ).
Attention Mean leads to the lowest accuracy loss, lower than Attention Sum and Attention Max by 0.25% and 0.13%, respectively.
Second, we analyze Attention Mean with difference values of p ( p “ 1 , 2 , 4 ).
When p is set to 1 , it leads to the lowest accuracy loss (0.38% vs 0.42% and 0.47%).
Also, all the forms of
Attention Mean outperforms L1-Norm, which validates that the proposed activation-based attention mapping function is more efﬁcient than weight magnitude based methods to evaluate the importance of ﬁlters.
The effect of adaptive pruning.
We compare the proposed adaptive pruning strategy with nonadaptive iterative pruning strategies that prune a ﬁxed percentage of ﬁlters in each pruning round.
One baseline prunes a ﬁxed percentage of ﬁlters with lowest weight values, noted as Iterative L1Norm Pruning (ILP); the other prunes a fxied percentage of ﬁlters using proposed attention mapping function with p equal to 1 , noted as Iterative Attention Pruning (IAP).
Table 4 compares the Topaccuracy loss of the VGG-16 model with sparsity of 70% pruned by IAP, ILP, and the proposed adaptive pruning strategy on CIFAR-10.
For IAP and ILP, the pruning rate is set to 5%, that is, 5% of ﬁlters are pruned in each pruning round.
With 70% reduced parameters, the proposed adaptive pruning strategy leads to a higher accuracy than IAP and ILP by 1.19% and 1.39%, respectively.
C ONCLUSIONS
This paper proposes an adaptive, activation-based, iterative structured pruning approach to automatically and efﬁciently generate small, accurate, and hardware-efﬁcient models that can meet diverse user requirements.
We show that activation-based attention value is a more precise indicator to identify unimportant ﬁlters to prune than the commonly used weight magnitude value.
We also offer an effective way to perform structured pruning in an iterative pruning process, leveraging the
LTH theory to ﬁnd small and accurate sub-networks that are at the same time hardware efﬁcient.
Finally, we argue that automatic pruning is essential for this method to be useful in practice, and proposes an adaptive pruning method that can automatically meet user-speciﬁed objectives in terms of model accuracy, size, and inference speed but without user intervention.
Our results conﬁrm that the proposed method outperforms existing structured pruning approaches with a large margin.
A A PPENDIX A. I MPLEMENTATION D ETAILS
We implemented our proposed method on PyTorch version 1.6.0.
For ResNet models on CIFAR10, the learning rate is set to 0.1 initially and decays with a rate of 0.1 at the epoch 91 and 136.
Weight decay is set to 0.0002.
For ResNet-50 on ImageNet, the learning rate increases to 0.256 in a warmup mechanism during the ﬁrst 5 epochs, and decays with a factor of 0.1 at epochs 30, 60, 80 (Renda et al.
(2020); Frankle et al.
Nesterov SGD optimizer is used with a momentum of 0.9 for all models.
The simple data augmentation (random crop and random horizontal ﬂip) is used for all training images.
Learning rate rewinding is used for iterative pruning, which rewinds to the epoch at roughly 60% of the total training duration (see Appendix A.5 for an analysis of the effect of rewinding epochs on the accuracy of the pruned models).
For adaptive pruning, T and λ are initialized to 0.0 and 0.005, respectively.
A. T HE E FFECT OF A TTENTIONS
Figure 2 shows the attention of each ﬁlter of the ﬁrst convolution layer of ResNet-50 on ImageNet with different values of p (p=1, 2, 4).
The setting where p is equal to 1 tends to be best, since iit promotes the effectiveness of the pruning by enabling the gap between the mean values of the useful ﬁlters and useless ﬁlters to be large.
[Figure]
-50 on ImageNet with different values of p ( p “ 1 , 2 , 4 ).
A. M EMORY - CONSTRAINED A DAPTIVE P RUNING
The Memory-constrained Adaptive Pruning Algorithm is shown in Algorithm 4.
A. FLOP S - CONSTRAINED A DAPTIVE P RUNING
The FLOPs-constrained Adaptive Pruning Algorithm is shown in Algorithm 5.
A. T HE E FFECT OF R EWINDING E POCH
In order to understand how the rewinding impacts accuracy of the pruned models, we analyze stability to pruning as proposed by Frankle et al.
(2019), which is deﬁned as the L2 distance between the masked weights of the pruned network and the original network at the end of training.
We validate the observations proposed by Frankle et al.
that for deep networks, rewinding to very early stages is sub-optimal as the network has not learned considerably by then; and rewinding to very late training stages is also sub-optimal because there is not enough time to retrain.
Speciﬁcally,
Figure 3(a) shows the Top-1 test accuracy of the pruned ResNet-50 with 83.74% remaining parameters when learning rate is rewound to different epochs, and Figure 3(b) shows the stability values at the corresponding rewinding epochs.
We observe that there is a region, 65 to 80 epochs, where [Table] [Table] Terminate 11: else 12:
15: else 16:
Go to Step 15 20: else 21: back the resulting accuracy is high.
We ﬁnd that L2 distance closely follows this pattern, showing large distance for early training epochs and small distance for later training epochs.
Our ﬁndings show that rewinding to 75%-90% of training time leads to good accuracy.
[Figure]
A. T HE E FFECT OF I TERATIVE P RUNING
[Figure]
To illustrate the effectiveness of our proposed adaptive pruning, in Figure we show how the accuracy loss and parameter reduction change over the pruning rounds as the pruning threshold adapts according to Algorithm 2, in the experiment of pruning ResNet-56 on CIFAR-10, given the target of 1% accuracy loss.
From Round 1 to
Round 24, the accuracy loss of the pruned model is lower than the target accuracy loss (1%), so the algorithm increases the pruning aggressiveness gradually by increasing the threshold.
At Round 25, the accuracy loss exceeds the target, so the algorithm rolls back the model weights and the pruning threshold to Round 24, and restart the pruning from there more conservatively.
The above process repeats until after Round 39, the model size converges, and the algorithm terminates at reducing totally 88.23% of parameters.