R ETHINKING THE V ALUE OF N ETWORK P RUNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings.
A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and ﬁne-tuning.
During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy.
In this work, we make several surprising observations which contradict common beliefs.
For all the six state-ofthe-art pruning algorithms we examined, ﬁne-tuning a pruned model only gives comparable or even worse performance than training that model with randomly initialized weights.
For pruning algorithms which assume a predeﬁned target network architecture, one can get rid of the full pipeline and directly train the target network from scratch.
Our observations are consistent for a wide variety of pruning algorithms with multiple network architectures, datasets, and tasks.
Our results have several implications: 1) training a large, over-parameterized model is not necessary to obtain an efﬁcient ﬁnal model, 2) learned “important” weights of the large model are not necessarily useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited “important” weights, is what leads to the efﬁciency beneﬁt in the ﬁnal model, which suggests that some pruning algorithms could be seen as performing network architecture search; we further show that it is possible to use the sparsity patterns from the pruned network to design efﬁcient models.
I NTRODUCTION
Over-parameterization is a widely-recognized property of deep neural networks (Denton et al., 2014;
Ba & Caruana, 2014), which leads to high computational cost and high memory footprint for inference.
As a remedy, network pruning (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015;
Molchanov et al., 2016; Li et al., 2017) has been identiﬁed as an effective technique to improve the efﬁciency of deep networks for applications with limited computational budget.
A typical procedure of network pruning consists of three stages: 1) train a large, over-parameterized model (sometimes there are pretrained models available), 2) prune the trained large model according to a certain criterion, and 3) ﬁne-tune the pruned model to regain the lost performance.
[Figure]
Generally, there are two common beliefs behind this pruning procedure.
First, it is believed that starting with training a large, over-parameterized network is important (Luo et al., 2017; CarreiraPerpinán & Idelbayev, 2018), as it provides a highperformance model (due to stronger representation & optimization power) from which one can safely remove a set of redundant parameters without signiﬁcantly hurting the accuracy.
Therefore, this is usually believed, and reported to be superior to directly training a smaller network from scratch (Li et al., 2017; Luo et al., 2017;
He et al., 2017b;
Yu et al., 2018) – a commonly used baseline approach.
Second, both the pruned architecture and its associated weights are believed to be essential for obtaining the ﬁnal efﬁcient model (Han et al., 2015).
Thus most existing pruning techniques choose to ﬁne-tune a pruned model instead of training it from scratch.
The preserved weights after pruning are usually considered to be critical, as how to accurately select the set of important weights is a very active research topic in the literature (Molchanov et al., 2016; Li et al., 2017; Luo et al., 2017;
He et al., 2017b; Liu et al., 2017; Suau et al., 2018).
In this work, we show that both of the beliefs mentioned above are not necessarily true.
Based on an extensive empirical evaluation of state-of-the-art pruning algorithms on multiple datasets with multiple network architectures, we make two surprising observations.
First, for pruning algorithms with predeﬁned target network architectures (Figure 2), directly training the small target model from random initialization can achieve the same, if not better, performance, as the model obtained from the three-stage pipeline.
In this case, starting with a large model is not necessary and one could instead directly train the target model from scratch.
Second, for pruning algorithms without a predeﬁned target network, training the pruned model from scratch can also achieve comparable or even better performance than ﬁne-tuning.
This observation shows that for these pruning algorithms, what matters is the obtained architecture, instead of the preserved weights, despite training the large model is required to ﬁnd that target architecture.
However, in both cases, if a pretrained large model is already available, pruning and ﬁne-tuning from it can still greatly save the training time to obtain the efﬁcient model.
The contradiction between our results and those reported in the literature might be explained by less carefully chosen hyper-parameters, data augmentation schemes and unfair computation budget for evaluating this baseline approach.
[Figure]
Our results advocate a rethinking of existing network pruning algorithms.
It seems that the overparameterization during the ﬁrst-stage training is not as beneﬁcial as previously thought.
Also, inheriting weights from a large model is not necessarily optimal, and might trap the pruned model into a bad local minimum, even if the weights are considered “important” by the pruning criterion.
Instead, our results suggest that the value of automatic pruning algorithms may lie in identifying efﬁcient structures and performing implicit architecture search, rather than selecting “important” weights.
In Section 5, we verify this hypothesis through carefully designed experiments, and show the patterns in the pruned model could provide design guidelines for efﬁcient architectures.
The rest of the paper is organized as follows: in
Section 2, we introduce background and some related works on network pruning; in Section 3, we describe our methodology for training the pruned model from scratch; in Section 4 we experiment on various pruning methods and show our main results for both pruning methods with predeﬁned or automatically discovered target architectures; in Section 5, we argue that the value of automatic pruning methods indeed lies in searching efﬁcient network architectures, and show the pruned architectures can provide transferable efﬁcient design guidelines, as supported by experiments; in Section 6 we discuss some implications and conclude the paper.
B ACKGROUND
Recent success of deep convolutional networks (Girshick et al., 2014; Long et al., 2015; He et al., 2016; 2017a) has been coupled with increased requirement of computation resources.
In particular, the model size, memory footprint, the number of computation operations (FLOPs) and power usage are major aspects inhibiting the use of deep neural networks in some resource-constrained settings.
Those large models can be infeasible to store, and run in real time on embedded systems.
To address this issue, many methods have been proposed such as low-rank approximation of weights (Denton et al., 2014; Lebedev et al., 2014), weight quantization (Courbariaux et al., 2016; Rastegari et al., 2016), knowledge distillation (Hinton et al., 2014; Romero et al., 2015) and network pruning (Han et al., 2015; Li et al., 2017), among which network pruning has gained notable attention due to their competitive performance and compatibility.
One major branch of network pruning methods is individual weight pruning, and it dates back to
Optimal Brain Damage (LeCun et al., 1990) and Optimal Brain Surgeon (Hassibi & Stork, 1993), which prune weights based on Hessian of the loss function.
More recently, Han et al.
(2015) proposes to prune network weights with small magnitude, and this technique is further incorporated into the “Deep Compression” pipeline (Han et al., 2016b) to obtain highly compressed models.
Srinivas
& Babu (2015) proposes a data-free algorithm to remove redundant neurons iteratively.
However, one drawback of these non-structured pruning methods is that the resulting weight matrices are sparse, which cannot lead to compression and speedup without dedicated hardware/libraries (Han et al., 2016a).
In contrast, structured pruning methods prune at the level of channels or even layers.
Since the original convolution structure is still preserved, no dedicated hardware/libraries are required to realize the beneﬁts.
Among structured pruning methods, channel pruning is the most popular, since it operates at the most ﬁne-grained level while still ﬁtting in conventional deep learning frameworks.
Some heuristic methods include pruning channels based on their corresponding ﬁlter weight norm (Li et al., 2017) and average percentage of zeros in the output (Hu et al., 2016).
Group sparsity is also widely used to smooth the pruning process after training (Wen et al., 2016; Alvarez & Salzmann, 2016; Lebedev & Lempitsky, 2016; Zhou et al., 2016).
Liu et al.
(2017) and Ye et al.
(2018) impose sparsity constraints on channel-wise scaling factors during training, whose magnitudes are then used for channel pruning.
Huang & Wang (2018) uses a similar technique to prune coarser structures such as residual blocks.
He et al.
(2017b)
and Luo et al.
(2017) minimizes next layer’s feature reconstruction error to determine which channels to keep.
Similarly, Yu et al.
(2018) optimizes the reconstruction error of the ﬁnal response layer and propagates a “importance score” for each channel.
Molchanov et al.
(2016) uses Taylor expansion to approximate each channel’s inﬂuence over the ﬁnal loss and prune accordingly.
Suau et al.
(2018) analyzes the intrinsic correlation within each layer and prune redundant channels.
Chin et al.
(2018) proposes a layer-wise compensate ﬁlter pruning algorithm to improve commonly-adopted heuristic pruning metrics.
He et al.
(2018a) proposes to allow pruned ﬁlters to recover during the training process.
Lin et al.
Wang et al.
(2017) prune certain structures in the network based on the current input.
Our work is also related to some recent studies on the characteristics of pruning algorithms.
Mittal et al.
(2018) shows that random channel pruning (Anwar & Sung, 2016) can perform on par with a variety of more sophisticated pruning criteria, demonstrating the plasticity of network models.
Anonymous (2019) hypothesizes that certain connections, together with their randomly initialized weights, is particularly effective for training, and a pruning algorithm can help ﬁnd such sub-networks.
Zhu & Gupta (2018) shows that training a small-dense model cannot achieve the same accuracy as a pruned large-sparse model with identical memory footprint.
In this work, we reveal a different and rather surprising characteristic of network pruning methods: ﬁne-tuning the pruned model with inherited weights is not better than training it from scratch; the resulting pruned architecture is what brings the beneﬁt and could even provide design guidelines for efﬁcient models.
M ETHODOLOGY
In this section, we describe in detail our methodology for training a small target model from scratch.
Target Pruned Architectures.
We ﬁrst divide network pruning methods into two categories.
In a pruning pipeline, the target pruned model’s architecture can be determined by either human (i.e., predeﬁned) or the pruning algorithm (i.e., automatic) (see Figure 2).
When human predeﬁnes the target architecture, a common criterion is the ratio of channels to prune in each layer.
For example, we may want to prune 50% channels in each layer of VGG.
In this case, no matter which speciﬁc channels are pruned, the pruned target architecture remains the same, because the pruning algorithm only locally prunes the least important 50% channels in each layer.
In practice, the ratio in each layer is usually selected through empirical studies or heuristics.
When the target architecture is automatically determined by a pruning algorithm, it is usually based on a pruning criterion that globally compares the importance of structures (e.g., channels) across layers.
Examples include Liu et al.
(2017), Huang & Wang (2018), Molchanov et al.
(2016) and Suau et al.
Non-structured weight pruning (Han et al., 2015) also falls into this category, where the sparsity patterns are determined by the magnitude of trained weights.
Note that it is possible to adapt a method with predeﬁned architectures to an automatic method, through appropriate modiﬁcations.
Datasets, Network Architectures and Pruning Methods.
In the network pruning literature,
CIFAR-10, CIFAR-100 (Krizhevsky, 2009), and ImageNet (Deng et al., 2009) datasets are the de-facto benchmarks, while VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016) and
DenseNet (Huang et al., 2017) are the common network architectures.
We evaluate three pruning methods with predeﬁned target architectures, Li et al.
(2017), Luo et al.
, He et al.
(2017b)
and three which automatically discover target models, Liu et al.
(2017), Huang & Wang (2018), Han et al.
For the ﬁrst ﬁve methods, we evaluate using the same (target model, dataset) pairs as presented in the original paper to keep our results comparable.
For the last one (Han et al., 2015), we use the aforementioned architectures instead, since the ones in the original paper are no longer state-of-the-art.
On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean and standard deviation of the accuracy.
Training Budget.
One crucial question is how long should we train the small pruned model from scratch?
Naively training for the same number of epochs as we train the large model might be unfair, since the small pruned model requires signiﬁcantly less computation for one epoch.
Alternatively, we could compute the ﬂoating point operations (FLOPs) for both the pruned and large models, and choose the number of training epoch for the pruned model that would lead to the same amount of computation as training the large model.
In our experiments, we use Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B to denote training for the same amount of computation budget (on ImageNet, if the pruned model saves more than 2 × FLOPs, we just double the number of epochs for training
Scratch-B, which amounts to less computation budget than large model training).
One may argue that we should instead train the small target model for fewer epochs since it typically converges faster.
However, in practice we found that increasing the training epochs within a reasonable range is rarely harmful.
We hypothesize that this is because smaller models are less prone to over-ﬁtting.
Implementation.
In order to keep our setup as close to the original paper as possible, we use the following protocols: 1) If a previous pruning method’s training setup is publicly available, e.g. Liu et al.
and Huang & Wang (2018), we adopt the original implementation; 2) Otherwise, for simpler pruning methods, e.g., Li et al.
and Han et al. (2015), we re-implement the threestage pruning procedure and achieve similar results to the original paper; 3)
For the remaining two methods (Luo et al., 2017; He et al., 2017b), the pruned models are publicly available but without the training setup, thus we choose to re-train both large and small target models from scratch.
Interestingly, the accuracy of our re-trained large model is higher than what is reported in the original paper 1 .
In this case, to accommodate the effects of different frameworks and training setups, we report the relative accuracy drop from the unpruned large model.
In all these implementations, we use standard training hyper-parameters and data-augmentation schemes.
For random weight initialization, we adopt the scheme proposed inHe et al.
For results of models ﬁne-tuned from inherited weights, we either use the released models from original papers (case 3 above) or follow the common practice of ﬁne-tuning the model using the lowest learning rate when training the large model (Li et al., 2017; He et al., 2017b).
For CIFAR, training/ﬁnetuning takes 160/40 epochs.
For ImageNet, training/ﬁne-tuning takes 90/20 epochs.
The code to reproduce the results and the trained models are available at this anynomous link: https: //drive.google.com/
open?id=1HB_1FphsWtbuMAdgHbODSLdAMnj3B6TM .
E XPERIMENTS
In this section we present our experimental results comparing training pruned models from scratch and ﬁne-tuning from inherited weights, for methods with predeﬁned and automatically discovered target architectures.
We also include an experiment on transfer learning from image classiﬁcation to object detection.
4. P REDEFINED T ARGET A RCHITECTURES
L 1 -norm based Filter Pruning (Li et al., 2017) is one of the earliest work on ﬁlter/channel pruning for convolutional networks.
In each layer, a certain percentage of ﬁlters with smaller L 1 -norm will be pruned.
Table 1 shows our results.
The Pruned Model column shows the list of predeﬁned target models (see (Li et al., 2017) for conﬁguration details on each model).
We observe that in each row, scratch-trained models achieve at least the same level of accuracy as ﬁne-tuned models, with
Scratch-B slightly higher than Scratch-E in most cases.
On ImageNet, both Scratch-B models are better than the ﬁne-tuned ones by a noticeable margin
[Table]
ThiNet (Luo et al., 2017) greedily prunes the channel that has the smallest effect on the next layer’s activation values.
As shown in Table 2, for VGG-16 and ResNet-50, both Scratch-E and ScratchB can almost always achieve better performance than the ﬁne-tuned model, often by a signiﬁcant margin.
The only exception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG-16 (FLOPs reduced by 15 × ), and as a result, drastically reducing the training budget for
Scratch-E.
The training budget of Scratch-B for this model is also 7 times smaller than the original large model, yet it can achieve the same level of accuracy as the ﬁne-tuned model
[Table] [Table]
Regression based Feature Reconstruction (He et al., 2017b)
prunes channels by minimizing the feature map reconstruction error of the next layer.
Different from ThiNet (Luo et al., 2017), this optimization problem is solved by LASSO regression.
Results are shown in Table 3.
Again, in terms of relative accuracy drop from the large models, scratch-trained models are better than the ﬁne-tuned models
[Table]
In summary, for pruning methods with predeﬁned target architectures, training the small models for the same number of epochs as the large model (Scratch-E), is often enough to achieve the same accuracy as models output by the three-stage pipeline.
Combined with the fact that the target architecture is predeﬁned, in practice one would prefer to train the small model from scratch directly.
Moreover, when provided with the same amount of computation budget (measured by FLOPs) as the large model, scratch-trained models can even lead to better performance than the ﬁne-tuned models.
4. A UTOMATICALLY D ISCOVERED T ARGET A RCHITECTURES
Network Slimming (Liu et al., 2017) imposes L 1 -sparsity on channel-wise scaling factors from
Batch Normalization layers (Ioffe & Szegedy, 2015) during training, and prunes channels with lower scaling factors afterward.
Since the channel scaling factors are compared across layers, this method produces automatically discovered target architectures.
As shown in Table
[Table]
for all networks, the small models trained from scratch can reach the same accuracy as the ﬁne-tuned models.
More speciﬁcally, we found that Scratch-B consistently outperforms (8 out of 10 experiments) the ﬁnetuned models, while Scratch-E is slightly worse but still mostly within the standard deviation.
[Table]
Sparse Structure Selection (Huang & Wang, 2018) also uses sparsiﬁed scaling factors to prune structures, and can be seen as a generalization of Network Slimming.
Other than channels, pruning can be on residual blocks in ResNet or groups in ResNeXt (Xie et al., 2017).
We examine residual blocks pruning, where ResNet-50 are pruned to be ResNet-41, ResNet-32 and ResNet-26.
Tableshows our results.
On average Scratch-E outperforms pruned models
[Table]
and for all models Scratch-B is better than both
[Table]
Non-structured Weight Pruning (Han et al., 2015) prunes individual weights that have small magnitudes.
This pruning granularity leaves the weight matrices sparse, hence it is commonly referred to as non-structured weight pruning.
Because all the network architectures we evaluated are fullyconvolutional (except for the last fully-connected layer), for simplicity, we only prune weights in convolution layers here.
Before training the pruned sparse model from scratch, we re-scale the standard deviation of the Gaussian distribution for weight initialization, based on how many non-zero weights remain in this layer.
This is to keep a constant scale of backward gradient signal (He et al., 2015).
As shown in Table 6, on CIFAR datasets, Scratch-E sometimes falls short of the ﬁne-tuned results, but Scratch-B is able to perform at least on par with the latter.
On ImageNet, we note that sometimes even Scratch-B is slightly worse than ﬁne-tuned result.
This is the only case where
Scratch-B does not achieve comparable accuracy in our attempts.
We hypothesize this could be due to the task complexity of ImageNet and the ﬁne pruning granularity.
Another possible reason is that compared with structured pruning, non-structured pruning signiﬁcantly changes the weight distribution (more details in Appendix G)
[Table] [Table]
4. T RANSFER L EARNING TO O BJECT D ETECTION
We have shown that the small pruned model can be trained from scratch to match the accuracy of the ﬁne-tuned model in classiﬁcation tasks.
To see whether this phenomenon would also hold for transfer learning to other vision tasks, we evaluate the L 1 -norm based pruning method (Li et al., 2017) on the PASCAL VOC object detection task, using Faster-RCNN (Ren et al., 2015).
Object detection frameworks usually require transferring model weights pre-trained on ImageNet classiﬁcation, and one can perform pruning either before or after the weight transfer.
More speciﬁcally, the former could be described as “train on classiﬁcation, prune on classiﬁcation, ﬁne-tune on classiﬁcation, transfer to detection”, while the latter is “train on classiﬁcation, transfer to detection, prune on detection, ﬁne-tune on detection”.
We call these two approaches Prune-C (classiﬁcation) and Prune-D (detection) respectively, and report the results in Table 7.
With a slight abuse of notation, here Scratch-E/B denotes "train the small model on classiﬁcation, transfer to detection", and is different from the setup of detection without ImageNet pre-training as in Shen et al.
[Table]
For this experiment, we adopt the code and default hyper-parameters from Yang et al.
and use PASCAL VOC 07 trainval/test set as our training/test set.
For backbone networks, we evaluate
ResNet-34-A and ResNet-34-B from the L 1 -norm based channel pruning (Li et al., 2017), which are pruned from ResNet-34.
Table 7 shows our result, and we can see that the model trained from scratch can surpass the performance of ﬁne-tuned models under the transfer setting.
Another interesting observation from Table 7 is that Prune-C is able to outperform Prune-D, which is surprising since if our goal task is detection, directly pruning away weights that are considered unimportant for detection should presumably be better than pruning on the pre-trained classiﬁcation models.
We hypothesize that this might be because pruning early in the classiﬁcation stage makes the ﬁnal model less prone to being trapped in a bad local minimum caused by inheriting weights from the large model.
This is in line with our observation that Scratch-E/B, which trains the small models from scratch starting even earlier at the classiﬁcation stage, can achieve further performance improvement.
N ETWORK P RUNING AS A RCHITECTURE SEARCH
While we have shown that the inherited weights in the pruned architecture are not better than random, the pruned architecture itself turns out to be what brings the efﬁciency beneﬁts.
In this section, we demonstrate through empirical studies that the value of automatic network pruning algorithms (Figure 2) actually lies in searching efﬁcient architectures.
Note that the connection between network pruning and architecture learning has also been made in some prior works (Han et al., 2015;
Liu et al., 2017; Gordon et al., 2018).
Parameter Efﬁciency of Pruned Architectures.
In Figure 3(left), we compare the parameter efﬁciency of architectures obtained by an automatic channel pruning method (Network Slimming (Liu et al., 2017)) with a naive predeﬁned pruning strategy that uniformly prunes the same percentage of channels in each layer.
All architectures are trained from random initialization for the same number of epochs.
We see that the architectures obtained by Network Slimming are more parameter efﬁcient, as they could achieve the same level of accuracy using 5 × fewer parameters than uniformly pruning architectures.
For non-structured weight pruning (Han et al., 2015), we conducted a similar experiment shown in Figure 3(right).
Here we uniformly sparsify all individual weights at a ﬁxed probability, and the architectures obtained this way are much less efﬁcient than the pruned architectures.
[Figure]
Weight Sparsiﬁed VGG
[Figure]
[Figure] [Figure]
[Table]
[Figure]
We also found the channel/weight pruned architectures exhibit very consistent patterns (see Tableand Figure 4).
This suggests the original large models may be redundantly designed for the task and the pruning algorithm can help us improve the efﬁciency.
Combined with the results presented in
Section 4, we hypothesize that the value of automatic pruning methods actually lies in the resulting architecture rather than the inherited weights.
Generalizable Design Principles from Pruned Architectures.
Given that the automatically discovered architectures tend to be parameter efﬁcient, one may wonder: can we derive generalizable principles from them on how to design a better architecture?
To answer this question, we conduct experiments on Network Slimming and non-structured pruning with various network architectures.
For Network Slimming, we use the average number of channels in each layer stage (layers with the same feature map size) from pruned architectures to construct a new set of architectures, and we call this approach “Guided Pruning”; for non-structured pruning, we analyze the sparsity patterns (Table 4) in the pruned architectures, and apply them to construct a new set of sparse models, which we call “Guided Sparsifying”.
The results are shown in Figure 5 and Figure 6.
It can be seen that for both Network Slimming (Figure 5 left and Figure 6) and non-structured pruning (Figure 5 right), guided design of architectures (green) can perform on par with pruned architectures (blue).
[Figure]
[Figure]
Interestingly, these guided design patterns can sometimes be transferred to a different architecture and/or a different dataset.
In Figure 5, we distill the patterns of pruned architectures from VGG-  on CIFAR-10 and apply them to design efﬁcient VGG-19 on CIFAR-100.
Similarly, in Figure 6, we transfer the average pruned sparsity patterns from DenseNet-76 on CIFAR-100 and VGGon ImageNet respectively to DenseNet-40 on CIFAR-100 and VGG-19 on ImageNet.
These sets of architectures are denoted as “Transferred Guided Pruning/Sparsifying”.
We can observe that they (brown) may sometimes be slightly worse than architectures directly pruned (blue), but are signiﬁcantly better than uniform pruning/sparsifying (red).
In these cases, one does not need to train a large model to obtain an efﬁcient model as well, as transferred design patterns can help us achieve the efﬁciency directly.
However, there also exist cases where the architectures obtained by pruning are not better than uniformly pruned ones, and we defer the results and analysis in Appendix F due to space limit.
Comparison with Traditional Architecture Search Methods.
Conventional techniques for network architecture search include reinforcement learning (Zoph & Le, 2017; Baker et al., 2017) and evolutionary algorithms (Xie & Yuille, 2017; Liu et al., 2018a).
In each iteration, a randomly initialized network is trained and evaluated to guide the search, and the search process usually requires thousands of iterations to ﬁnd the goal architecture.
In contrast, using network pruning as architecture search only requires a one-pass training, however the search space is restricted to the set of all “sub-networks” inside a large network, whereas traditional methods can search for more variations, e.g., activation functions or different layer orders.
Recently, Gordon et al.
(2018) uses a similar pruning technique to Network Slimming (Liu et al., 2017) to automate the design of network architectures;
He et al.
(2018c) prune channels using reinforcement learning and automatically compresses the architecture.
On the other hand, in the network architecture search literature, sharing/inheriting trained parameters (Pham et al., 2018; Liu et al., 2018b) has become a popular approach to accelerate the convergence and reduce the training budget during searching, though it would be interesting to investigate whether training from scratch would sometimes yield better results as we observed in network pruning.
We can see that these two areas, namely network pruning and architecture search, share many common traits and start to borrow wisdom from each other.
D ISCUSSION AND C ONCLUSION
We suggest future pruning methods be evaluated on appropriately strong baselines, especially when the target pruned architectures are predeﬁned.
In addition to high accuracy, training predeﬁned target models from scratch has the following beneﬁts over conventional network pruning procedures:
• Since the model is smaller, we can train the model using less GPU memory and possibly faster than training the original large model.
• There is no need to implement the pruning criterion and procedure, which sometimes requires ﬁne-tuning layer by layer (Luo et al., 2017) and/or needs to be customized for different network architectures (Li et al., 2017; Liu et al., 2017).
• We avoid tuning additional hyper-parameters involved in the pruning procedure.
Our results support the use of pruning methods for ﬁnding efﬁcient architectures or sparsity patterns.
This can be done using automatic pruning approaches.
In addition, there are still some cases where conventional pruning methods can be much faster than training from scratch:
• When a pre-trained large model is already given and little or no training budget is available.
• There is a need to obtain multiple models of different sizes, in this situation one can train a large model and then prune it by different ratios.
In summary, our experiments have shown that training the small pruned model from scratch can almost always achieve comparable or higher level of accuracy than the model obtained from the typical “training, pruning and ﬁne-tuning” procedure.
This changed our understanding about the necessity of over-parameterization for obtaining a ﬁnal efﬁcient model, and the effectiveness of inheriting weights that are considered important by the pruning criteria.
We further demonstrated the value of automatic pruning algorithms could be regarded as ﬁnding efﬁcient architectures and providing architecture design guidelines.
A PPENDIX A R ESULTS ON S OFT F ILTER P RUNING (H E ET AL ., 2018 A )
Soft Filter Pruning (SFP) (He et al., 2018a) prunes ﬁlters every epoch during training but also keeps updating the pruned ﬁlters,
i
[Table]
e., the pruned weights have the chance to be recovered.
In the original paper, SFP can either run upon a random initialized model or a pretrained model.
It falls into the category of predeﬁned methods (Figure 2).
Table 9 shows our results without using pretrained models and Table 10 shows the results with a pretrained model.
We use authors’ code (He et al., 2018b) for obtaining the results.
It can be seen that Scratch-E outperforms pruned models for most of the time and Scratch-B outperforms pruned models in nearly all cases.
Therefore, our observations also hold on soft ﬁlter pruning
[Table] [Table]
B E XPERIMENTS ON THE L OTTERY T ICKET H YPOTHESIS (A NONYMOUS , 2019)
The lottery ticket hypothesis (Anonymous, 2019) conjectures that inside the large network, a subnetwork together with their initialization makes the training particularly effective, and together they are termed the “winning ticket”.
In this hypothesis, the original initialization of the sub-network (before large model training) is needed for it to achieve competitive performance when trained in isolation.
This is shown in their Figure 3, where training the sub-network with randomly re-initialized weights performs worse than training it with the original initialization inside the large network.
In contrast, our work does not require reuse of the original initialization of the pruned model, and shows that random initialization is enough for the pruned model to achieve competitive performance.
We note that the experiments in Anonymous (2019) are conducted on very small plain networks ( < 6 layers), and use Adam with a small learning rate, which could not lead to high-performance on common benchmarks such as CIFAR-10.
Our experiments are on modern network architectures
(e.g., ResNet and DenseNet) and follow the standard training practice (e
[Table]
g., use SGD with momentum and a large initial learning rate of 0.1) in both the image classiﬁcation literature (He et al., 2016; Huang et al., 2017) and the original setting of the pruning algorithms we evaluate.
Moreover,
Anonymous (2019) tests the hypothesis on non-structured weight pruning, while we consider six pruning methods on various levels of structures
[Table]
[Table]
We investigate whether the original initialization is needed to achieve competitive performance in a standard setting, by comparing the models trained with the original initialization (“winning ticket”) and that trained from randomly re-initialized weights.
We show our results for non-structured weight pruning (Han et al., 2015) in Table 11 and L 1 -norm based ﬁlter pruning (Li et al., 2017) in Table
[Table]
It can be seen that using the original initialization as in (Anonymous, 2019) does not provide any advantage over random initialization in a standard setting.
Note that the authors of (Anonymous, 2019) also report in their Appendix D, that the “winning ticket” cannot be found on a ResNetusing the original large learning rate used in ResNet (He et al., 2016).
The reason why the original initialization is helpful in the setting of Anonymous (2019) might be that the used learning rate is relatively small, thus the weights of the ﬁnal trained model are not changed to a large degree from the original initialization.
C A GGRESSIVELY P RUNED M ODELS
It would be interesting to see whether our observation still holds if the model is very aggressively pruned, since they might not have enough capacity to be trained from scratch and achieve decent accuracy.
Here we provide results using Network Slimming (Liu et al., 2017).
From Table 13, it can be seen that when the pruned ratio is large, training from scratch is better than ﬁne-tuned models by an even larger margin, and in this case ﬁne-tuned models are signiﬁcantly worse than the unpruned accuracy.
Note that in Table 2, the VGG-Tiny model we evaluated for ThiNet (Luo et al., 2017) is also a very aggressively pruned model (FLOPs reduced by 15 × and parameters reduced by 100 × ).
[Table]
D F INE - TUNING E POCHS
Generally
[Table]
pruning algorithms use fewer epochs for ﬁne-tuning than training the large model (Li et al., 2017; He et al., 2017b; Luo et al., 2017).
For example, L 1 -norm based ﬁlter pruning (Li et al., 2017) uses 164 epochs for training on CIFAR-10 datasets
[Table]
and only ﬁne-tunes the pruned networks for 40 epochs.
This is due to that mostly small learning rates are used for ﬁne-tuning to better preserve the weights from the large model.
Here we experiment with ﬁne-tuning for more epochs (e.g., for the same number of epochs as Scratch-E) and show it does not bring noticeable performance improvement
[Table]
We use L 1 -norm ﬁlter pruning (Li et al., 2017) for this experiment.
Table 14 shows our results with different number of epochs for ﬁne-tuning.
It can be seen that ﬁne-tuning for more epochs gives negligible accuracy increase and sometimes small decrease, and Scratch-E models are still on par with models ﬁne-tuned for enough epochs.
E E XTENDING THE STANDARD TRAINING SCHEDULE
In our experiments, we use the standard training schedule for both CIFAR (160 epochs) and ImageNet (90 epochs).
Here we show that our observation still holds after we extend the standard training schedule.
We use L 1 -norm based ﬁlter pruning (Li et al., 2017) for this experiment.
Table 15 shows our results when we extend the standard training schedule of CIFAR from 160 toepochs.
We observe that scratch trained models still perform at least on par with ﬁne-tuned models.
[Table]
F A DDITIONAL R ESULTS FOR A RCHITECTURE S EARCH
Continuing on section
[Table]
we show some cases where the pruned architectures or guided pruning/sparsifying are not better than architectures obtained by uniform pruning/sparsifying in Figure 7.
The blue and green curves are not noticeably better than the red curve.
This phenomenon happens more likely on modern architectures like ResNet and DenseNet.
When we investigate the sparsity patterns of those pruned architectures (shown in Table 16, Table 17 and Table 18)
[Figure]
we ﬁnd that they exhibit near-uniform sparsity patterns across stages, which might be the reason why it can only perform on par with uniform pruning.
In contrast, for VGG the pruned sparsity patterns can always beat the uniform ones as shown in Figure 3 and Figure 5.
We also show the sparsity patterns of
VGG pruned by Network Slimming (Liu et al., 2017) in Table
[Figure]
and they are rather far from uniform.
Compared to ResNet and DenseNet, we can see that VGG’s redundancy is rather inbalanced across layer stages
[Figure]
while modern networks’ redundancy is shared more evenly.
Network pruning techniques can help us identify the redundancy better in the former case
[Figure]
[Table] [Table]
G W EIGHT D ISTRIBUTION
We show the weight distribution of unpruned large models, ﬁne-tuned pruned models and scratchtrained pruned models, for two pruning methods: Network Slimming (Liu et al., 2017) and nonstructured weight pruning (Han et al., 2015).
We choose DenseNet-40 and CIFAR-10 for visualization and compare the weight distribution of unpruned models, ﬁne-tuned models and scratch-trained
[Table]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
models.
For Network Slimming, the prune ratio is chosen to be 60%.
For non-structured weight pruning, the prune ratio is chosen to be 80%.
Figure 8 shows our result.
We can see that the weight distribution of ﬁne-tuned models and scratch-trained pruned models are different from the unpruned large models – the weights that are close to zero are much fewer.
This seems to imply that there are less redundant structures in the found pruned architecture, and support the effect of architecture learning for automatic pruning methods.
For non-structured weight pruning, the ﬁne-tuned model also has signiﬁcantly different weight distribution from the scratch-trained model – it has nearly no near-zero values.
This might be a reason why on ImageNet sometimes scratch trained models cannot achieve the accuracy of the ﬁne-tuned models, as shown in Table 6.