Original: Rethinking the Value of Network Pruning∗
Revised: Rethinking the Value of Network Pruning

Original: Anonymous Author s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Network pruning is widely used for reducing the heavy computational cost ofdeep models.
Revised: Network pruning is widely used for reducing the heavy computational cost of deep models.

Original: A typical pruning algorithm is a three-stage pipeline, i e., training (alarge model), pruning and ﬁne-tuning.
Revised: A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and ﬁne-tuning.

Original: In this work, we make a rather surprisingobservation: ﬁne-tuning a pruned model only gives comparable or even worseperformance than training that model with randomly initialized weights.
Revised: In this work, we make a rather surprising observation: ﬁne-tuning a pruned model only gives comparable or even worse performance than training that model with randomly initialized weights.

Original: Ourresults have several implications: 1) training a large, over-parameterized model isnot necessary to obtain an efﬁcient ﬁnal model, 2) learned “important” weightsof the large model are not necessarily useful for the small pruned model, 3) thepruned architecture itself, rather than a set of inherited weights, is what leads to theefﬁciency beneﬁt in the ﬁnal model, which suggests that some pruning algorithmscould be seen as performing network architecture search
Revised: Our results have several implications: 1) training a large, over-parameterized model is not necessary to obtain an efﬁcient ﬁnal model, 2) learned “important” weights of the large model are not necessarily useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited weights, is what leads to the efﬁciency beneﬁt in the ﬁnal model, which suggests that some pruning algorithms could be seen as performing network architecture search.

Original:  Introduction
Revised:  Introduction

Original: Network pruning is a commonly used approach for obtaining an efﬁcient neural network.
Revised: Network pruning is a commonly used approach for obtaining an efﬁcient neural network.

Original: A typicalprocedure of network pruning consists of three stages: 1) train a large, over-parameterized model, 2)prune the unimportant weights according to a certain criterion, and 3) ﬁne-tune the pruned model toregain accuracy.
Revised: A typical procedure of network pruning consists of three stages: 1) train a large, over-parameterized model, 2) prune the unimportant weights according to a certain criterion, and 3) ﬁne-tune the pruned model to regain accuracy.

Original: Generally, there are two common beliefs behind this pruning procedure.
Revised: Generally, there are two common beliefs behind this pruning procedure.

Original: First, it isbelieved that training a large network ﬁrst is important [2] and the three-stage pipeline can outperformdirectly training the small model from scratch.
Revised: First, it is believed that training a large network ﬁrst is important [1] and the three-stage pipeline can outperform directly training the small model from scratch.

Original: Second, both the architectures and the weights of thepruned model are believed to be essential for the ﬁnal efﬁcient model, which is why most existingpruning methods choose to ﬁne-tune the pruned model instead of training it from scratch.
Revised: Second, both the architectures and the weights of the pruned model are believed to be essential for the ﬁnal efﬁcient model, which is why most existing pruning methods choose to ﬁne-tune the pruned model instead of training it from scratch.

Original: Alsobecause of this, how to select the set of important weights is a very active research topic [2, 3, 4, 5].
Revised: Also because of this, how to select the set of important weights is a very active research topic [1, 2, 3, 4].

Original: [Figure]
Revised: [Figure]

Original: In this work, we show that both of the beliefs mentionedabove are not necessarily true.
Revised: In this work, we show that both of the beliefs mentioned above are not necessarily true.

Original: We make a surprising observation that directly training the target pruned model fromrandom initialization can achieve the same or better performance as the model obtained from the three-stage pipeline.
Revised: We make a surprising observation that directly training the target pruned model from random initialization can achieve the same or better performance as the model obtained from the three-stage pipeline.

Original: This means that, for pruning methods with a predeﬁnedtarget architecture (Figure 1), starting with a large modelis not necessary and one could instead directly train thetarget model from scratch.
Revised: This means that, for pruning methods with a predeﬁned target architecture (Figure 1), starting with a large model is not necessary and one could instead directly train the target model from scratch.

Original: For pruning algorithms withautomatically discovered target architectures, what bringsthe efﬁciency beneﬁt is the obtained architecture, insteadof the inherited weights.
Revised: For pruning algorithms with automatically discovered target architectures, what brings the efﬁciency beneﬁt is the obtained architecture, instead of the inherited weights.

Original: Our results advocate a rethinkingof existing network pruning algorithms: the preserved “important” weights from the large model are not necessaryfor obtaining an efﬁcient ﬁnal model; instead, the value ofautomatic network pruning methods may lie in identifyingefﬁcient architectures and performing implicit architecture search
Revised: Our results advocate a rethinking of existing network pruning algorithms: the preserved “important” weights from the large model are not necessary for obtaining an efﬁcient ﬁnal model; instead, the value of automatic network pruning methods may lie in identifying efﬁcient architectures and performing implicit architecture search.

Original:  Methodology
Revised:  Methodology

Original: Target Pruned Architectures.
Revised: Target Pruned Architectures.

Original: We divide pruning methods by whether the target pruned architectureis determined by either human or the pruning algorithm (see Figure 1).
Revised: We divide pruning methods by whether the target pruned architecture is determined by either human or the pruning algorithm (see Figure 1).

Original: An example of designingpredeﬁned target architecture is specifying how many channels to prune in each layer.
Revised: An example of designing predeﬁned target architecture is specifying how many channels to prune in each layer.

Original: In contrast,when the target architecture is automatically discovered, the pruning algorithm determines how manychannels to prune in each layer, by comparing the importance of channels across layers.
Revised: In contrast, when the target architecture is automatically discovered, the pruning algorithm determines how many channels to prune in each layer, by comparing the importance of channels across layers.

Original: Training from scratch.
Revised: Training from scratch.

Original: Because the pruned model requires less computation, it may be unfair totrain the pruned model for the same number of epochs as the large model.
Revised: Because the pruned model requires less computation, it may be unfair to train the pruned model for the same number of epochs as the large model.

Original: In our experiments, weuse Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B todenote training for the same amount of computation budget (measured by FLOPs).
Revised: In our experiments, we use Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B to denote training for the same amount of computation budget (measured by FLOPs).

Original: We use standardtraining hyper-parameters and data-augmentation schemes.
Revised: We use standard training hyper-parameters and data-augmentation schemes.

Original: The optimization method used is SGDwith Nesterov momentum.
Revised: The optimization method used is SGD with Nesterov momentum.

Original:  Experiments
Revised:  Experiments

Original: In this section we present our experimental results comparing training pruned models from scratchwith ﬁne-tuning.
Revised: In this section we present our experimental results comparing training pruned models from scratch with ﬁne-tuning.

Original: For method with predeﬁned architectures, we evaluate L 1 -norm based channelpruning method [4].
Revised: For method with predeﬁned architectures, we evaluate L 1 -norm based channel pruning method [3].

Original: For method with automatically discovered target architectures, we use Network
Revised: For method with automatically discovered target architectures, we use Network

Original: Slimming [6].
Revised: Slimming [5].

Original: The models, datasets and the number of epochs for ﬁne-tuning are the same as thosein the original paper.
Revised: The models, datasets and the number of epochs for ﬁne-tuning are the same as those in the original paper.

Original: More results for other pruning methods and transfer learning can be found in
Revised: More results for other pruning methods and transfer learning can be found in

Original: Appendix B.
Revised: Appendix B.

Original: 3.1 Predeﬁned Target Architectures
Revised: 3.1 Predeﬁned Target Architectures

Original: L 1 -norm based Channel Pruning [4] is one of the earliest work on channel pruning for convolutional networks.
Revised: L 1 -norm based Channel Pruning [3] is one of the earliest work on channel pruning for convolutional networks.

Original: In each layer, a certain percentage of channels with smaller L 1 -norm of its ﬁlterweights will be pruned.
Revised: In each layer, a certain percentage of channels with smaller L 1 -norm of its ﬁlter weights will be pruned.

Original: Table 1 shows our results.
Revised: Table 1 shows our results.

Original: The Pruned Model column shows the list ofpredeﬁned target models.
Revised: The Pruned Model column shows the list of predeﬁned target models.

Original: We observe that in each row, scratch-trained models achieve at least thesame level of accuracy as ﬁne-tuned models, with Scratch-B slightly higher than Scratch-E in mostcases.
Revised: We observe that in each row, scratch-trained models achieve at least the same level of accuracy as ﬁne-tuned models, with Scratch-B slightly higher than Scratch-E in most cases.

Original: On ImageNet, both Scratch-B models are better than the ﬁne-tuned ones
Revised: On ImageNet, both Scratch-B models are better than the ﬁne-tuned ones.

Original: [Table]
Revised: [Table]

Original: 3.2 Automatically Discovered Target Architectures
Revised: 3.2 Automatically Discovered Target Architectures

Original: [Table]
Revised: [Table]

Original: Network Slimming [6] imposes L 1 -sparsity regularization on channel-wise scaling factors from
Revised: Network Slimming [5] imposes L 1 -sparsity regularization on channel-wise scaling factors from

Original: Batch Normalization layers [7] during training, and prunes channels with lower scaling factorsafterward.
Revised: Batch Normalization layers [6] during training, and prunes channels with lower scaling factors afterward.

Original: Since the channel scaling factors are compared across layers, this method producesautomatically discovered target architectures.
Revised: Since the channel scaling factors are compared across layers, this method produces automatically discovered target architectures.

Original: As shown in Table 2, for all networks, the smallmodels trained from scratch can reach the same accuracy as the ﬁne-tuned models, where Scratch-Bconsistently outperforms the ﬁne-tuned models.
Revised: As shown in Table 2, for all networks, the small models trained from scratch can reach the same accuracy as the ﬁne-tuned models, where Scratch-B consistently outperforms the ﬁne-tuned models.

Original: Morever, when we extend the standard training oflarge model, the above observation still holds.
Revised: Morever, when we extend the standard training of large model, the above observation still holds.

Original:  Network Pruning as Architecture Search
Revised:  Network Pruning as Architecture Search

Original: In this section, we demonstrate that the value of automatic network pruning methods actually lies insearching efﬁcient architectures.
Revised: In this section [Figure] we demonstrate that the value of automatic network pruning methods actually lies in searching efﬁcient architectures.

Original: We use Network Slimming [6] as an example automatic method.
Revised: We use Network Slimming [5] as an example automatic method.

Original: Parameter Efﬁciency of Pruned Architectures.
Revised: Parameter Efﬁciency of Pruned Architectures.

Original: In Figure 2(left), we compare the parameterefﬁciency of architectures obtained by Network Slimming with a naive predeﬁned pruning strategythat uniformly prunes the same percentage of channels in all layer.
Revised: In Figure 2(left), we compare the parameter efﬁciency of architectures obtained by Network Slimming with a naive predeﬁned pruning strategy that uniformly prunes the same percentage of channels in all layer.

Original: All architectures are trained fromrandom initialization for the same number of epochs without sparsity regularization.
Revised: All architectures are trained from random initialization for the same number of epochs without sparsity regularization.

Original: We see that thearchitectures obtained by Network Slimming are more parameter efﬁcient, as they could achieve thesame level of accuracy using 5 × fewer parameters than uniformly pruning architectures.
Revised: We see that the architectures obtained by Network Slimming are more parameter efﬁcient, as they could achieve the same level of accuracy using 5 × fewer parameters than uniformly pruning architectures.

Original: [Figure]
Revised: [Figure]

Original: Generalizable Design Principles from Pruned Architectures.
Revised: Generalizable Design Principles from Pruned Architectures.

Original: Given that the automatically discovered architectures tend to be parameter efﬁcient, here we show a generalizable principle on designingbetter architectures.
Revised: Given that the automatically discovered architectures tend to be parameter efﬁcient, here we show a generalizable principle on designing better architectures.

Original: We show two design strategies: 1) “Guided Pruning”: use the average numberof channels in each layer stage (layers with the same feature map size) from pruned architecturesto construct a new set of architectures.
Revised: We show two design strategies: 1) “Guided Pruning”: use the average number of channels in each layer stage (layers with the same feature map size) from pruned architectures to construct a new set of architectures.

Original: 2) “Transferred Guided Pruning”: use these patterns from adifferent architecture on a different dataset.
Revised: 2) “Transferred Guided Pruning”: use these patterns from a different architecture on a different dataset.

Original: Figure 2(right) shows our results.
Revised: Figure 2(right) shows our results.

Original: Here for “Transferred
Revised: Here for “Transferred

Original: Guided Pruning”
Revised: Guided Pruning”

Original: , we use the patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100.
Revised: , we use the patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100.

Original: We observe that both “Guided Pruning” (green) and “Transferred
Revised: We observe that both “Guided Pruning” (green) and “Transferred

Original: Guided Pruning” (brown) can both perform on par architectures directly pruned on the VGG-19 and
Revised: Guided Pruning” (brown) can both perform on par architectures directly pruned on the VGG-19 and

Original: CIFAR-100 (blue), and are signiﬁcantly better than uniform pruning (red).
Revised: CIFAR-100 (blue), and are signiﬁcantly better than uniform pruning (red).

Original: This means we coulddistill generalizable design principles from pruned architectures.
Revised: This means we could distill generalizable design principles from pruned architectures.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: In practice, for methods with predeﬁned target architectures, training from scratch is more computationally efﬁcient and saves us from implementing the pruning procedure and tuning the additionalhyper-parameters.
Revised: In practice, for methods with predeﬁned target architectures, training from scratch is more computationally efﬁcient and saves us from implementing the pruning procedure and tuning the additional hyper-parameters.

Original: Still, pruning methods are useful when a pretrained large model is already given,in this case ﬁne-tuning is much faster.
Revised: Still, pruning methods are useful when a pretrained large model is already given, in this case ﬁne-tuning is much faster.

Original: Also, obtaining multiple models of different sizes can be donequickly by pruning from a large model by different ratios.
Revised: Also, obtaining multiple models of different sizes can be done quickly by pruning from a large model by different ratios.

Original: In summary, our experiments have shown that training the small pruned model from scratch can almostalways achieve comparable or higher level of accuracy than the model obtained from the typical “training, pruning and ﬁne-tuning” procedure.
Revised: In summary, our experiments have shown that training the small pruned model from scratch can almost always achieve comparable or higher level of accuracy than the model obtained from the typical “training, pruning and ﬁne-tuning” procedure.

Original: This changed our previous belief that over-parameterization is necessary for obtaining a ﬁnal efﬁcient model, and the understanding on effectiveness of inheriting weights that are considered important by the pruning criteria.
Revised: This changed our previous belief that over-parameterization is necessary for obtaining a ﬁnal efﬁcient model, and the understanding on effectiveness of inheriting weights that are considered important by the pruning criteria.

Original: We further demonstrated the value of automatic pruning algorithms could be regarded as ﬁnding efﬁcient architectures and providing architecture design guidelines.
Revised: We further demonstrated the value of automatic pruning algorithms could be regarded as ﬁnding efﬁcient architectures and providing architecture design guidelines.

Original: AppendixA Experiment Setups
Revised: Appendix A Experiment Setups

Original: Here we provide additional details on our experiment setups.
Revised: Here we provide additional details on our experiment setups.

Original: Datasets, Network Architectures and Pruning Methods.
Revised: Datasets, Network Architectures and Pruning Methods.

Original: In the network pruning literature, CIFAR10, CIFAR-100 [8], and ImageNet [9] datasets are the de-facto benchmarks, while VGG [10], ResNet[11] and DenseNet [12] are the common network architectures.
Revised: In the network pruning literature, CIFAR10, CIFAR-100 [7], and ImageNet [8] datasets are the de-facto benchmarks, while VGG [9], ResNet [10] and DenseNet [11] are the common network architectures.

Original: We evaluate three pruning methodswith predeﬁned target architectures, [2, 4, 5], and three which automatically discover target models,[3, 6, 13].
Revised: We evaluate three pruning methods with predeﬁned target architectures, [1, 3, 4], and three which automatically discover target models, [2, 5, 12].

Original: For the ﬁrst ﬁve methods, we evaluate using the same (target model, dataset) pairs aspresented in the original paper to keep our results comparable.
Revised: For the ﬁrst ﬁve methods, we evaluate using the same (target model, dataset) pairs as presented in the original paper to keep our results comparable.

Original: For the last one [3], we use theaforementioned architectures instead, since the ones in the original paper are no longer state-of-theart.
Revised: For the last one [2], we use the aforementioned architectures instead, since the ones in the original paper are no longer state-of-theart.

Original: On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean andstandard deviation of the accuracy.
Revised: On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean and standard deviation of the accuracy.

Original: Implementation.
Revised: Implementation.

Original: In order to keep our setup as close to the original paper as possible, we use thefollowing protocols: 1) If a previous pruning method’s training setup is publicly available, e.g.[6] and [13], we adopt the original implementation; 2) Otherwise, for simpler pruning methods,e.g., [3, 4], we re-implement the three-stage pruning procedure and achieve similar results to theoriginal paper; 3)
Revised: In order to keep our setup as close to the original paper as possible, we use the following protocols: 1) If a previous pruning method’s training setup is publicly available, e.g. [5] and [12], we adopt the original implementation; 2) Otherwise, for simpler pruning methods, e.g., [2, 3], we re-implement the three-stage pruning procedure and achieve similar results to the original paper; 3)

Original: For the remaining two methods [2, 5], the pruned models are publicly availablebut without the training setup, thus we choose to re-train both large and small target models fromscratch.
Revised: For the remaining two methods [1, 4], the pruned models are publicly available but without the training setup, thus we choose to re-train both large and small target models from scratch.

Original: Interestingly, the accuracy of our re-trained large model is higher than what is reported in theoriginal paper 2 .
Revised: Interestingly, the accuracy of our re-trained large model is higher than what is reported in the original paper 3 .

Original: In this case, to accommodate the effects of different frameworks and training setups,we report the relative accuracy drop from the unpruned large model.
Revised: In this case, to accommodate the effects of different frameworks and training setups, we report the relative accuracy drop from the unpruned large model.

Original: For random weight initialization, we adopt the scheme proposed in [16].
Revised: For random weight initialization, we adopt the scheme proposed in [15].

Original: For results of modelsﬁne-tuned from inherited weights, we either use the released models from original papers (for case 3above) or follow the common practice of ﬁne-tuning the model using the lowest learning rate whentraining the large model [4, 5].
Revised: For results of models ﬁne-tuned from inherited weights, we either use the released models from original papers (for caseabove) or follow the common practice of ﬁne-tuning the model using the lowest learning rate when training the large model [3, 4].

Original: The code to reproduce the results and trained ImageNet models arepublicly available in the following anonymous link: https://drive.google.com/open?id=1HB_1FphsWtbuMAdgHbODSLdAMnj3B6TM .
Revised: The code to reproduce the results and trained ImageNet models are publicly available in the following anonymous link: https://drive.google.com/open?id=1HB_ 1FphsWtbuMAdgHbODSLdAMnj3B6TM .

Original: B Additional Experiment Results
Revised: B Additional Experiment Results

Original: The results of two pruning methods are already shown in the extended abstract, here in appendix weprovide results of the remaining four pruning methods.
Revised: The results of two pruning methods are already shown in the extended abstract, here in appendix we provide results of the remaining four pruning methods.

Original: We also include an experiment on transferlearning from image classiﬁcation to object detection.
Revised: We also include an experiment on transfer learning from image classiﬁcation to object detection.

Original: B.1 Predeﬁned Target Architectures
Revised: B.1 Predeﬁned Target Architectures

Original: ThiNet [2] greedily prunes the channel that has the smallest effect on the next layer’s activationvalues.
Revised: ThiNet [1] greedily prunes the channel that has the smallest effect on the next layer’s activation values.

Original: As shown in Table 3, for VGG-16 and ResNet-50, both Scratch-E and Scratch-B can almostalways achieve better performance than the ﬁne-tuned model, often by a signiﬁcant margin.
Revised: As shown in Table 3, for VGG-16 and ResNet-50, both Scratch-E and Scratch-B can almost always achieve better performance than the ﬁne-tuned model, often by a signiﬁcant margin.

Original: The onlyexception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG-16(FLOPs reduced by 15 × ), and as a result, drastically reducing the training budget for Scratch-E. Thetraining budget of Scratch-B for this model is also 7 times smaller than the original large model, yetit can achieve the same level of accuracy as the ﬁne-tuned model.
Revised: The only exception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG- (FLOPs reduced by 15 × ), and as a result, drastically reducing the training budget for Scratch-E. The training budget of Scratch-B for this model is also 7 times smaller than the original large model, yet it can achieve the same level of accuracy as the ﬁne-tuned model.

Original: Regression based Feature Reconstruction [5] prunes channels by minimizing the feature mapreconstruction error of the next layer.
Revised: Regression based Feature Reconstruction [4] prunes channels by minimizing the feature map reconstruction error of the next layer.

Original: Different from ThiNet [2], this optimization problem is solvedby LASSO regression.
Revised: Different from ThiNet [1], this optimization problem is solved by LASSO regression.

Original: Results are shown in Table 4.
Revised: Results are shown in Table 4.

Original: Again, in terms of relative accuracy drop fromthe large models, scratch-trained models are better than the ﬁne-tuned models.
Revised: Again, in terms of relative accuracy drop from the large models, scratch-trained models are better than the ﬁne-tuned models.

Original: In summary, for pruning methods with predeﬁned target architectures, training the small modelsfor the same number of epochs as the large model (Scratch-E), is often enough to achieve thesame accuracy as models output by the three-stage pipeline.
Revised: In summary, for pruning methods with predeﬁned target architectures, training the small models for the same number of epochs as the large model (Scratch-E), is often enough to achieve the same accuracy as models output by the three-stage pipeline.

Original: Combined with the fact that the targetarchitecture is predeﬁned, in practice one would prefer to train the small model from scratch directly.
Revised: Combined with the fact that the target architecture is predeﬁned, in practice one would prefer to train the small model from scratch directly.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Similar to Table 3, we compare relative accuracy drop from unpruned large models.
Revised: Similar to Table 3, we compare relative accuracy drop from unpruned large models.

Original: Moreover, when provided with the same amount of computation budget (measured by FLOPs) as thelarge model, scratch-trained models can even lead to better performance than the ﬁne-tuned models.
Revised: Moreover, when provided with the same amount of computation budget (measured by FLOPs) as the large model, scratch-trained models can even lead to better performance than the ﬁne-tuned models.

Original: B.2 Automatically Discovered Target Architectures
Revised: B.2 Automatically Discovered Target Architectures

Original: Sparse Structure Selection [13] also imposes sparsity regularization on the scaling factors duringtraining to prune structures, and can be seen as a generalization of Network Slimming.
Revised: Sparse Structure Selection [12] also imposes sparsity regularization on the scaling factors during training to prune structures, and can be seen as a generalization of Network Slimming.

Original: Other thanchannels, pruning can be on residual blocks in ResNet or groups in ResNeXt [17].
Revised: Other than channels, pruning can be on residual blocks in ResNet or groups in ResNeXt [16].

Original: We examineresidual blocks pruning, where ResNet-50 are pruned to be ResNet-41, ResNet-32 and ResNet-26.
Revised: We examine residual blocks pruning, where ResNet-50 are pruned to be ResNet-41, ResNet-32 and ResNet-26.

Original: Table 7 shows our results.
Revised: Table 7 shows our results.

Original: On average Scratch-E outperforms pruned models, and for all models
Revised: On average Scratch-E outperforms pruned models, and for all models

Original: Scratch-B is better than both.
Revised: Scratch-B is better than both.

Original: [Table]
Revised: [Table]

Original: Non-structured Weight Pruning [3] prunes individual weights that have small magnitudes.
Revised: Non-structured Weight Pruning [2] prunes individual weights that have small magnitudes.

Original: Thispruning granularity leaves the weight matrices sparse, hence it is commonly referred to as nonstructured weight pruning.
Revised: This pruning granularity leaves the weight matrices sparse, hence it is commonly referred to as nonstructured weight pruning.

Original: Because all the network architectures we evaluated are fully-convolutional(except for the last fully-connected layer), for simplicity, we only prune weights in convolution layershere.
Revised: Because all the network architectures we evaluated are fully-convolutional (except for the last fully-connected layer), for simplicity, we only prune weights in convolution layers here.

Original: Before training the pruned sparse model from scratch, we re-scale the standard deviation ofthe Gaussian distribution for weight initialization, based on how many non-zero weights remain inthis layer.
Revised: Before training the pruned sparse model from scratch, we re-scale the standard deviation of the Gaussian distribution for weight initialization, based on how many non-zero weights remain in this layer.

Original: This is to keep a constant scale of backward gradient signal [16].
Revised: This is to keep a constant scale of backward gradient signal [15].

Original: As shown in Table 6,on CIFAR datasets, Scratch-E sometimes falls short of the ﬁne-tuned results, but Scratch-B is able to perform at least on par with the latter.
Revised: As shown in Table 6, on CIFAR datasets, Scratch-E sometimes falls short of the ﬁne-tuned results, but Scratch-B is able to perform at least on par with the latter.

Original: On ImageNet, we note that sometimes even Scratch-Bis slightly worse than ﬁne-tuned result.
Revised: On ImageNet, we note that sometimes even Scratch-B is slightly worse than ﬁne-tuned result.

Original: This is the only case where Scratch-B does not achievecomparable accuracy in our attempts.
Revised: This is the only case where Scratch-B does not achieve comparable accuracy in our attempts.

Original: We hypothesize this could be due to the task complexity of
Revised: We hypothesize this could be due to the task complexity of

Original: ImageNet and the ﬁne pruning granularity.
Revised: ImageNet and the ﬁne pruning granularity

Original: [Table]
Revised: [Table]

Original: Effects of Sparsity Regularization.
Revised: Effects of Sparsity Regularization.

Original: Some methods [6, 13] use sparsity regularization during thetraining of large, over-parameterized models, to smooth the following pruning process
Revised: Some methods [5, 12] use sparsity regularization during the training of large, over-parameterized models, to smooth the following pruning process

Original: [Table]
Revised: [Table]

Original: and in ﬁnetuning, no such sparsity is imposed.
Revised: and in ﬁnetuning, no such sparsity is imposed.

Original: In all of our experiments presented above
Revised: In all of our experiments presented above

Original: [Table]
Revised: [Table]

Original: no sparsity is used fortraining the pruned models from scratch.
Revised: no sparsity is used for training the pruned models from scratch.

Original: Here we analyze the effects of using sparsity regularization(or not) for Network Slimming [6].
Revised: Here we analyze the effects of using sparsity regularization (or not) for Network Slimming [5].

Original: Table 7 shows the results when all training procedures are withsparsity regularization.
Revised: Table 7 shows the results when all training procedures are with sparsity regularization.

Original: We can see that Scratch-B are still able to be on par with the ﬁne-tuned models.
Revised: We can see that Scratch-B are still able to be on par with the ﬁne-tuned models.

Original: Table 8 shows the results when we do not use sparsity regularization in any training procedures,including large model training.
Revised: Table 8 shows the results when we do not use sparsity regularization in any training procedures, including large model training.

Original: We can see that when no sparsity is induced, Scratch-B can alsoconsistently achieve comparable results with the ﬁne-tuned models.
Revised: We can see that when no sparsity is induced, Scratch-B can also consistently achieve comparable results with the ﬁne-tuned models

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B.3 Transfer Learning to Object Detection
Revised: B.3 Transfer Learning to Object Detection

Original: We have shown that the small pruned model can be trained from scratch to match the accuracy of theﬁne-tuned model in classiﬁcation tasks.
Revised: We have shown that the small pruned model can be trained from scratch to match the accuracy of the ﬁne-tuned model in classiﬁcation tasks.

Original: To see whether this phenomenon would also hold for transferlearning to other vision tasks, we evaluate the L 1 -norm based pruning method [4] on the PASCAL
Revised: To see whether this phenomenon would also hold for transfer learning to other vision tasks we evaluate the L 1 -norm based pruning method [3] on the PASCAL

Original: VOC object detection task, using Faster-RCNN [18]
Revised: VOC object detection task, using Faster-RCNN [17]

Original: [Table] Object detection frameworks usually require transferring model weights pre-trained on ImageNetclassiﬁcation, and one can perform pruning either before or after the weight transfer.
Revised: Object detection frameworks usually require transferring model weights pre-trained on ImageNet classiﬁcation, and one can perform pruning either before or after the weight transfer.

Original: More speciﬁcally,the former could be described as “train on classiﬁcation, prune on classiﬁcation, ﬁne-tune onclassiﬁcation, transfer to detection”, while the latter is “train on classiﬁcation, transfer to detection,prune on detection, ﬁne-tune on detection”.
Revised: More speciﬁcally, the former could be described as “train on classiﬁcation, prune on classiﬁcation, ﬁne-tune on classiﬁcation, transfer to detection”, while the latter is “train on classiﬁcation, transfer to detection, prune on detection, ﬁne-tune on detection”.

Original: We call these two approaches Prune-C (classiﬁcation)and Prune-D (detection) respectively, and report the results in Table 9.
Revised: We call these two approaches Prune-C (classiﬁcation) and Prune-D (detection) respectively, and report the results in Table 9.

Original: With a slight abuse of notation,here Scratch-E/B denotes "train the small model on classiﬁcation, transfer to detection", and isdifferent from the setup of detection without ImageNet pre-training as in [19].
Revised: With a slight abuse of notation, here Scratch-E/B denotes "train the small model on classiﬁcation, transfer to detection", and is different from the setup of detection without ImageNet pre-training as in [18].

Original: [Table]
Revised: [Table]

Original: For this experiment, we adopt the code and default hyper-parameters from [20], and use PASCAL
Revised: For this experiment, we adopt the code and default hyper-parameters from [19], and use PASCAL

Original: VOC 07 trainval/test set as our training/test set.
Revised: VOC 07 trainval/test set as our training/test set.

Original: For backbone networks, we evaluate ResNet-34-Aand ResNet-34-B from the L 1 -norm based channel pruning [4], which are pruned from ResNet34.
Revised: For backbone networks, we evaluate ResNet-34-A and ResNet-34-B from the L 1 -norm based channel pruning [3], which are pruned from ResNet34.

Original: Table 9 shows our result, and we can see that the model trained from scratch can surpass theperformance of ﬁne-tuned models under the transfer setting.
Revised: Table 9 shows our result, and we can see that the model trained from scratch can surpass the performance of ﬁne-tuned models under the transfer setting.

Original: Another interesting observation from Table 9 is that Prune-C is able to outperform Prune-D, whichis surprising since if our goal task is detection, directly pruning away weights that are consideredunimportant for detection should presumably be better than pruning on the pre-trained classiﬁcationmodels.
Revised: Another interesting observation from Table 9 is that Prune-C is able to outperform Prune-D, which is surprising since if our goal task is detection, directly pruning away weights that are considered unimportant for detection should presumably be better than pruning on the pre-trained classiﬁcation models.

Original: We hypothesize that this might be because pruning early in the classiﬁcation stage makes theﬁnal model less prone to being trapped in a bad local minimum caused by inheriting weights from thelarge model.
Revised: We hypothesize that this might be because pruning early in the classiﬁcation stage makes the ﬁnal model less prone to being trapped in a bad local minimum caused by inheriting weights from the large model.

Original: This is in line with our observation that Scratch-E/B, which trains the small models fromscratch starting even earlier at the classiﬁcation stage, can achieve further performance improvement.
Revised: This is in line with our observation that Scratch-E/B, which trains the small models from scratch starting even earlier at the classiﬁcation stage, can achieve further performance improvement.

Original: C Network Pruning as Architecture Search
Revised: C Network Pruning as Architecture Search

Original: Here we provide results that complements Section 4 for non-structured weight pruning [3].
Revised: Here we provide results that complements Section 4 for non-structured weight pruning [2].

Original: Thenetworks and datasets used are the same as those used in Section 4.
Revised: The networks and datasets used are the same as those used in Section 4.

Original: We also discuss the relation withconventional architecture search methods at the end.
Revised: We also discuss the relation with conventional architecture search methods at the end.

Original: Figure 3(left) shows our results on parameter efﬁciency of architectures obtained by non-structuredpruning.
Revised: Figure 3(left) shows our results on parameter efﬁciency of architectures obtained by non-structured pruning.

Original: Here “Uniform Sparsifying” means uniformly sparsifying individual weights in the networkat a ﬁxed probability.
Revised: Here “Uniform Sparsifying” means uniformly sparsifying individual weights in the network at a ﬁxed probability.

Original: It can be seen that pruned architectures are more parameter-efﬁcient thanuniform sparsiﬁed architectures.[Table] [Figure] [Table]
Revised: It can be seen that pruned architectures are more parameter-efﬁcient than uniform sparsiﬁed architectures.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] Figure 3(right) shows our results for architectures obtained by other different design strategies.
Revised: Figure 3(right) shows our results for architectures obtained by other different design strategies.

Original: For“Guided Sparsifying” and “Transferred Guided Sparsifying”, we use the average sparsity patternsof 3 × 3 kernel in each layer stage to design new structures.
Revised: For “Guided Sparsifying” and “Transferred Guided Sparsifying”, we use the average sparsity patterns of 3 × 3 kernel in each layer stage to design new structures.

Original: Similar to Network Slimming, both“Guided Sparsifying” (green) and “Transferred Guided Sparsifying” (brown) are signiﬁcantly moreparameter-efﬁcient than uniform sparsifying (red).
Revised: Similar to Network Slimming, both “Guided Sparsifying” (green) and “Transferred Guided Sparsifying” (brown) are signiﬁcantly more parameter-efﬁcient than uniform sparsifying (red).

Original: Comparison with Traditional Architecture Search Methods.
Revised: Comparison with Traditional Architecture Search Methods.

Original: Conventional techniques for network architecture search include reinforcement learning [21, 22] and evolutionary algorithms [23, 24].
Revised: Conventional techniques for network architecture search include reinforcement learning [20, 21] and evolutionary algorithms [22, 23].

Original: In each iteration, a randomly initialized network is trained and evaluated to guide the search, and thesearch process usually requires thousands of iterations to ﬁnd the goal architecture.
Revised: In each iteration, a randomly initialized network is trained and evaluated to guide the search, and the search process usually requires thousands of iterations to ﬁnd the goal architecture.

Original: In contrast, usingnetwork pruning as architecture search only requires a one-pass training, however the search space isrestricted to the set of all “sub-networks” inside a large network, whereas traditional methods cansearch for more variations, e.g., activation functions or different layer orders.
Revised: In contrast, using network pruning as architecture search only requires a one-pass training, however the search space is restricted to the set of all “sub-networks” inside a large network, whereas traditional methods can search for more variations, e.g., activation functions or different layer orders.

Original: Recently, [25] uses a similar pruning technique to Network Slimming [6] to automate the design of network architectures; [26] prune channels using reinforcement learning and automatically compressesthe architecture.
Revised: Recently, [24] uses a similar pruning technique to Network Slimming [5] to automate the design of network architectures; [25] prune channels using reinforcement learning and automatically compresses the architecture.

Original: On the other hand, in the network architecture search literature, sharing/inheritingtrained parameters [27, 28] has become a popular approach to accelerate the convergence and reducethe training budget during searching, though it would be interesting to investigate whether trainingfrom scratch would sometimes yield better results as we observed in network pruning.
Revised: On the other hand, in the network architecture search literature, sharing/inheriting trained parameters [26, 27] has become a popular approach to accelerate the convergence and reduce the training budget during searching, though it would be interesting to investigate whether training from scratch would sometimes yield better results as we observed in network pruning.

Original: We can seethat these two areas, namely network pruning and architecture search, share many common traits andstart to borrow wisdom from each other.
Revised: We can see that these two areas, namely network pruning and architecture search, share many common traits and start to borrow wisdom from each other.

Original: D Weight Distribution
Revised: D Weight Distribution

Original: [Figure]
Revised: [Figure]

Original: We show the weight distribution of unpruned large models, ﬁne-tuned pruned models and scratchtrained pruned models, for two pruning methods: Network Slimming [6] and non-structured weightlevel pruning [3].
Revised: We show the weight distribution of unpruned large models, ﬁne-tuned pruned models and scratchtrained pruned models, for two pruning methods: Network Slimming [5] and non-structured weight level pruning [2].

Original: We choose DenseNet-40 and CIFAR-10 for visualization and compare the weightdistribution of unpruned models, ﬁne-tuned models and scratch-trained models.
Revised: We choose DenseNet-40 and CIFAR-10 for visualization and compare the weight distribution of unpruned models, ﬁne-tuned models and scratch-trained models.

Original: For Network Slimming, the prune ratio is chosen to be 60%.
Revised: For Network Slimming, the prune ratio is chosen to be 60%.

Original: For non-structured weight level pruning, the prune ratio ischosen to be 80%.
Revised: For non-structured weight level pruning, the prune ratio is chosen to be 80%.

Original: Figure 4 shows our result.
Revised: Figure 4 shows our result.

Original: We can see that the weight distribution of ﬁne-tunedmodels and scratch-trained pruned models are different from the unpruned large models – the weights that are close to zero are much fewer.
Revised: We can see that the weight distribution of ﬁne-tuned models and scratch-trained pruned models are different from the unpruned large models – the weights that are close to zero are much fewer.

Original: This seems to imply that there are less redundant structures inthe found pruned architecture, and support the effect of architecture search for automatic pruningmethods.
Revised: This seems to imply that there are less redundant structures in the found pruned architecture, and support the effect of architecture search for automatic pruning methods.