Original: Rethinking the Value of Network Pruning ∗
Revised: Rethinking the Value of Network Pruning ∗

Original: Anonymous Author s)
Revised: Anonymous Author s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Network pruning is widely used for reducing the heavy computational cost ofdeep models.
Revised: Network pruning is widely used for reducing the heavy computational cost ofdeep models.

Original: A typical pruning algorithm is a three-stage pipeline, i e., training (alarge model), pruning and ﬁne-tuning.
Revised: A typical pruning algorithm is a three-stage pipeline, i e., training (alarge model), pruning and ﬁne-tuning.

Original: In this work, we make a rather surprisingobservation: ﬁne-tuning a pruned model only gives comparable or even worseperformance than training that model with randomly initialized weights.
Revised: In this work, we make a rather surprisingobservation: ﬁne-tuning a pruned model only gives comparable or even worseperformance than training that model with randomly initialized weights.

Original: Ourresults have several implications: 1) training a large, over-parameterized model isnot necessary to obtain an efﬁcient ﬁnal model, 2) learned “important” weightsof the large model are not necessarily useful for the small pruned model, 3) thepruned architecture itself, rather than a set of inherited weights, is what leads to theefﬁciency beneﬁt in the ﬁnal model, which suggests that some pruning algorithmscould be seen as performing network architecture search
Revised: Ourresults have several implications: 1) training a large, over-parameterized model isnot necessary to obtain an efﬁcient ﬁnal model, 2) learned “important” weightsof the large model are not necessarily useful for the small pruned model, 3) thepruned architecture itself, rather than a set of inherited weights, is what leads to theefﬁciency beneﬁt in the ﬁnal model, which suggests that some pruning algorithmscould be seen as performing network architecture search

Original:  Introduction
Revised:  Introduction

Original: Network pruning is a commonly used approach for obtaining an efﬁcient neural network.
Revised: Network pruning is a commonly used approach for obtaining an efﬁcient neural network.

Original: A typicalprocedure of network pruning consists of three stages: 1) train a large, over-parameterized model, 2)prune the unimportant weights according to a certain criterion, and 3) ﬁne-tune the pruned model toregain accuracy.
Revised: A typicalprocedure of network pruning consists of three stages: 1) train a large, over-parameterized model, 2)prune the unimportant weights according to a certain criterion, and 3) ﬁne-tune the pruned model toregain accuracy.

Original: Generally, there are two common beliefs behind this pruning procedure.
Revised: Generally, there are two common beliefs behind this pruning procedure.

Original: First, it isbelieved that training a large network ﬁrst is important [2] and the three-stage pipeline can outperformdirectly training the small model from scratch.
Revised: First, it isbelieved that training a large network ﬁrst is important [2] and the three-stage pipeline can outperformdirectly training the small model from scratch.

Original: Second, both the architectures and the weights of thepruned model are believed to be essential for the ﬁnal efﬁcient model, which is why most existingpruning methods choose to ﬁne-tune the pruned model instead of training it from scratch.
Revised: Second, both the architectures and the weights of thepruned model are believed to be essential for the ﬁnal efﬁcient model, which is why most existingpruning methods choose to ﬁne-tune the pruned model instead of training it from scratch.

Original: Alsobecause of this, how to select the set of important weights is a very active research topic [2, 3, 4, 5].
Revised: Alsobecause of this, how to select the set of important weights is a very active research topic [2, 3, 4, 5].

Original: [Figure]
Revised: [Figure]

Original: In this work, we show that both of the beliefs mentionedabove are not necessarily true.
Revised: In this work, we show that both of the beliefs mentionedabove are not necessarily true.

Original: We make a surprising observation that directly training the target pruned model fromrandom initialization can achieve the same or better performance as the model obtained from the three-stage pipeline.
Revised: We make a surprising observation that directly training the target pruned model fromrandom initialization can achieve the same or better performance as the model obtained from the three-stage pipeline.

Original: This means that, for pruning methods with a predeﬁnedtarget architecture (Figure 1), starting with a large modelis not necessary and one could instead directly train thetarget model from scratch.
Revised: This means that, for pruning methods with a predeﬁnedtarget architecture (Figure 1), starting with a large modelis not necessary and one could instead directly train thetarget model from scratch.

Original: For pruning algorithms withautomatically discovered target architectures, what bringsthe efﬁciency beneﬁt is the obtained architecture, insteadof the inherited weights.
Revised: For pruning algorithms withautomatically discovered target architectures, what bringsthe efﬁciency beneﬁt is the obtained architecture, insteadof the inherited weights.

Original: Our results advocate a rethinkingof existing network pruning algorithms: the preserved “important” weights from the large model are not necessaryfor obtaining an efﬁcient ﬁnal model; instead, the value ofautomatic network pruning methods may lie in identifyingefﬁcient architectures and performing implicit architecture search
Revised: Our results advocate a rethinkingof existing network pruning algorithms: the preserved “important” weights from the large model are not necessaryfor obtaining an efﬁcient ﬁnal model; instead, the value ofautomatic network pruning methods may lie in identifyingefﬁcient architectures and performing implicit architecture search

Original:  Methodology
Revised:  Methodology

Original: Target Pruned Architectures.
Revised: Target Pruned Architectures.

Original: We divide pruning methods by whether the target pruned architectureis determined by either human or the pruning algorithm (see Figure 1).
Revised: We divide pruning methods by whether the target pruned architectureis determined by either human or the pruning algorithm (see Figure 1).

Original: An example of designingpredeﬁned target architecture is specifying how many channels to prune in each layer.
Revised: An example of designingpredeﬁned target architecture is specifying how many channels to prune in each layer.

Original: In contrast,when the target architecture is automatically discovered, the pruning algorithm determines how manychannels to prune in each layer, by comparing the importance of channels across layers.
Revised: In contrast,when the target architecture is automatically discovered, the pruning algorithm determines how manychannels to prune in each layer, by comparing the importance of channels across layers.

Original: Training from scratch.
Revised: Training from scratch.

Original: Because the pruned model requires less computation, it may be unfair totrain the pruned model for the same number of epochs as the large model.
Revised: Because the pruned model requires less computation, it may be unfair totrain the pruned model for the same number of epochs as the large model.

Original: In our experiments, we use Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B to denotetraining for the same amount of computation budget (measured by FLOPs).
Revised: In our experiments, weuse Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B todenote training for the same amount of computation budget (measured by FLOPs).

Original: We use standard traininghyper-parameters and data-augmentation schemes.
Revised: We use standardtraining hyper-parameters and data-augmentation schemes.

Revised: The optimization method used is SGDwith Nesterov momentum.

Original:  Experiments
Revised:  Experiments

Original: In this section we present our experimental results comparing training pruned models from scratchwith ﬁne-tuning.
Revised: In this section we present our experimental results comparing training pruned models from scratchwith ﬁne-tuning.

Original: For method with predeﬁned architectures, we evaluate L 1 -norm based channelpruning method [4].
Revised: For method with predeﬁned architectures, we evaluate L 1 -norm based channelpruning method [4].

Original: For method with automatically discovered target architectures, we use Network Slimming [6].
Revised: For method with automatically discovered target architectures, we use Network Slimming [6].

Revised: The models, datasets and the number of epochs for ﬁne-tuning are the same as thosein the original paper.

Original: More results for other pruning methods and transfer learning can be found in
Revised: More results for other pruning methods and transfer learning can be found in

Original: Appendix B.
Revised: Appendix B.

Original: 3.1 Predeﬁned Target Architectures
Revised: 3.1 Predeﬁned Target Architectures

Original: L 1 -norm based Channel Pruning [4] is one of the earliest work on channel pruning for convolutional networks.
Revised: L 1 -norm based Channel Pruning [4] is one of the earliest work on channel pruning for convolutional networks.

Original: In each layer, a certain percentage of channels with smaller L 1 -norm of its ﬁlterweights will be pruned.
Revised: In each layer, a certain percentage of channels with smaller L 1 -norm of its ﬁlterweights will be pruned.

Original: Table 1 shows our results.
Revised: Table 1 shows our results.

Original: The Pruned Model column shows the list ofpredeﬁned target models.
Revised: The Pruned Model column shows the list ofpredeﬁned target models.

Original: We observe that in each row, scratch-trained models achieve at least thesame level of accuracy as ﬁne-tuned models, with Scratch-B slightly higher than Scratch-E in mostcases.
Revised: We observe that in each row, scratch-trained models achieve at least thesame level of accuracy as ﬁne-tuned models, with Scratch-B slightly higher than Scratch-E in mostcases.

Original: On ImageNet, both Scratch-B models are better than the ﬁne-tuned ones.
Revised: On ImageNet, both Scratch-B models are better than the ﬁne-tuned ones

Original: [Table]
Revised: [Table]

Original: 3.2 Automatically Discovered Target Architectures
Revised: 3.2 Automatically Discovered Target Architectures

Original: [Table]
Revised: [Table]

Original: Network Slimming [6] imposes L 1 -sparsity  on channel-wise scaling factors from Batch Normalization layers [7] during training, and prunes channels with lower scaling factors afterward.
Revised: Network Slimming [6] imposes L 1 -sparsity regularization on channel-wise scaling factors from Batch Normalization layers [7] during training, and prunes channels with lower scaling factorsafterward.

Original: Since thechannel scaling factors are compared across layers, this method produces automatically discoveredtarget architectures.
Revised: Since the channel scaling factors are compared across layers, this method producesautomatically discovered target architectures.

Original: As shown in Table 2, for all networks, the small models trained from scratchcan reach the same accuracy as the ﬁne-tuned models. More speciﬁcally, we found that Scratch-Bconsistently outperforms the ﬁne-tuned models, while Scratch-E is slightly worse but still mostlywithin the standard deviation.
Revised: As shown in Table 2, for all networks, the smallmodels trained from scratch can reach the same accuracy as the ﬁne-tuned models, where Scratch-Bconsistently outperforms the ﬁne-tuned models. Morever, when we extend the standard training oflarge model, the above observation still holds.

Original:  Network Pruning as Architecture Search
Revised:  Network Pruning as Architecture Search

Original: In this section, we demonstrate that the value of automatic network pruning methods actually lies insearching efﬁcient architectures.
Revised: In this section, we demonstrate that the value of automatic network pruning methods actually lies insearching efﬁcient architectures.

Original: We use Network Slimming [6] as an example automatic method.
Revised: We use Network Slimming [6] as an example automatic method.

Original: Parameter Efﬁciency of Pruned Architectures.
Revised: Parameter Efﬁciency of Pruned Architectures.

Original: In Figure 2(left), we compare the parameterefﬁciency of architectures obtained by Network Slimming with a naive predeﬁned pruning strategythat uniformly prunes the same percentage of channels in all layer.
Revised: In Figure 2(left), we compare the parameterefﬁciency of architectures obtained by Network Slimming with a naive predeﬁned pruning strategythat uniformly prunes the same percentage of channels in all layer.

Original: All architectures are trainedfrom random initialization for the same number of epochs.
Revised: All architectures are trained fromrandom initialization for the same number of epochs without sparsity regularization.

Original: We see that the architectures obtained by Network Slimming are more parameter efﬁcient, as they could achieve the same level of accuracyusing 5 × fewer parameters than uniformly pruning architectures.
Revised: We see that thearchitectures obtained by Network Slimming are more parameter efﬁcient, as they could achieve thesame level of accuracy using 5 × fewer parameters than uniformly pruning architectures.

Original: [Figure]
Revised: [Figure]

Original: Generalizable Design Principles from Pruned Architectures.
Revised: Generalizable Design Principles from Pruned Architectures.

Original: Given that the automatically discovered architectures tend to be parameter efﬁcient, here we show a generalizable principle on designingbetter architectures.
Revised: Given that the automatically discovered architectures tend to be parameter efﬁcient, here we show a generalizable principle on designingbetter architectures.

Original: We show two design strategies: 1) “Guided Pruning”: use the average numberof channels in each layer stage (layers with the same feature map size) from pruned architecturesto construct a new set of architectures.
Revised: We show two design strategies: 1) “Guided Pruning”: use the average numberof channels in each layer stage (layers with the same feature map size) from pruned architecturesto construct a new set of architectures.

Original: 2) “Transferred Guided Pruning”: use these patterns from adifferent architecture on a different dataset.
Revised: 2) “Transferred Guided Pruning”: use these patterns from adifferent architecture on a different dataset.

Original: Figure 2(right) shows our results.
Revised: Figure 2(right) shows our results.

Original: Here for “Transferred
Revised: Here for “Transferred

Original: Guided Pruning”
Revised: Guided Pruning”

Original: , we use the patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100.
Revised: , we use the patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100.

Original: We observe that both “Guided Pruning” (green) and “Transferred
Revised: We observe that both “Guided Pruning” (green) and “Transferred

Original: Guided Pruning” (brown) can both perform on par architectures directly pruned on the VGG-19 and
Revised: Guided Pruning” (brown) can both perform on par architectures directly pruned on the VGG-19 and

Original: CIFAR-100 (blue), and are signiﬁcantly better than uniform pruning (red).
Revised: CIFAR-100 (blue), and are signiﬁcantly better than uniform pruning (red).

Original: This means we coulddistill generalizable design principles from pruned architectures.
Revised: This means we coulddistill generalizable design principles from pruned architectures.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: In practice, for methods with predeﬁned target architectures, training from scratch is more computationally efﬁcient and saves us from implementing the pruning procedure and tuning the additionalhyper-parameters.
Revised: In practice, for methods with predeﬁned target architectures, training from scratch is more computationally efﬁcient and saves us from implementing the pruning procedure and tuning the additionalhyper-parameters.

Original: Still, pruning methods are useful when a pretrained large model is already given,in this case ﬁne-tuning is much faster.
Revised: Still, pruning methods are useful when a pretrained large model is already given,in this case ﬁne-tuning is much faster.

Original: Also, obtaining multiple models of different sizes can be donequickly by pruning from a large model by different ratios.
Revised: Also, obtaining multiple models of different sizes can be donequickly by pruning from a large model by different ratios.

Original: In summary, our experiments have shown that training the small pruned model from scratch canalmost always achieve comparable or higher level of accuracy than the model obtained from thetypical “training, pruning and ﬁne-tuning” procedure.
Revised: In summary, our experiments have shown that training the small pruned model from scratch can almostalways achieve comparable or higher level of accuracy than the model obtained from the typical “training, pruning and ﬁne-tuning” procedure.

Original: This changed our understanding about the necessity of over-parameterization, and the  effectiveness of inheriting weights that are considered important by the pruning criteria
Revised: This changed our previous belief that over-parameterization is necessary for obtaining a ﬁnal efﬁcient model, and the understanding on effectiveness of inheriting weights that are considered important by the pruning criteria.

Original: We further demonstrated the value of automatic pruning algorithms could be  ﬁnding efﬁcient architectures and providing architecture design guidelines
Revised: We further demonstrated the value of automatic pruning algorithms could be regarded as ﬁnding efﬁcient architectures and providing architecture design guidelines.

Original: AppendixA Experiment Setups
Revised: AppendixA Experiment Setups

Original: Here we provide additional details on our experiment setups.
Revised: Here we provide additional details on our experiment setups.

Original: Datasets, Network Architectures and Pruning Methods.
Revised: Datasets, Network Architectures and Pruning Methods.

Original: In the network pruning literature, CIFAR10, CIFAR-100 [8], and ImageNet [9] datasets are the de-facto benchmarks, while VGG [10], ResNet[11] and DenseNet [12] are the common network architectures.
Revised: In the network pruning literature, CIFAR10, CIFAR-100 [8], and ImageNet [9] datasets are the de-facto benchmarks, while VGG [10], ResNet[11] and DenseNet [12] are the common network architectures.

Original: We evaluate three pruning methodswith predeﬁned target architectures, [2, 4, 5], and three which automatically discover target models,[3, 6, 13].
Revised: We evaluate three pruning methodswith predeﬁned target architectures, [2, 4, 5], and three which automatically discover target models,[3, 6, 13].

Original: For the ﬁrst ﬁve methods, we evaluate using the same (target model, dataset) pairs aspresented in the original paper to keep our results comparable.
Revised: For the ﬁrst ﬁve methods, we evaluate using the same (target model, dataset) pairs aspresented in the original paper to keep our results comparable.

Original: For the last one [3], we use theaforementioned architectures instead, since the ones in the original paper are no longer state-of-theart.
Revised: For the last one [3], we use theaforementioned architectures instead, since the ones in the original paper are no longer state-of-theart.

Original: On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean andstandard deviation of the accuracy.
Revised: On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean andstandard deviation of the accuracy.

Original: Implementation.
Revised: Implementation.

Original: In order to keep our setup as close to the original paper as possible, we use thefollowing protocols: 1) If a previous pruning method’s training setup is publicly available, e.g.[6] and [13], we adopt the original implementation; 2) Otherwise, for simpler pruning methods,e.g., [3, 4], we re-implement the three-stage pruning procedure and achieve similar results to theoriginal paper; 3)
Revised: In order to keep our setup as close to the original paper as possible, we use thefollowing protocols: 1) If a previous pruning method’s training setup is publicly available, e.g.[6] and [13], we adopt the original implementation; 2) Otherwise, for simpler pruning methods,e.g., [3, 4], we re-implement the three-stage pruning procedure and achieve similar results to theoriginal paper; 3)

Original: For the remaining two methods [2, 5], the pruned models are publicly availablebut without the training setup, thus we choose to re-train both large and small target models fromscratch.
Revised: For the remaining two methods [2, 5], the pruned models are publicly availablebut without the training setup, thus we choose to re-train both large and small target models fromscratch.

Original: Interestingly, the accuracy of our re-trained large model is higher than what is reported in theoriginal paper 2 .
Revised: Interestingly, the accuracy of our re-trained large model is higher than what is reported in theoriginal paper 2 .

Original: In this case, to accommodate the effects of different frameworks and training setups,we report the relative accuracy drop from the unpruned large model.
Revised: In this case, to accommodate the effects of different frameworks and training setups,we report the relative accuracy drop from the unpruned large model.

Original: For random weight initialization, we adopt the scheme proposed in [16].
Revised: For random weight initialization, we adopt the scheme proposed in [16].

Original: For results of modelsﬁne-tuned from inherited weights, we either use the released models from original papers (for case 3above) or follow the common practice of ﬁne-tuning the model using the lowest learning rate whentraining the large model [4, 5].
Revised: For results of modelsﬁne-tuned from inherited weights, we either use the released models from original papers (for case 3above) or follow the common practice of ﬁne-tuning the model using the lowest learning rate whentraining the large model [4, 5].

Original: The code to reproduce the results and trained ImageNet models arepublicly available in the following anonymous link: https://drive.google.com/open?id=1HB_1FphsWtbuMAdgHbODSLdAMnj3B6TM .
Revised: The code to reproduce the results and trained ImageNet models arepublicly available in the following anonymous link: https://drive.google.com/open?id=1HB_1FphsWtbuMAdgHbODSLdAMnj3B6TM .

Original: B Additional Experiment Results
Revised: B Additional Experiment Results

Original: The results of two pruning methods are already shown in the extended abstract, here in appendix weprovide results of the remaining four pruning methods.
Revised: The results of two pruning methods are already shown in the extended abstract, here in appendix weprovide results of the remaining four pruning methods.

Original: We also include an experiment on transferlearning from image classiﬁcation to object detection.
Revised: We also include an experiment on transferlearning from image classiﬁcation to object detection.

Original: B.1 Predeﬁned Target Architectures
Revised: B.1 Predeﬁned Target Architectures

Original: ThiNet [2] greedily prunes the channel that has the smallest effect on the next layer’s activationvalues.
Revised: ThiNet [2] greedily prunes the channel that has the smallest effect on the next layer’s activationvalues.

Original: As shown in Table 3, for VGG-16 and ResNet-50, both Scratch-E and Scratch-B can almostalways achieve better performance than the ﬁne-tuned model, often by a signiﬁcant margin.
Revised: As shown in Table 3, for VGG-16 and ResNet-50, both Scratch-E and Scratch-B can almostalways achieve better performance than the ﬁne-tuned model, often by a signiﬁcant margin.

Original: The onlyexception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG-16(FLOPs reduced by 15 × ), and as a result, drastically reducing the training budget for Scratch-E. Thetraining budget of Scratch-B for this model is also 7 times smaller than the original large model, yetit can achieve the same level of accuracy as the ﬁne-tuned model.
Revised: The onlyexception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG-16(FLOPs reduced by 15 × ), and as a result, drastically reducing the training budget for Scratch-E. Thetraining budget of Scratch-B for this model is also 7 times smaller than the original large model, yetit can achieve the same level of accuracy as the ﬁne-tuned model.

Original: Regression based Feature Reconstruction [5] prunes channels by minimizing the feature mapreconstruction error of the next layer.
Revised: Regression based Feature Reconstruction [5] prunes channels by minimizing the feature mapreconstruction error of the next layer.

Original: Different from ThiNet [2], this optimization problem is solvedby LASSO regression.
Revised: Different from ThiNet [2], this optimization problem is solvedby LASSO regression.

Original: Results are shown in Table 4.
Revised: Results are shown in Table 4.

Original: Again, in terms of relative accuracy drop fromthe large models, scratch-trained models are better than the ﬁne-tuned models.
Revised: Again, in terms of relative accuracy drop fromthe large models, scratch-trained models are better than the ﬁne-tuned models.

Original: In summary, for pruning methods with predeﬁned target architectures, training the small modelsfor the same number of epochs as the large model (Scratch-E), is often enough to achieve thesame accuracy as models output by the three-stage pipeline.
Revised: In summary, for pruning methods with predeﬁned target architectures, training the small modelsfor the same number of epochs as the large model (Scratch-E), is often enough to achieve thesame accuracy as models output by the three-stage pipeline.

Original: Combined with the fact that the targetarchitecture is predeﬁned, in practice one would prefer to train the small model from scratch directly.
Revised: Combined with the fact that the targetarchitecture is predeﬁned, in practice one would prefer to train the small model from scratch directly.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Similar to Table 3, we compare relative accuracy drop from unpruned large models.
Revised: Similar to Table 3, we compare relative accuracy drop from unpruned large models.

Original: Moreover, when provided with the same amount of computation budget (measured by FLOPs) as thelarge model, scratch-trained models can even lead to better performance than the ﬁne-tuned models.
Revised: Moreover, when provided with the same amount of computation budget (measured by FLOPs) as thelarge model, scratch-trained models can even lead to better performance than the ﬁne-tuned models.

Original: B.2 Automatically Discovered Target Architectures
Revised: B.2 Automatically Discovered Target Architectures

Original: Sparse Structure Selection [13] also uses sparsiﬁed scaling factors  to prune structures, and can beseen as a generalization of Network Slimming.
Revised: Sparse Structure Selection [13] also imposes sparsity regularization on the scaling factors duringtraining to prune structures, and can be seen as a generalization of Network Slimming.

Original: Other than channels, pruning can be on residualblocks in ResNet or groups in ResNeXt [17].
Revised: Other thanchannels, pruning can be on residual blocks in ResNet or groups in ResNeXt [17].

Original: We examine residual blocks pruning, where ResNet-50are pruned to be ResNet-41, ResNet-32 and ResNet-26.
Revised: We examineresidual blocks pruning, where ResNet-50 are pruned to be ResNet-41, ResNet-32 and ResNet-26.

Original: Table 5 shows our results.
Revised: Table 7 shows our results.

Original: On average Scratch-E outperforms pruned models, and for all models Scratch-B is better than both.
Revised: On average Scratch-E outperforms pruned models, and for all models Scratch-B is better than both.

Original: [Table]
Revised: [Table]

Original: Non-structured Weight Pruning [3] prunes individual weights that have small magnitudes.
Revised: Non-structured Weight Pruning [3] prunes individual weights that have small magnitudes.

Original: Thispruning granularity leaves the weight matrices sparse, hence it is commonly referred to as nonstructured weight pruning.
Revised: Thispruning granularity leaves the weight matrices sparse, hence it is commonly referred to as nonstructured weight pruning.

Original: Because all the network architectures we evaluated are fully-convolutional(except for the last fully-connected layer), for simplicity, we only prune weights in convolution layershere.
Revised: Because all the network architectures we evaluated are fully-convolutional(except for the last fully-connected layer), for simplicity, we only prune weights in convolution layershere.

Original: Before training the pruned sparse model from scratch, we re-scale the standard deviation ofthe Gaussian distribution for weight initialization, based on how many non-zero weights remain inthis layer.
Revised: Before training the pruned sparse model from scratch, we re-scale the standard deviation ofthe Gaussian distribution for weight initialization, based on how many non-zero weights remain inthis layer.

Original: This is to keep a constant scale of backward gradient signal [16].
Revised: This is to keep a constant scale of backward gradient signal [16].

Original: As shown in Table 6, datasets, E sometimes falls short of the ﬁne-tuned results but Scratch-B is ableto perform at least  par with the latter
Revised: As shown in Table 6,on CIFAR datasets, Scratch-E sometimes falls short of the ﬁne-tuned results, but Scratch-B is able to perform at least on par with the latter.

Original: ImageNet we note that sometimes even Scratch-B is slightly worse than ﬁne-tuned result.
Revised: On ImageNet, we note that sometimes even Scratch-Bis slightly worse than ﬁne-tuned result.

Original: This is the only case where Scratch-B does not achievecomparable accuracy in our attempts.
Revised: This is the only case where Scratch-B does not achievecomparable accuracy in our attempts.

Original: We hypothesize this could be due to the task complexity of
Revised: We hypothesize this could be due to the task complexity of

Original: ImageNet and the ﬁne pruning granularity.
Revised: ImageNet and the ﬁne pruning granularity.

Original: [Table]
Revised: [Table]

Revised: Effects of Sparsity Regularization.

Revised: Some methods [6, 13] use sparsity regularization during thetraining of large, over-parameterized models, to smooth the following pruning process

Revised: [Table]

Revised: and in ﬁnetuning, no such sparsity is imposed.

Revised: In all of our experiments presented above

Revised: [Table]

Revised: no sparsity is used fortraining the pruned models from scratch.

Revised: Here we analyze the effects of using sparsity regularization(or not) for Network Slimming [6].

Revised: Table 7 shows the results when all training procedures are withsparsity regularization.

Revised: We can see that Scratch-B are still able to be on par with the ﬁne-tuned models.

Revised: Table 8 shows the results when we do not use sparsity regularization in any training procedures,including large model training.

Revised: We can see that when no sparsity is induced, Scratch-B can alsoconsistently achieve comparable results with the ﬁne-tuned models.

Revised: [Table]

Revised: [Table]

Original: B.3 Transfer Learning to Object Detection
Revised: B.3 Transfer Learning to Object Detection

Original: We have shown that the small pruned model can be trained from scratch to match the accuracy of theﬁne-tuned model in classiﬁcation tasks.
Revised: We have shown that the small pruned model can be trained from scratch to match the accuracy of theﬁne-tuned model in classiﬁcation tasks.

Original: To see whether this phenomenon would also hold for transferlearning to other vision tasks, we evaluate the L 1 -norm based pruning method [4] on the PASCAL
Revised: To see whether this phenomenon would also hold for transferlearning to other vision tasks, we evaluate the L 1 -norm based pruning method [4] on the PASCAL

Original: VOC object detection task, using Faster-RCNN [18].
Revised: VOC object detection task, using Faster-RCNN [18] [Table]

Original: Object detection frameworks usually require transferring model weights pre-trained on ImageNetclassiﬁcation, and one can perform pruning either before or after the weight transfer.
Revised: Object detection frameworks usually require transferring model weights pre-trained on ImageNetclassiﬁcation, and one can perform pruning either before or after the weight transfer.

Original: More speciﬁcally,the former could be described as “train on classiﬁcation, prune on classiﬁcation, ﬁne-tune onclassiﬁcation, transfer to detection”, while the latter is “train on classiﬁcation, transfer to detection,prune on detection, ﬁne-tune on detection”.
Revised: More speciﬁcally,the former could be described as “train on classiﬁcation, prune on classiﬁcation, ﬁne-tune onclassiﬁcation, transfer to detection”, while the latter is “train on classiﬁcation, transfer to detection,prune on detection, ﬁne-tune on detection”.

Original: We call these two approaches Prune-C (classiﬁcation)and Prune-D (detection) respectively, and report the results in Table 7.
Revised: We call these two approaches Prune-C (classiﬁcation)and Prune-D (detection) respectively, and report the results in Table 9.

Original: With a slight abuse of notation,here Scratch-E/B denotes "train the small model on classiﬁcation, transfer to detection", and isdifferent from the setup of detection without ImageNet pre-training as in [19].
Revised: With a slight abuse of notation,here Scratch-E/B denotes "train the small model on classiﬁcation, transfer to detection", and isdifferent from the setup of detection without ImageNet pre-training as in [19].

Original: [Table]
Revised: [Table]

Original: For this experiment, we adopt the code and default hyper-parameters from [20], and use PASCAL
Revised: For this experiment, we adopt the code and default hyper-parameters from [20], and use PASCAL

Original: VOC 07 trainval/test set as our training/test set.
Revised: VOC 07 trainval/test set as our training/test set.

Original: For backbone networks, we evaluate ResNet-34-Aand ResNet-34-B from the L 1 -norm based channel pruning [4], which are pruned from ResNet34.
Revised: For backbone networks, we evaluate ResNet-34-Aand ResNet-34-B from the L 1 -norm based channel pruning [4], which are pruned from ResNet34.

Original: Table 7 shows our result, and we can see that the model trained from scratch can surpass theperformance of ﬁne-tuned models under the transfer setting.
Revised: Table 9 shows our result, and we can see that the model trained from scratch can surpass theperformance of ﬁne-tuned models under the transfer setting.

Original: Another interesting observation from Table 7 is that Prune-C is able to outperform Prune-D, whichis surprising since if our goal task is detection, directly pruning away weights that are considered unimportant for detection should presumably be better than pruning on the pre-trained classiﬁcationmodels.
Revised: Another interesting observation from Table 9 is that Prune-C is able to outperform Prune-D, whichis surprising since if our goal task is detection, directly pruning away weights that are consideredunimportant for detection should presumably be better than pruning on the pre-trained classiﬁcationmodels.

Original: We hypothesize that this might be because pruning early in the classiﬁcation stage makes theﬁnal model less prone to being trapped in a bad local minimum caused by inheriting weights from thelarge model.
Revised: We hypothesize that this might be because pruning early in the classiﬁcation stage makes theﬁnal model less prone to being trapped in a bad local minimum caused by inheriting weights from thelarge model.

Original: This is in line with our observation that Scratch-E/B, which trains the small models fromscratch starting even earlier at the classiﬁcation stage, can achieve further performance improvement.
Revised: This is in line with our observation that Scratch-E/B, which trains the small models fromscratch starting even earlier at the classiﬁcation stage, can achieve further performance improvement.

Original: C Network Pruning as Architecture Search
Revised: C Network Pruning as Architecture Search

Original: Here we provide results that complements Section 4 for non-structured weight pruning [3].
Revised: Here we provide results that complements Section 4 for non-structured weight pruning [3].

Original: Thenetworks and datasets used are the same as those used in Section 4.
Revised: Thenetworks and datasets used are the same as those used in Section 4.

Original: We also discuss the relation withconventional architecture search methods at the end.
Revised: We also discuss the relation withconventional architecture search methods at the end.

Original: Figure 3(left) shows our results on parameter efﬁciency of architectures obtained by non-structuredpruning.
Revised: Figure 3(left) shows our results on parameter efﬁciency of architectures obtained by non-structuredpruning.

Original: Here “Uniform Sparsifying” means uniformly sparsifying individual weights in the networkat a ﬁxed probability.
Revised: Here “Uniform Sparsifying” means uniformly sparsifying individual weights in the networkat a ﬁxed probability.

Original: It can be seen that pruned architectures are more parameter-efﬁcient thanuniform sparsiﬁed architectures. 
Revised: It can be seen that pruned architectures are more parameter-efﬁcient thanuniform sparsiﬁed architectures. [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original:  Figure 3(right) shows our results for architectures obtained by other different design strategies.
Revised: [Table] [Figure] [Table] Figure 3(right) shows our results for architectures obtained by other different design strategies.

Original: For“Guided Sparsifying” and “Transferred Guided Sparsifying”, we use the average sparsity patternsof 3 × 3 kernel in each layer stage to design new structures.
Revised: For“Guided Sparsifying” and “Transferred Guided Sparsifying”, we use the average sparsity patternsof 3 × 3 kernel in each layer stage to design new structures.

Original: Similar to Network Slimming, both“Guided Sparsifying” (green) and “Transferred Guided Sparsifying” (brown) are signiﬁcantly moreparameter-efﬁcient than uniform sparsifying (red).
Revised: Similar to Network Slimming, both“Guided Sparsifying” (green) and “Transferred Guided Sparsifying” (brown) are signiﬁcantly moreparameter-efﬁcient than uniform sparsifying (red).

Original: Comparison with Traditional Architecture Search Methods.
Revised: Comparison with Traditional Architecture Search Methods.

Original: Conventional techniques for network architecture search include reinforcement learning [21, 22] and evolutionary algorithms [23, 24].
Revised: Conventional techniques for network architecture search include reinforcement learning [21, 22] and evolutionary algorithms [23, 24].

Original: In each iteration, a randomly initialized network is trained and evaluated to guide the search, and thesearch process usually requires thousands of iterations to ﬁnd the goal architecture.
Revised: In each iteration, a randomly initialized network is trained and evaluated to guide the search, and thesearch process usually requires thousands of iterations to ﬁnd the goal architecture.

Original: In contrast, usingnetwork pruning as architecture search only requires a one-pass training, however the search space isrestricted to the set of all “sub-networks” inside a large network, whereas traditional methods cansearch for more variations, e.g., activation functions or different layer orders.
Revised: In contrast, usingnetwork pruning as architecture search only requires a one-pass training, however the search space isrestricted to the set of all “sub-networks” inside a large network, whereas traditional methods cansearch for more variations, e.g., activation functions or different layer orders.

Original: Recently, [25] uses a similar pruning technique to Network Slimming [6] to automate the design of network architectures; [26] prune channels using reinforcement learning and automatically compressesthe architecture.
Revised: Recently, [25] uses a similar pruning technique to Network Slimming [6] to automate the design of network architectures; [26] prune channels using reinforcement learning and automatically compressesthe architecture.

Original: On the other hand, in the network architecture search literature, sharing/inheritingtrained parameters [27, 28] has become a popular approach to accelerate the convergence and reducethe training budget during searching, though it would be interesting to investigate whether trainingfrom scratch would sometimes yield better results as we observed in network pruning.
Revised: On the other hand, in the network architecture search literature, sharing/inheritingtrained parameters [27, 28] has become a popular approach to accelerate the convergence and reducethe training budget during searching, though it would be interesting to investigate whether trainingfrom scratch would sometimes yield better results as we observed in network pruning.

Original: We can seethat these two areas, namely network pruning and architecture search, share many common traits andstart to borrow wisdom from each other. 
Revised: We can seethat these two areas, namely network pruning and architecture search, share many common traits andstart to borrow wisdom from each other. D Weight Distribution

Revised: [Figure]

Revised: We show the weight distribution of unpruned large models, ﬁne-tuned pruned models and scratchtrained pruned models, for two pruning methods: Network Slimming [6] and non-structured weightlevel pruning [3].

Revised: We choose DenseNet-40 and CIFAR-10 for visualization and compare the weightdistribution of unpruned models, ﬁne-tuned models and scratch-trained models.

Revised: For Network Slimming, the prune ratio is chosen to be 60%.

Revised: For non-structured weight level pruning, the prune ratio ischosen to be 80%.

Revised: Figure 4 shows our result.

Revised: We can see that the weight distribution of ﬁne-tunedmodels and scratch-trained pruned models are different from the unpruned large models – the weights that are close to zero are much fewer.

Revised: This seems to imply that there are less redundant structures inthe found pruned architecture, and support the effect of architecture search for automatic pruningmethods.