Original: Rethinking the Value of Network Pruning ∗
Revised: Rethinking the Value of Network Pruning ∗

Original: Anonymous Author(s)
Revised: Anonymous Author s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Network pruning is widely used for reducing the heavy computational cost ofdeep models.
Revised: Network pruning is widely used for reducing the heavy computational cost ofdeep models.

Original: A typical pruning algorithm consists of a three-stage pipeline, i.e.,training (a large model), pruning and ﬁne-tuning.
Revised: A typical pruning algorithm is a three-stage pipeline, i e., training (alarge model), pruning and ﬁne-tuning.

Original: In this work, we make a rathersurprising observation: ﬁne-tuning a pruned model only gives comparable or evenworse performance than training that model with randomly initialized weights.
Revised: In this work, we make a rather surprisingobservation: ﬁne-tuning a pruned model only gives comparable or even worseperformance than training that model with randomly initialized weights.

Original: Ourresults have several implications: 1) training a large, over-parameterized model isnot necessary to obtain an efﬁcient ﬁnal model, 2) learned “important” weightsof the large model are not necessarily useful for the small pruned model, 3) thepruned architecture itself, rather than a set of inherited weights, is what leads to theefﬁciency beneﬁt in the ﬁnal model, which suggests that some pruning algorithmscould be seen as performing network architecture search.
Revised: Ourresults have several implications: 1) training a large, over-parameterized model isnot necessary to obtain an efﬁcient ﬁnal model, 2) learned “important” weightsof the large model are not necessarily useful for the small pruned model, 3) thepruned architecture itself, rather than a set of inherited weights, is what leads to theefﬁciency beneﬁt in the ﬁnal model, which suggests that some pruning algorithmscould be seen as performing network architecture search

Original:  Introduction
Revised:  Introduction

Original: Network pruning is a commonly used approach for obtaining an efﬁcient neural network.
Revised: Network pruning is a commonly used approach for obtaining an efﬁcient neural network.

Original: A typicalprocedure of network pruning consists of three stages: 1) train a large, over-parameterized model, 2)prune the unimportant weights according to a certain criterion, and 3) ﬁne-tune the pruned model toregain accuracy.
Revised: A typicalprocedure of network pruning consists of three stages: 1) train a large, over-parameterized model, 2)prune the unimportant weights according to a certain criterion, and 3) ﬁne-tune the pruned model toregain accuracy.

Original: Generally, there are two common beliefs behind this pruning procedure.
Revised: Generally, there are two common beliefs behind this pruning procedure.

Original: First, it isbelieved that training a large network ﬁrst is important [2] and the three-stage pipeline can outperformdirectly training the small model from scratch.
Revised: First, it isbelieved that training a large network ﬁrst is important [2] and the three-stage pipeline can outperformdirectly training the small model from scratch.

Original: Second, both the architectures and the weights of thepruned model are believed to be essential for the ﬁnal efﬁcient model, which is why most existingpruning methods choose to ﬁne-tune the pruned model instead of training it from scratch.
Revised: Second, both the architectures and the weights of thepruned model are believed to be essential for the ﬁnal efﬁcient model, which is why most existingpruning methods choose to ﬁne-tune the pruned model instead of training it from scratch.

Original: Alsobecause of this, how to select the set of important weights is a very active research topic [2, 3, 4, 5].
Revised: Alsobecause of this, how to select the set of important weights is a very active research topic [2, 3, 4, 5].

Original: [Figure]
Revised: [Figure]

Original: In this work, we show that both of the beliefs mentionedabove are not necessarily true.
Revised: In this work, we show that both of the beliefs mentionedabove are not necessarily true.

Original: We make a surprising observation that directly training the target pruned model fromrandom initialization can achieve the same or better performance as the model obtained from the three-stage pipeline.
Revised: We make a surprising observation that directly training the target pruned model fromrandom initialization can achieve the same or better performance as the model obtained from the three-stage pipeline.

Original: This means that, for pruning methods with a predeﬁnedtarget architecture (Figure 1), starting with a large modelis not necessary and one could instead directly train thetarget model from scratch.
Revised: This means that, for pruning methods with a predeﬁnedtarget architecture (Figure 1), starting with a large modelis not necessary and one could instead directly train thetarget model from scratch.

Original: For pruning algorithms withautomatically discovered target architectures, what bringsthe efﬁciency beneﬁt is the obtained architecture, insteadof the inherited weights.
Revised: For pruning algorithms withautomatically discovered target architectures, what bringsthe efﬁciency beneﬁt is the obtained architecture, insteadof the inherited weights.

Original: Our results advocate a rethinkingof existing network pruning algorithms: the preserved “important” weights from the large model are not necessaryfor obtaining an efﬁcient ﬁnal model; instead, the value ofautomatic network pruning methods may lie in identifyingefﬁcient architectures and performing implicit architecture search
Revised: Our results advocate a rethinkingof existing network pruning algorithms: the preserved “important” weights from the large model are not necessaryfor obtaining an efﬁcient ﬁnal model; instead, the value ofautomatic network pruning methods may lie in identifyingefﬁcient architectures and performing implicit architecture search

Original:  Methodology
Revised:  Methodology

Original: Target Pruned Architectures.
Revised: Target Pruned Architectures.

Original: We divide pruning methods by whether the target pruned architectureis determined by either human or the pruning algorithm (see Figure 1).
Revised: We divide pruning methods by whether the target pruned architectureis determined by either human or the pruning algorithm (see Figure 1).

Original: An example of designingpredeﬁned target architecture is specifying how many channels to prune in each layer.
Revised: An example of designingpredeﬁned target architecture is specifying how many channels to prune in each layer.

Original: In contrast,when the target architecture is automatically discovered, the pruning algorithm determines how manychannels to prune in each layer, by comparing the importance of channels across layers.
Revised: In contrast,when the target architecture is automatically discovered, the pruning algorithm determines how manychannels to prune in each layer, by comparing the importance of channels across layers.

Original: Training from scratch.
Revised: Training from scratch.

Original: Because the pruned model requires less computation, it may be unfair totrain the pruned model for the same number of epochs as the large model.
Revised: Because the pruned model requires less computation, it may be unfair totrain the pruned model for the same number of epochs as the large model.

Original: In our experiments, we use
Revised: In our experiments, we use

Original: Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B to denotetraining for the same amount of computation budget (measured by FLOPs).
Revised: Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B to denotetraining for the same amount of computation budget (measured by FLOPs).

Original: We use standard traininghyper-parameters and data-augmentation schemes.
Revised: We use standard traininghyper-parameters and data-augmentation schemes.

Original:  Experiments
Revised:  Experiments

Original: In this section we present our experimental results comparing training pruned models from scratchwith ﬁne-tuning.
Revised: In this section we present our experimental results comparing training pruned models from scratchwith ﬁne-tuning.

Original: For method with predeﬁned architectures, we evaluate L 1 -norm based channelpruning method [4].
Revised: For method with predeﬁned architectures, we evaluate L 1 -norm based channelpruning method [4].

Original: For method with automatically discovered target architectures, we use Network Slimming [6].
Revised: For method with automatically discovered target architectures, we use Network Slimming [6].

Original: More results for other pruning methods and transfer learning can be found in
Revised: More results for other pruning methods and transfer learning can be found in

Original: Appendix B.
Revised: Appendix B.

Original: 3.1 Predeﬁned Target Architectures
Revised: 3.1 Predeﬁned Target Architectures

Original: L 1 -norm based Channel Pruning [4] is one of the earliest work on channel pruning for convolutional networks.
Revised: L 1 -norm based Channel Pruning [4] is one of the earliest work on channel pruning for convolutional networks.

Original: In each layer, a certain percentage of channels with smaller L 1 -norm of its ﬁlterweights will be pruned.
Revised: In each layer, a certain percentage of channels with smaller L 1 -norm of its ﬁlterweights will be pruned.

Original: Table 1 shows our results.
Revised: Table 1 shows our results.

Original: The Pruned Model column shows the list ofpredeﬁned target models.
Revised: The Pruned Model column shows the list ofpredeﬁned target models.

Original: We observe that in each row, scratch-trained models achieve at least thesame level of accuracy as ﬁne-tuned models, with Scratch-B slightly higher than Scratch-E in mostcases.
Revised: We observe that in each row, scratch-trained models achieve at least thesame level of accuracy as ﬁne-tuned models, with Scratch-B slightly higher than Scratch-E in mostcases.

Original: On ImageNet, both Scratch-B models are better than the ﬁne-tuned ones.
Revised: On ImageNet, both Scratch-B models are better than the ﬁne-tuned ones.

Original: [Table]
Revised: [Table]

Original: 3.2 Automatically Discovered Target Architectures
Revised: 3.2 Automatically Discovered Target Architectures

Original: [Table]
Revised: [Table]

Original: Network Slimming [6] imposes L 1 -sparsity on channel-wise scaling factors from Batch Normalization layers [7] during training, and prunes channels with lower scaling factors afterward.
Revised: Network Slimming [6] imposes L 1 -sparsity on channel-wise scaling factors from Batch Normalization layers [7] during training, and prunes channels with lower scaling factors afterward.

Original: Since thechannel scaling factors are compared across layers, this method produces automatically discoveredtarget architectures.
Revised: Since thechannel scaling factors are compared across layers, this method produces automatically discoveredtarget architectures.

Original: As shown in Table 2, for all networks, the small models trained from scratchcan reach the same accuracy as the ﬁne-tuned models.
Revised: As shown in Table 2, for all networks, the small models trained from scratchcan reach the same accuracy as the ﬁne-tuned models.

Original: More speciﬁcally, we found that Scratch-Bconsistently outperforms the ﬁne-tuned models, while Scratch-E is slightly worse but still mostlywithin the standard deviation.
Revised: More speciﬁcally, we found that Scratch-Bconsistently outperforms the ﬁne-tuned models, while Scratch-E is slightly worse but still mostlywithin the standard deviation.

Original:  Network Pruning as Architecture Search
Revised:  Network Pruning as Architecture Search

Original: In this section, we demonstrate that the value of automatic network pruning methods actually lies insearching efﬁcient architectures.
Revised: In this section, we demonstrate that the value of automatic network pruning methods actually lies insearching efﬁcient architectures.

Original: We use Network Slimming [6] as an example automatic method.
Revised: We use Network Slimming [6] as an example automatic method.

Original: Parameter Efﬁciency of Pruned Architectures.
Revised: Parameter Efﬁciency of Pruned Architectures.

Original: In Figure 2(left), we compare the parameterefﬁciency of architectures obtained by Network Slimming with a naive predeﬁned pruning strategythat uniformly prunes the same percentage of channels in all layer.
Revised: In Figure 2(left), we compare the parameterefﬁciency of architectures obtained by Network Slimming with a naive predeﬁned pruning strategythat uniformly prunes the same percentage of channels in all layer.

Original: All architectures are trainedfrom random initialization for the same number of epochs.
Revised: All architectures are trainedfrom random initialization for the same number of epochs.

Original: We see that the architectures obtained by
Revised: We see that the architectures obtained by

Original: Network Slimming are more parameter efﬁcient, as they could achieve the same level of accuracyusing 5 × fewer parameters than uniformly pruning architectures.
Revised: Network Slimming are more parameter efﬁcient, as they could achieve the same level of accuracyusing 5 × fewer parameters than uniformly pruning architectures.

Original: [Figure]
Revised: [Figure]

Original: Generalizable Design Principles from Pruned Architectures.
Revised: Generalizable Design Principles from Pruned Architectures.

Original: Given that the automatically discovered architectures tend to be parameter efﬁcient, here we show a generalizable principle on designingbetter architectures.
Revised: Given that the automatically discovered architectures tend to be parameter efﬁcient, here we show a generalizable principle on designingbetter architectures.

Original: We show two design strategies: 1) “Guided Pruning”: use the average numberof channels in each layer stage (layers with the same feature map size) from pruned architecturesto construct a new set of architectures.
Revised: We show two design strategies: 1) “Guided Pruning”: use the average numberof channels in each layer stage (layers with the same feature map size) from pruned architecturesto construct a new set of architectures.

Original: 2) “Transferred Guided Pruning”: use these patterns from adifferent architecture on a different dataset.
Revised: 2) “Transferred Guided Pruning”: use these patterns from adifferent architecture on a different dataset.

Original: Figure 2(right) shows our results.
Revised: Figure 2(right) shows our results.

Original: Here for “Transferred
Revised: Here for “Transferred

Original: Guided Pruning”
Revised: Guided Pruning”

Original: , we use the patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100.
Revised: , we use the patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100.

Original: We observe that both “Guided Pruning” (green) and “Transferred
Revised: We observe that both “Guided Pruning” (green) and “Transferred

Original: Guided Pruning” (brown) can both perform on par architectures directly pruned on the VGG-19 and
Revised: Guided Pruning” (brown) can both perform on par architectures directly pruned on the VGG-19 and

Original: CIFAR-100 (blue), and are signiﬁcantly better than uniform pruning (red).
Revised: CIFAR-100 (blue), and are signiﬁcantly better than uniform pruning (red).

Original: This means we coulddistill generalizable design principles from pruned architectures.
Revised: This means we coulddistill generalizable design principles from pruned architectures.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: In practice, for methods with predeﬁned target architectures, training from scratch is more computationally efﬁcient and saves us from implementing the pruning procedure and tuning the additionalhyper-parameters.
Revised: In practice, for methods with predeﬁned target architectures, training from scratch is more computationally efﬁcient and saves us from implementing the pruning procedure and tuning the additionalhyper-parameters.

Original: Still, pruning methods are useful when a pretrained large model is already given,in this case ﬁne-tuning is much faster.
Revised: Still, pruning methods are useful when a pretrained large model is already given,in this case ﬁne-tuning is much faster.

Original: Also, obtaining multiple models of different sizes can be donequickly by pruning from a large model by different ratios.
Revised: Also, obtaining multiple models of different sizes can be donequickly by pruning from a large model by different ratios.

Original: In summary, our experiments have shown that training the small pruned model from scratch canalmost always achieve comparable or higher level of accuracy than the model obtained from thetypical “training, pruning and ﬁne-tuning” procedure.
Revised: In summary, our experiments have shown that training the small pruned model from scratch canalmost always achieve comparable or higher level of accuracy than the model obtained from thetypical “training, pruning and ﬁne-tuning” procedure.

Original: This changed our understanding about the necessity of over-parameterization, and the effectiveness of inheriting weights that are considered important by the pruning criteria
Revised: This changed our understanding about the necessity of over-parameterization, and the effectiveness of inheriting weights that are considered important by the pruning criteria

Original: We further demonstrated the value of automatic pruning algorithms could be ﬁnding efﬁcient architectures and providing architecture design guidelines
Revised: We further demonstrated the value of automatic pruning algorithms could be ﬁnding efﬁcient architectures and providing architecture design guidelines

Original: AppendixA Experiment Setups
Revised: AppendixA Experiment Setups

Original: Here we provide additional details on our experiment setups.
Revised: Here we provide additional details on our experiment setups.

Original: Datasets, Network Architectures and Pruning Methods.
Revised: Datasets, Network Architectures and Pruning Methods.

Original: In the network pruning literature, CIFAR10, CIFAR-100 [8], and ImageNet [9] datasets are the de-facto benchmarks, while VGG [10], ResNet[11] and DenseNet [12] are the common network architectures.
Revised: In the network pruning literature, CIFAR10, CIFAR-100 [8], and ImageNet [9] datasets are the de-facto benchmarks, while VGG [10], ResNet[11] and DenseNet [12] are the common network architectures.

Original: We evaluate three pruning methodswith predeﬁned target architectures, [2, 4, 5], and three which automatically discover target models,[3, 6, 13].
Revised: We evaluate three pruning methodswith predeﬁned target architectures, [2, 4, 5], and three which automatically discover target models,[3, 6, 13].

Original: For the ﬁrst ﬁve methods, we evaluate using the same (target model, dataset) pairs aspresented in the original paper to keep our results comparable.
Revised: For the ﬁrst ﬁve methods, we evaluate using the same (target model, dataset) pairs aspresented in the original paper to keep our results comparable.

Original: For the last one [3], we use theaforementioned architectures instead, since the ones in the original paper are no longer state-of-theart.
Revised: For the last one [3], we use theaforementioned architectures instead, since the ones in the original paper are no longer state-of-theart.

Original: On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean andstandard deviation of the accuracy.
Revised: On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean andstandard deviation of the accuracy.

Original: Implementation.
Revised: Implementation.

Original: In order to keep our setup as close to the original paper as possible, we use thefollowing protocols: 1) If a previous pruning method’s training setup is publicly available, e.g.[6] and [13], we adopt the original implementation; 2) Otherwise, for simpler pruning methods,e.g., [3, 4], we re-implement the three-stage pruning procedure and achieve similar results to theoriginal paper; 3)
Revised: In order to keep our setup as close to the original paper as possible, we use thefollowing protocols: 1) If a previous pruning method’s training setup is publicly available, e.g.[6] and [13], we adopt the original implementation; 2) Otherwise, for simpler pruning methods,e.g., [3, 4], we re-implement the three-stage pruning procedure and achieve similar results to theoriginal paper; 3)

Original: For the remaining two methods [2, 5], the pruned models are publicly availablebut without the training setup, thus we choose to re-train both large and small target models fromscratch.
Revised: For the remaining two methods [2, 5], the pruned models are publicly availablebut without the training setup, thus we choose to re-train both large and small target models fromscratch.

Original: Interestingly, the accuracy of our re-trained large model is higher than what is reported in theoriginal paper 2 .
Revised: Interestingly, the accuracy of our re-trained large model is higher than what is reported in theoriginal paper 2 .

Original: In this case, to accommodate the effects of different frameworks and training setups,we report the relative accuracy drop from the unpruned large model.
Revised: In this case, to accommodate the effects of different frameworks and training setups,we report the relative accuracy drop from the unpruned large model.

Original: For random weight initialization, we adopt the scheme proposed in [16].
Revised: For random weight initialization, we adopt the scheme proposed in [16].

Original: For results of modelsﬁne-tuned from inherited weights, we either use the released models from original papers (for case 3above) or follow the common practice of ﬁne-tuning the model using the lowest learning rate whentraining the large model [4, 5].
Revised: For results of modelsﬁne-tuned from inherited weights, we either use the released models from original papers (for case 3above) or follow the common practice of ﬁne-tuning the model using the lowest learning rate whentraining the large model [4, 5].

Original: The code to reproduce the results and trained ImageNet models arepublicly available in the following anonymous link: https://drive.google.com/open?id=1HB_1FphsWtbuMAdgHbODSLdAMnj3B6TM .
Revised: The code to reproduce the results and trained ImageNet models arepublicly available in the following anonymous link: https://drive.google.com/open?id=1HB_1FphsWtbuMAdgHbODSLdAMnj3B6TM .

Original: B Additional Experiment Results
Revised: B Additional Experiment Results

Original: The results of two pruning methods are already shown in the extended abstract, here in appendix weprovide results of the remaining four pruning methods.
Revised: The results of two pruning methods are already shown in the extended abstract, here in appendix weprovide results of the remaining four pruning methods.

Original: We also include an experiment on transferlearning from image classiﬁcation to object detection.
Revised: We also include an experiment on transferlearning from image classiﬁcation to object detection.

Original: B.1 Predeﬁned Target Architectures
Revised: B.1 Predeﬁned Target Architectures

Original: ThiNet [2] greedily prunes the channel that has the smallest effect on the next layer’s activationvalues.
Revised: ThiNet [2] greedily prunes the channel that has the smallest effect on the next layer’s activationvalues.

Original: As shown in Table 3, for VGG-16 and ResNet-50, both Scratch-E and Scratch-B can almostalways achieve better performance than the ﬁne-tuned model, often by a signiﬁcant margin.
Revised: As shown in Table 3, for VGG-16 and ResNet-50, both Scratch-E and Scratch-B can almostalways achieve better performance than the ﬁne-tuned model, often by a signiﬁcant margin.

Original: The onlyexception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG-16(FLOPs reduced by 15 × ), and as a result, drastically reducing the training budget for Scratch-E. Thetraining budget of Scratch-B for this model is also 7 times smaller than the original large model, yetit can achieve the same level of accuracy as the ﬁne-tuned model.
Revised: The onlyexception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG-16(FLOPs reduced by 15 × ), and as a result, drastically reducing the training budget for Scratch-E. Thetraining budget of Scratch-B for this model is also 7 times smaller than the original large model, yetit can achieve the same level of accuracy as the ﬁne-tuned model.

Original: Regression based Feature Reconstruction [5] prunes channels by minimizing the feature mapreconstruction error of the next layer.
Revised: Regression based Feature Reconstruction [5] prunes channels by minimizing the feature mapreconstruction error of the next layer.

Original: Different from ThiNet [2], this optimization problem is solvedby LASSO regression.
Revised: Different from ThiNet [2], this optimization problem is solvedby LASSO regression.

Original: Results are shown in Table 4.
Revised: Results are shown in Table 4.

Original: Again, in terms of relative accuracy drop fromthe large models, scratch-trained models are better than the ﬁne-tuned models.
Revised: Again, in terms of relative accuracy drop fromthe large models, scratch-trained models are better than the ﬁne-tuned models.

Original: In summary, for pruning methods with predeﬁned target architectures, training the small modelsfor the same number of epochs as the large model (Scratch-E), is often enough to achieve thesame accuracy as models output by the three-stage pipeline.
Revised: In summary, for pruning methods with predeﬁned target architectures, training the small modelsfor the same number of epochs as the large model (Scratch-E), is often enough to achieve thesame accuracy as models output by the three-stage pipeline.

Original: Combined with the fact that the targetarchitecture is predeﬁned, in practice one would prefer to train the small model from scratch directly.
Revised: Combined with the fact that the targetarchitecture is predeﬁned, in practice one would prefer to train the small model from scratch directly.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Similar to Table 3, we compare relative accuracy drop from unpruned large models.
Revised: Similar to Table 3, we compare relative accuracy drop from unpruned large models.

Original: Moreover, when provided with the same amount of computation budget (measured by FLOPs) as thelarge model, scratch-trained models can even lead to better performance than the ﬁne-tuned models.
Revised: Moreover, when provided with the same amount of computation budget (measured by FLOPs) as thelarge model, scratch-trained models can even lead to better performance than the ﬁne-tuned models.

Original: B.2 Automatically Discovered Target Architectures
Revised: B.2 Automatically Discovered Target Architectures

Original: Sparse Structure Selection [13] also uses sparsiﬁed scaling factors to prune structures, and can beseen as a generalization of Network Slimming.
Revised: Sparse Structure Selection [13] also uses sparsiﬁed scaling factors to prune structures, and can beseen as a generalization of Network Slimming.

Original: Other than channels, pruning can be on residualblocks in ResNet or groups in ResNeXt [17].
Revised: Other than channels, pruning can be on residualblocks in ResNet or groups in ResNeXt [17].

Original: We examine residual blocks pruning, where ResNet-50are pruned to be ResNet-41, ResNet-32 and ResNet-26.
Revised: We examine residual blocks pruning, where ResNet-50are pruned to be ResNet-41, ResNet-32 and ResNet-26.

Original: Table 5 shows our results.
Revised: Table 5 shows our results.

Original: On average
Revised: On average

Original: Scratch-E outperforms pruned models, and for all models Scratch-B is better than both.
Revised: Scratch-E outperforms pruned models, and for all models Scratch-B is better than both.

Original: [Table]
Revised: [Table]

Original: Non-structured Weight Pruning [3] prunes individual weights that have small magnitudes.
Revised: Non-structured Weight Pruning [3] prunes individual weights that have small magnitudes.

Original: Thispruning granularity leaves the weight matrices sparse, hence it is commonly referred to as nonstructured weight pruning.
Revised: Thispruning granularity leaves the weight matrices sparse, hence it is commonly referred to as nonstructured weight pruning.

Original: Because all the network architectures we evaluated are fully-convolutional(except for the last fully-connected layer), for simplicity, we only prune weights in convolution layershere.
Revised: Because all the network architectures we evaluated are fully-convolutional(except for the last fully-connected layer), for simplicity, we only prune weights in convolution layershere.

Original: Before training the pruned sparse model from scratch, we re-scale the standard deviation ofthe Gaussian distribution for weight initialization, based on how many non-zero weights remain inthis layer.
Revised: Before training the pruned sparse model from scratch, we re-scale the standard deviation ofthe Gaussian distribution for weight initialization, based on how many non-zero weights remain inthis layer.

Original: This is to keep a constant scale of backward gradient signal [16].
Revised: This is to keep a constant scale of backward gradient signal [16].

Original: As shown in Table 6, datasets,
Revised: As shown in Table 6, datasets,

Original: E sometimes falls short of the ﬁne-tuned results but Scratch-B is ableto perform at least par with the latter
Revised: E sometimes falls short of the ﬁne-tuned results but Scratch-B is ableto perform at least par with the latter

Original: ImageNet we note that sometimes even Scratch-B is slightly worse than ﬁne-tuned result.
Revised: ImageNet we note that sometimes even Scratch-B is slightly worse than ﬁne-tuned result.

Original: This is the only case where Scratch-B does not achievecomparable accuracy in our attempts.
Revised: This is the only case where Scratch-B does not achievecomparable accuracy in our attempts.

Original: We hypothesize this could be due to the task complexity of
Revised: We hypothesize this could be due to the task complexity of

Original: ImageNet and the ﬁne pruning granularity.
Revised: ImageNet and the ﬁne pruning granularity.

Original: [Table]
Revised: [Table]

Original: B.3 Transfer Learning to Object Detection
Revised: B.3 Transfer Learning to Object Detection

Original: We have shown that the small pruned model can be trained from scratch to match the accuracy of theﬁne-tuned model in classiﬁcation tasks.
Revised: We have shown that the small pruned model can be trained from scratch to match the accuracy of theﬁne-tuned model in classiﬁcation tasks.

Original: To see whether this phenomenon would also hold for transferlearning to other vision tasks, we evaluate the L 1 -norm based pruning method [4] on the PASCAL
Revised: To see whether this phenomenon would also hold for transferlearning to other vision tasks, we evaluate the L 1 -norm based pruning method [4] on the PASCAL

Original: VOC object detection task, using Faster-RCNN [18].
Revised: VOC object detection task, using Faster-RCNN [18].

Original: Object detection frameworks usually require transferring model weights pre-trained on ImageNetclassiﬁcation, and one can perform pruning either before or after the weight transfer.
Revised: Object detection frameworks usually require transferring model weights pre-trained on ImageNetclassiﬁcation, and one can perform pruning either before or after the weight transfer.

Original: More speciﬁcally,the former could be described as “train on classiﬁcation, prune on classiﬁcation, ﬁne-tune onclassiﬁcation, transfer to detection”, while the latter is “train on classiﬁcation, transfer to detection,prune on detection, ﬁne-tune on detection”.
Revised: More speciﬁcally,the former could be described as “train on classiﬁcation, prune on classiﬁcation, ﬁne-tune onclassiﬁcation, transfer to detection”, while the latter is “train on classiﬁcation, transfer to detection,prune on detection, ﬁne-tune on detection”.

Original: We call these two approaches Prune-C (classiﬁcation)and Prune-D (detection) respectively, and report the results in Table 7.
Revised: We call these two approaches Prune-C (classiﬁcation)and Prune-D (detection) respectively, and report the results in Table 7.

Original: With a slight abuse of notation,here Scratch-E/B denotes "train the small model on classiﬁcation, transfer to detection", and isdifferent from the setup of detection without ImageNet pre-training as in [19].
Revised: With a slight abuse of notation,here Scratch-E/B denotes "train the small model on classiﬁcation, transfer to detection", and isdifferent from the setup of detection without ImageNet pre-training as in [19].

Original: [Table]
Revised: [Table]

Original: For this experiment, we adopt the code and default hyper-parameters from [20], and use PASCAL
Revised: For this experiment, we adopt the code and default hyper-parameters from [20], and use PASCAL

Original: VOC 07 trainval/test set as our training/test set.
Revised: VOC 07 trainval/test set as our training/test set.

Original: For backbone networks, we evaluate ResNet-34-Aand ResNet-34-B from the L 1 -norm based channel pruning [4], which are pruned from ResNet34.
Revised: For backbone networks, we evaluate ResNet-34-Aand ResNet-34-B from the L 1 -norm based channel pruning [4], which are pruned from ResNet34.

Original: Table 7 shows our result, and we can see that the model trained from scratch can surpass theperformance of ﬁne-tuned models under the transfer setting.
Revised: Table 7 shows our result, and we can see that the model trained from scratch can surpass theperformance of ﬁne-tuned models under the transfer setting.

Original: Another interesting observation from Table 7 is that Prune-C is able to outperform Prune-D, whichis surprising since if our goal task is detection, directly pruning away weights that are considered unimportant for detection should presumably be better than pruning on the pre-trained classiﬁcationmodels.
Revised: Another interesting observation from Table 7 is that Prune-C is able to outperform Prune-D, whichis surprising since if our goal task is detection, directly pruning away weights that are considered unimportant for detection should presumably be better than pruning on the pre-trained classiﬁcationmodels.

Original: We hypothesize that this might be because pruning early in the classiﬁcation stage makes theﬁnal model less prone to being trapped in a bad local minimum caused by inheriting weights from thelarge model.
Revised: We hypothesize that this might be because pruning early in the classiﬁcation stage makes theﬁnal model less prone to being trapped in a bad local minimum caused by inheriting weights from thelarge model.

Original: This is in line with our observation that Scratch-E/B, which trains the small models fromscratch starting even earlier at the classiﬁcation stage, can achieve further performance improvement.
Revised: This is in line with our observation that Scratch-E/B, which trains the small models fromscratch starting even earlier at the classiﬁcation stage, can achieve further performance improvement.

Original: C Network Pruning as Architecture Search
Revised: C Network Pruning as Architecture Search

Original: Here we provide results that complements Section 4 for non-structured weight pruning [3].
Revised: Here we provide results that complements Section 4 for non-structured weight pruning [3].

Original: Thenetworks and datasets used are the same as those used in Section 4.
Revised: Thenetworks and datasets used are the same as those used in Section 4.

Original: We also discuss the relation withconventional architecture search methods at the end.
Revised: We also discuss the relation withconventional architecture search methods at the end.

Original: Figure 3(left) shows our results on parameter efﬁciency of architectures obtained by non-structuredpruning.
Revised: Figure 3(left) shows our results on parameter efﬁciency of architectures obtained by non-structuredpruning.

Original: Here “Uniform Sparsifying” means uniformly sparsifying individual weights in the networkat a ﬁxed probability.
Revised: Here “Uniform Sparsifying” means uniformly sparsifying individual weights in the networkat a ﬁxed probability.

Original: It can be seen that pruned architectures are more parameter-efﬁcient thanuniform sparsiﬁed architectures.
Revised: It can be seen that pruned architectures are more parameter-efﬁcient thanuniform sparsiﬁed architectures.

Original: [Figure]
Revised: [Figure]

Original: Figure 3(right) shows our results for architectures obtained by other different design strategies.
Revised: Figure 3(right) shows our results for architectures obtained by other different design strategies.

Original: For“Guided Sparsifying” and “Transferred Guided Sparsifying”, we use the average sparsity patternsof 3 × 3 kernel in each layer stage to design new structures.
Revised: For“Guided Sparsifying” and “Transferred Guided Sparsifying”, we use the average sparsity patternsof 3 × 3 kernel in each layer stage to design new structures.

Original: Similar to Network Slimming, both“Guided Sparsifying” (green) and “Transferred Guided Sparsifying” (brown) are signiﬁcantly moreparameter-efﬁcient than uniform sparsifying (red).
Revised: Similar to Network Slimming, both“Guided Sparsifying” (green) and “Transferred Guided Sparsifying” (brown) are signiﬁcantly moreparameter-efﬁcient than uniform sparsifying (red).

Original: Comparison with Traditional Architecture Search Methods.
Revised: Comparison with Traditional Architecture Search Methods.

Original: Conventional techniques for network architecture search include reinforcement learning [21, 22] and evolutionary algorithms [23, 24].
Revised: Conventional techniques for network architecture search include reinforcement learning [21, 22] and evolutionary algorithms [23, 24].

Original: In each iteration, a randomly initialized network is trained and evaluated to guide the search, and thesearch process usually requires thousands of iterations to ﬁnd the goal architecture.
Revised: In each iteration, a randomly initialized network is trained and evaluated to guide the search, and thesearch process usually requires thousands of iterations to ﬁnd the goal architecture.

Original: In contrast, usingnetwork pruning as architecture search only requires a one-pass training, however the search space isrestricted to the set of all “sub-networks” inside a large network, whereas traditional methods cansearch for more variations, e.g., activation functions or different layer orders.
Revised: In contrast, usingnetwork pruning as architecture search only requires a one-pass training, however the search space isrestricted to the set of all “sub-networks” inside a large network, whereas traditional methods cansearch for more variations, e.g., activation functions or different layer orders.

Original: Recently, [25] uses a similar pruning technique to Network Slimming [6] to automate the design of network architectures; [26] prune channels using reinforcement learning and automatically compressesthe architecture.
Revised: Recently, [25] uses a similar pruning technique to Network Slimming [6] to automate the design of network architectures; [26] prune channels using reinforcement learning and automatically compressesthe architecture.

Original: On the other hand, in the network architecture search literature, sharing/inheritingtrained parameters [27, 28] has become a popular approach to accelerate the convergence and reducethe training budget during searching, though it would be interesting to investigate whether trainingfrom scratch would sometimes yield better results as we observed in network pruning.
Revised: On the other hand, in the network architecture search literature, sharing/inheritingtrained parameters [27, 28] has become a popular approach to accelerate the convergence and reducethe training budget during searching, though it would be interesting to investigate whether trainingfrom scratch would sometimes yield better results as we observed in network pruning.

Original: We can seethat these two areas, namely network pruning and architecture search, share many common traits andstart to borrow wisdom from each other.
Revised: We can seethat these two areas, namely network pruning and architecture search, share many common traits andstart to borrow wisdom from each other.