From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper7/Authors
comment: Thank you very much for your review.

We fully understand your concern that this small set of experiments is not enough evidence to generalise to various scenarios and plan to address it in future work. Given the short nature of the abstract, we attempted to find the most general case possible to give a succinct argument. We used a very popular network and dataset, and examined a pruning scheme that is state-of-the-art.  We intended for this to serve both as a useful insight for neural network designers, and also as motivation for further study on the finetuning pattern commonly used by pruning techniques and whether this could be useful in the context of architecture search.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper7/Authors
comment: Thanks for your review! Regarding your comments:

1. Why WideResNet?
We chose to use WideResNet as our base network given the popularity of the ResNet architecture - with modular blocks and skip connections it is representative of a large number of commonly used networks and we believe the results we obtained will generalise to a good number of use cases. 
Furthermore, WideResNet is a reasonably compact architecture with good accuracy, and, for example, no large fully connected layer that may give misleading results in terms of pruning percentages.

We will add this justification to the paper as per your suggestion.

2. Bold statements
We accept that these statements are bold and will continue to provide further evidence in future work. We hope that this preliminary work will be a useful insight to those working in this field.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Well written paper with interesting conclusions.  The employed Fisher Pruning is state-of-the-art.

 For me, the choice of the base network 'WideResNet' is not clear and should be motived briefly. 

The paper makes rather bold statements about pruning, which might be a bit early considering that the experiments have been done only with one base network - for other networks the experiments might paint a different picture.

Rating: clear accept
Reviewer conficence: 3 / 5

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper7/AnonReviewer3
title: This is an interesting paper that compares the pruned network and simple network. The major concern is whether the conclusions reached are valid based on such limited experiments. 
review: This paper examines and compares whether the network pruning is better compared to the simple network structures. In principle, such comparisons are meaningful. However, my major concern is that whether such comparisons have enough generalization capability, such that they could be meaningful and useful in various applications scenarios. For example, the conclusions based on the employed dataset, base network and  pruning method may not still work for other network structures and tasks. Therefore, the conclusions are constrained to very particular scenarios and much future work need to be done. 
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: ~Ting-Wu_Chin1
comment: Hi Elliot,

Thanks for trying to address my concerns. In summary, I have two opinions based on the reply:

- Why comparing networks with two different meta-architectures when discussing pruning? Pruning can be applied to any architecture. I agree that there are architectures that are inherently more efficient (e.g., MobileNet [13,14]), however, they can also be further pruned to achieve even better results [2,4,9] (better than simply do width-multiplier to control the complexity). In Fig. 1 of the paper, you are comparing the same meta-architecture (WRN), which makes sense, however, there is only one pruning technique (red line, Fisher pruning). I think if replacing that with some other techniques [2,3,4,9], "We can see that these simple networks trained from scratch consistently outperform the pruned-and-tuned networks." might not hold.

- I agree transfer learning may or may not happen, which means weights may or may not be important. In this case, your claim: " it is the resulting structure — not the learnt weights — that is important" is still too strong in my opinion. Moreover, transfer learning is practical, I'm sure not every task is CIFAR-10 or ImageNet, which is why we need transfer learning since our target dataset is limited. If we are talking about vision applications, object detection, VQA, and semantic segmentation all use pre-trained weights from ImageNet.

In general, I definitely agree with the authors on the viewpoint of pruning as an architecture search, which also has been exploited by my previous work [4], however, I do see the value in the inherited weights of the pruned network, for example, transfer learning. Moreover, if one wants to traverse the space of complexity and accuracy efficiently on a large dataset, weights are especially important since you do not want to train each candidate 90 epochs, for example, on ImageNet, and I do not think assuming we have a pre-trained model on ImageNet nowadays is strong. If we assume we have no pre-trained model at all, then actually the pruning problem does not exist in the first place, the problem reduce to architecture search given some complexity constraint, e.g., [15].

Thanks,
Ting-Wu

[13] Sandler, Mark, et al. "MobileNetV2: Inverted Residuals and Linear Bottlenecks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[14] Howard, Andrew G., et al. "Mobilenets: Efficient convolutional neural networks for mobile vision applications." arXiv preprint arXiv:1704.04861 (2017).
[15] Dong, Jin-Dong, et al. "DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures." arXiv preprint arXiv:1806.08198 (2018).
title: Thanks, but two more thoughts

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper7/Authors
comment: Hi there,

Thanks for the interest in our work! I will try to address the points you have raised:

Now, when removing channels in groups, or even most of them at once, there is still the issue that one is comparing (i) training a large network from scratch + pruning overheads, vs. (ii) training a small network from scratch. Even if the pruning overheads are substantially reduced (as achieved in e.g. [4]) it is still preferable to train the smaller network, assuming it performs well. Moreover, there are other factors at play (e.g. how many update steps to take before pruning) controlling expense.

Regarding learning rate, [1,2,4,5,6,7,9,12] use reduced learning rates. Even [3] uses a reduced learning rate for that matter.

Unfortunately, [3] does not lead to comparable/better results than training from scratch: for example for CIFAR-10, a top-performing network in [3] (6.07% error @ 1.82x10^8 FLOPS) is not preferable to training a simple bottleneck model from scratch (5.81% error @ 1.04x10^8 FLOPS).

In general, several works (https://arxiv.org/abs/1711.08856, https://arxiv.org/abs/1706.05394, https://arxiv.org/abs/1807.05031) emphasise the importance of the early phase of learning, and simple regularisations (such as using a large LR) are unlikely to replace training from scratch. It would be interesting to verify this for truly large learning rates, like those used in https://arxiv.org/abs/1708.07120.

Although inherited weights can be beneficial for transfer learning, this assumes we have them to begin with which involves training a large model from scratch, which is expensive. In a world where not every task is CIFAR-10 or ImageNet (perish the thought!), and we want to learn a new network, on new data quickly, we don’t have any such weights to rely on.

title: A reply

From: ~Lixin_Fan1
comment: Authors please send author information to workshop chairs (lixin.fan@nokia.com), so that you submission can be assigned to reviewers without conflict of interests.
title: Authors please send author information to workshop chairs (lixin.fan@nokia.com).

From: ~Ting-Wu_Chin1
comment: Hi,

I find this article interesting and would like to point out some of the conclusions that might not hold for other pruning methods.

- The way the paper is phrased seem to imply that greedily pruning filter one-by-one is the common way and the well-adopted method of pruning, however, that is not true. [1-5] are all pruning multiple filters at a time without the so-called expensive procedure (one-by-one pruning). [9-12] prunes with optimization perspective that consumes that time it takes to train a regular network.

- The results of one-by-one pruning is always beaten by a simple counterpart is a little misleading without explanation. This is happening just because you have a small learning rate for the pruning while a large learning rate for the counter-part. However, such small learning rate for pruning is not necessary, for example [3] uses larger learning rate for pruning and achieves compelling results.

- In general, I think it is true to say "pruning filters one-by-one greedily and fine-tune with small learning rate" will let the network stuck at local optimum and is worse than training from scratch given the same architecture. However, that is just for this certain kind of pruning technique [6,7] and it is misleading to conclude "Pruning is a very expensive procedure, and should be avoided when one is time-constrained." Pruning in [1] is not expensive at all where you just rank the filters and prune by top-k, and with suitable hyperparameters as suggested in [3] could lead to at least comparable results to train from scratch if not better. Pruning actually should be adopted when time is constrained since the inherited weight provides much faster convergence compared to training from scratch.

- Last, the conclusion "it is the resulting structure — not the learnt weights — that is important." is misleading. The inherited weights is important for faster convergence if you only have limited time. Additionally, it is important in the scenario of transfer learning setting. Even in the Anonymous paper you cite [8], they at least point this fact out instead of having the claim you are making.

[1] Li, Hao, et al. "Pruning filters for efficient convnets." arXiv preprint arXiv:1608.08710 (2016).
[2] He, Yihui, et al. "Amc: Automl for model compression and acceleration on mobile devices." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[3] He, Yang, et al. "Soft filter pruning for accelerating deep convolutional neural networks." arXiv preprint arXiv:1808.06866 (2018).
[4] Chin, Ting-Wu, Cha Zhang, and Diana Marculescu. "Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks." arXiv preprint arXiv:1810.00518 (2018).
[5] Yang, Tien-Ju, et al. "NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[6] Molchanov, Pavlo, et al. "Pruning convolutional neural networks for resource efficient inference." arXiv preprint arXiv:1611.06440 (2016).
[7] Theis, Lucas, et al. "Faster gaze prediction with dense networks and Fisher pruning." arXiv preprint arXiv:1801.05787 (2018).
[8] Anonymous.  Rethinking the value of network pruning.  In Submitted to International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJlnB3C5Ym. under review.
[9] Gordon, Ariel, et al. "Morphnet: Fast & simple resource-constrained structure learning of deep networks." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
[10] Dai, Bin, Chen Zhu, and David Wipf. "Compressing Neural Networks using the Variational Information Bottleneck." arXiv preprint arXiv:1802.10399 (2018).
[11] Louizos, Christos, Karen Ullrich, and Max Welling. "Bayesian compression for deep learning." Advances in Neural Information Processing Systems. 2017.
[12] Liu, Zhuang, et al. "Learning efficient convolutional networks through network slimming." Computer Vision (ICCV), 2017 IEEE International Conference on. IEEE, 2017.
title: Interesting but only one kind of pruning method