From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/Authors
comment: We thank you for your reply and interest, we will address the second point along with the comment:

According to the filter pruned, we prune also the corresponding batch normalization weight, this is essential because of two reasons, the first reason you have already mentioned in your second point, the second reason is better explained with an example; let's assume we have a convolutional layer with originally 20 filters, by pruning 50% we remove 10 filters, having 10 remaining filters, the corresponding batch normalization layer weights are pruned also according to the corresponding pruned filters, the same applies for biases (line 74). In general, compression of the model wouldn't have been achievable without adopting such a technique, as we finally create a compressed model having only 10 filters in this layer, and the corresponding batch normalization layer would contain only corresponding 10 weights, which are the non-pruned weights in the previous example. 

An important remark: after finishing all pruning iterations, we create a new model with the same size as the non-pruned number of parameters, and then copy the non-pruned parameters from the old to the new model, creating a non-sparse compressed model.

We hope that we addressed your comments. Please add more comments in case you require further clarification regarding these points or for additional questions

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/AnonReviewer8
comment: 
1. With the further justifications, I could get why global ranking is a better choice than layer-wise ranking like "L1 avoided pruning the most sensitive layers completely, introducing another limitation, not to mention unifying the pruning percentage on every ResNet block of layer". These are indeed the existed issues of L1 method. Thanks for the detailed comments.

2. The normalization is indeed necessary since "filters with a higher number of kernel weights have a higher probability to have higher L1-norm". I could also understand in practical, when using masks to restrict weights from changing instead of creating smaller models, normalizing the L1 norm of filters by non-zeros numbers is necessary.  But another concern I have is the global L1 norm may suffer from other issues (though experiments show it's not serious), e.g. BN layer. 

For example, in another paper [1], Section-4, the authors state that "When ReLU non-linearities are used, the network remains unchanged if we (for example) multiply the weights in one layer of a network by 10, and divide the next layer by 10. This invariance is even more prominent when batch normalization is used. In this case, the size (i.e., norm) of a filter is irrelevant because the output of each layer is re-scaled during batch normalization." This seems to be contradictive with the proposed global ranking of L1 norm since different layers can form different "scales" during training. Do authors have any opinions with this layer weights re-scaling statements?

1. Visualizing the Loss Landscape of Neural Nets, Li, et al. NIPS 2018

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/Authors
comment: We thank the reviewer for his constructive comments. We will address the comments in their same order.

1- First, we agree that this part needs to reflect the novelty more obviously.  Secondly, addressing your question; The limitation of layer-wise L1 ranking is that it requires extra calculation for layer importance to decide the pruning percentage of each layer. Let us take [1] for example, they didn't calculate filter sensitivity, instead, calculated each layer's sensitivity to pruning based on removing each layer and calculating the layer sensitivity based on the accuracy drop on a validation set, in addition to that, they avoided pruning the most sensitive layers completely, introducing another limitation, not to mention unifying the pruning percentage on every ResNet block of layers. On the other hand, our method considers all filters simultaneously independent of the layer or block, we simply calculate L1-norm per filter and then stack L1-norms of all filters from all layers, we hypothesize that not only the filter L1-norm per layer reflects the filter importance within the layer, but that it reflects the filter importance globally for all layers.

2- To address this comment, we have to clarify two factors affecting L1-norm calculation for filters, A) Number of channel kernel weights. B) As we and [1] hypothesize, the importance of the filter weights. For [1], as they pruned each layer independently, the filters had the same number of kernel weights, but while we consider filters from different layers, not normalizing would lead filters having a higher number of kernel weights to not get pruned (They have a higher probability to have higher L1-norm). So we normalize each filter over the number of its kernel weights, leaving the importance criteria as the only deciding factor and establishing fairness between filters across the network. 

3- We thank you for this comment. we agree that these terms can cause confusion, we will address your comments in the final version. Moreover, while we indicate "filters or neurons" to establish that we address both the convolutional layers and fully connected layers, we will work on addressing this part in a more efficient way in the final version.

We hope that we answered your comments/concerns. In case you require further clarification regarding these points, or additional comments, please do not hesitate to add them.


[1] Pruning Filters for Efficient ConvNets, Hao Li, et al. ICLR'17.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/AnonReviewer8
title: The idea of L1-norm ranking is well-known, but the global-L1-norm seems to work better according to authors' experimental results
review: Pros:
1. The authors propose a pruning method based on the filters' global ranking of L1-norm. According to the experimental results, the performance of the proposed method on ResNet is much better than [1] as shown in Table.1.

Cons:
1.  The novelty of the proposed method is not well justified. Authors should emphasis more details about why layer-wise L1 ranking is not as good as global ranking, or why global ranking could provide so much performance improvement.  The reasons mentioned in the paper "Such a strategy makes the layer importance calculation unnecessary, as stacking all filters and output features in the network provide a global importance insight" is not convincing enough.  Without good justification, the novelty of this paper is kind of limited.
2. I could understand the normalization is a very important step in calculating the global ranking, but in the paper, very few insights are provided except "normalization over the nonzero elements is needed" in Line 62.  I still cannot get why this is the best normalization method. 
3.  The writing is kind of misleading. For example, the authors should use a unified term in the paper instead of using "filters or neurons" in many places. Another example is the terms used in Table.2, "Non-Structured", "One-shot", "Non-blind", etc., which are very confusing even with the explanations. 

But resolving the above issues with good justification, I think the proposed method is still a good method and may have much potential. 

[1] Pruning Filters for Efficient ConvNets, Hao Li, et al. ICLR'17.
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/Authors
comment: We thank you for finding our idea interesting.

You can view our comments on the previous review, we think that it answers your questions.

In case you have further comments, please don't hesitate to ask.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/Authors
comment: We thank the reviewer for his comments. We will address your comments in their same order:

1- The key difference between [1] and our method, is that we don’t consider layers separately, all filters in all layers are stacked together independent from the layer before deciding which to prune. This key difference is explicitly mentioned in the introduction (line 37). Due to stacking, we don’t need to pre-calculate which percentage to prune from each layer, we just specify a global pruning parameter p to the network.

2- The definition of class-blind is that it doesn’t care to which layer a particular component (ex: weight, filter, output features,…etc) belongs. Class blind pruning was first applied by [13] on weights separately.

3- As we prune iteratively, p is increased step by step every pruning iteration until the maximum compression is reached without losing performance.  The same criteria were applied in all other papers that used iterative pruning.

4- A) To be able to compare our method to other methods in the same line of work, we tested the method on part of the already existing benchmark for structured pruning [1] [17]. B) While it is true that ResNet has a large number of parameters, it is very challenging to prune a large percentage of ResNet without losing the baseline accuracy, we even outperformed the other methods which introduced some loss in accuracy, while we haven’t. C) Generally, for any network, it may be fairly easy to prune a small portion of the network (ex: 1%) without losing performance, but it is very challenging to prune most of the network while maintaining the same performance, which other methods tried to achieve. We reported 97.4% compression on MNIST which is on par with Deep Compression paper [3] which used pruning, quantization, and Huffman coding to reach similar performance and they need customized libraries while we do not. Also, we outperformed state-of-the-art on the other tests.  Finally, it is true that it is very valuable to prune networks on ImageNet, and this is why this is well within our plan (line 103).

We hope that we answered your comments/concerns. Please don’t hesitate to add more comments if any.

[1] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. 137 CoRR, abs/1608.08710, 2017. URL http://arxiv.org/abs/1608.08710.
[3] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural network 112 with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149, 2015. URL 113 http://arxiv.org/abs/1510.00149.
[13] A. See, M. Luong, and C. D. Manning. Compression of neural machine translation models via
139 pruning. CoRR, abs/1606.09274, 2016. URL http://arxiv.org/abs/1606.09274.
[17] R. Yu, A. Li, C. Chen, J. Lai, V. I. Morariu, X. Han, M. Gao, C. Lin, and L. S. Davis. NISP: 147 pruning networks using neuron importance score propagation. CoRR, abs/1711.05908, 2018. 148 URL http://arxiv.org/abs/1711.05908.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/Authors
comment: We thank the reviewer for the constructive comments and interest in our work. We will address your comments in their same order:

1- It is true that our method prunes filters in the convolutional layer, but it also prunes the corresponding parameter in the fully connected layers- neurons. The importance calculation is not separated from our method, it is just a visualization of our algorithm, we define the priority to remove a specific filter or neurons, as their relative L1-norm to sum of all L1-norms across the network. Finally, prune() just removes the component from the network, it has no criteria to prune or not prune, so if the corresponding L1-norm to a neuron or filter is below the threshold, it is removed.

2- You have a point, we thank you for that. We will address your point in the last version of the paper.

3- We have different key components in our method: A) Blindness: Which considers all layers simultaneously. B) Structural Pruning: Which removes an entire filter or neurons instead of weights separately. We remove A from our method to create Non-Blind pruning : which prunes each layer separately, alternatively we remove B create Non-Structural Pruning : which removes weights separately based on L1-norm of individual weight. The previous methods were tested by pruning it on one-shot and observing how much compression can be achieved in each method. Finally, we just compare between the original method using iterative pruning, and pruning in one-shot.

We also would like to highlight something with regards to your comment; "It is able to achieve network compression while maintaining acceptable accuracy", we would like to state that we were able to achieve network compression with higher or the same accuracy as the baseline.

We hope that this answers your comments/concerns. Please don’t hesitate to add more comments if any.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/AnonReviewer6
title: Although this paper is technically sound, the points below are worth considering.
review: This paper proposes a new pruning method based on [12]. Compared with other compression methods, this method obtains more compression and decreased redundancy in parameters, exhibiting good performance.

1. According to the compression method by computing L1-norm, sorting filters and removing m filters with smallest L1-norm values in the reference paper [12], what is the main differences between your paper and [12]? Please give your innovations explicitly.
2. What is the meaning of “class-blind”?
3. How to choose the percentage p of sorted norm values in different networks, e.g., LeNet or ResNet?
4. Finally, the experiment in MNIST and Cifar10 is not persuasive. With a large number of parameters in ResNet, over-fitting may occur in Cifar10. Therefore pruning filters can alleviate over-fitting and obtain good results. It is recommended to do multiple experiments in  large-scale datasets, e.g., ImageNet.

[12] H. Li, A. Kadav, et al. Pruning ﬁlters for efﬁcient convnets. CoRR, abs/1608.08710, 2017. URL http://arxiv.org/abs/1608.08710
rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Thank authors for presenting the interesting idea!
I read through the paper and based on my understanding, the most difference of this paper's idea from L1 pruning [1] is that this paper proposes the global ranking of filters' importance. right?
And my understanding is that the basic pruning units are still filters, instead of the whole layers. Is that right?

[1] Pruning Filters for Efficient ConvNets, Hao Li, et al. ICLR'17.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper52/AnonReviewer2
title: The paper proposes a NN pruning technique that uses L1-norm of layer weights to decide whether or not to prune away that layer.  The technique is data blind.  It groups layers of a kind together and keep only those layers whose weights' L1-norm exceeds some threshold.  The result is a compact, yet non-sparse, NN.  Non-sparsity is a positive in that this network can be deployed any where the original NN can be used.
review: The technique appears to work in practice.  The authors have done a decent job of evaluating the technique on three representative deep networks on MNIST and CIFAR-10 datasets.  It is able to achieve network compression while maintaining acceptable accuracy.  I do have a few comments, however.

1. Algorithm refers to items which are either undefined or vague.  For example, Neuron isn't explicitly defined.  Similarly function prune() is vague.  I initially thought that it simply removes a filter; however, it seems it is doing a bit more than that.  See the section about importance calculation.  So, I assume, that prune() can choose to not delete a filter even though its weights' L1-norm falls below the threshold?

2. Table 2 should clearly indicate your approach.  In Table 1, you refer to your approach as "Ours."  Perhaps follow a similar strategy in Table 2.  Table 2 should also include a column that shows percentage compression achieved.  **I expected both tables to be somewhat similar in appearance.**

3. I couldn't really follow the discussion in para starting at Ln. 80.  You may want to clarify this further.

rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct