From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper29/Authors
comment: We are aiming for methods that are straightforward and computationally efficient to work in our industrial context (on-the-fly sparsification). Thus, we present intuitive and effective methods to sparsify a pre-trained CNN on-the-fly that have limited computational cost and introduce up to 73% sparsity without the need to retrain or fine-tune. 

Thanks for pointing this out. We might have been clearer in the text. By referring to our proposed method as model-independent we mean that it can be applied to any CNN models without having prior knowledge about it. Thus, each method on its own is model independent. It is true that one method outperforms the other two for a given model. However, using any methods on any desired pre-trained CNNs we can achieve significant sparsity. A method does not require any special type of CNN to start with and our results show that it is effective on the 5 state-of-the-art models.

The goal of the paper is to introduce and evaluate the sparsification methods. Our evaluation shows that there can be benefit from predictive modeling and this is left for future work. 

Due to space limitations, we left the full formal description of our three proposed method. We revised the submission to include some of the details, leaving the full details to a full-length paper version. Please see our revised paper for details.

Yes. The threshold for the relative method is the same for all layers (e.g., 40%, 50%, 60%, etc.) and we vary this threshold to study the inference accuracy as a function of the sparsity (Figure 2a-e). In the fine-tuning section, we vary the threshold per layer to show the extra sparsity we can gain by means of a further sensitivity analysis. We conclude that little further is gained attesting to the effectiveness of our simple methods.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper29/Authors
comment: We indeed do cite the work by Han et al. and also compare to it (now lines 99-103 in the revised submission). However, page restrictions limit us to be brief. The work of Han and others requires re-training. The novelty of our methods is that they are model-independent and require no re-training. To the best of our knowledge, we are the first to do so. The only similar result we noted in recent papers was the one reported in Han et al. 2015, using Caffe's Alexnet and we draw comparisons in the experimental evaluation section.

We revised the submission to fit a limited formal description on each method. The description now appears on lines 62-80 of the revised submission.

We are a bit unclear on what the reviewer exactly means by “connectivities between filters”. Our method address both convolution and fully-connected layers in a fine-grain manner (i.e., non-structured sparsity). Thus, we do address connectivities between filters.

Thanks for pointing this potential for overfitting out. This indeed may overfit. However, we do not use the fine-tuned model in our context, since it is likely to be computationally prohibitive. We rather use fine-tuning to point out that the gain from the process is small, leading us to conclude that our on-the-fly-methods are effective. Even with over fitting (as an upper bound), we get no more than 8% (and 4%, 3% on others).

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper29/AnonReviewer2
title: Review comments
review: This paper proposes approaches for pruning CNNs without retraining. Specifically, this paper introduced three schemes to determine the thresholds of pruning weights. The pruning step is based on evaluating the absolute values of weights against the threshold and insignificant weights are pruned to sparsify the network. The thresholds are determined by using a constant value, increasing value, or a layer-specific value, which correspond to three schemes proposed in the paper.   

My major concerns of this paper are its lack of both novelty and clarifications. Pruning based on the magnitude of weights against a threshold is similar with the seminal work by Han et al. 2015. Yet this paper lacks the mechanism to determine the connectivities between filters. In the ‘triangle’ and ‘relative’ schemes, the technical details in determining the thresholds are missing. I highly encourage the authors to focus on the technical expositions regarding these two schemes and add comparisons with state-of-the-art approaches. 

The paper claims that ‘We then vary the degree of sparsity of each layer in turn, attempting to maintain a no more than 5% drop in inference accuracy’ which makes me skeptical that the approach might be working by overfitting to the test set. 

rating: 2: Marginally below acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper29/AnonReviewer3
title: Useful for many applications, achieves good results, but straightforward and lacks some clarification
review: 
The paper describes a method for sparsification of CNNs without retraining. 

Pros:
-The problem is important and useful for many applications where re-training might not be an option. 
-The method achieves good results.

Cons:
-The proposed pruning strategies in the paper are quite straightforward.
-I would argue calling the method "model independent" since best sparsification for each model varies (relative or triangular methods). 
-Although the authors say predictive modelling is needed to identify the choice of model, no such modelling method was proposed in the paper. 
-The description of relative method is not complete. The authors say it defines a unique threshold that sparsifies a certain percentage of the weights in the layer. How was this certain percentage selected? Is it the same for each layer? If not how was it selected for each layer?

rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/CDNNRIA/Program_Chairs
comment: Dear reviewers,  

Would you please follow the guidance on the workshop webpage, to re-scale your score and also give the confidence rating?    https://nips.cc/Conferences/2018/Schedule?showEvent=10941 

"For reviewers, please rating for 1-5 for definitely accept (5) vs definitely reject (1), and a confidence rating from (1, my recommendation might be biased), (2, I will argue for my recommendation) (3, my recommendation is the best). Please give detailed comments to support your recommendation. Please also indicate whether you will recommend the paper for the best paper award"

Thanks!

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper29/Authors
comment: We thank the reviewer for the insightful comments. The use of Knowledge Transfer (KD) can indeed alleviate the need for the original training data by offsetting it with the use of a transfer set. However, it does not alleviate re-training altogether. It only avoids the need to retrain with the original labeled training data (knowledge is transferred to the distilled model by training it on a transfer set (Hinton et al. 2015)). In contrast, we avoid any re-training, which we see as particularly necessary on mobile platforms. In this context, the model itself is embedded within an app that is loaded by the user. The app uses the mobile platform vendor's runtime system (a library, such as NNlib for TensorFlow on Qualcomm devices) to build and execute the model. There is no opportunity to retrain on-the-fly. KD would address the lack of training data, but not the need for some retraining.
  
We revised the submission to (1) cite KD (line 20) and (2) edit the second paragraph of the Introduction (lines 21-30) to clarify the context we address and better motivate the need for on-the-fly sparsification.

We also should point out that our fine-tuning does not require re-training, and thus is not similar to KD. Further, it only adds 8% additional sparsity (even if it overfits). This indicates that our on-the-fly methods are very effective with no fine-tuning.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating: 3 of 5

Review:
This paper discusses pruning under the setting where re-training is not allowed. The paper propose three types of threshold heuristics to conduct magnitude-based pruning and analyze the performance of them on various modern deep neural networks. The paper demonstrates that up to 81% of sparsity with less than 5% drop in top-5 accuracy can be achieved without re-training. My only concern is the motivation regarding pruning without re-training. If the training data is not accessible during pruning, knowledge distillation [1] could be applied to conduct re-training. That is, unless it is shown empirically that distillation will be worse than no re-training, which I doubt, stronger motivations are needed for this work.

Pros
- Simple idea and easy to follow.
- Considers an interesting setting where re-training is not permitted and achieves good results.

Cons
- The result that achieves 81% with no more than 5% top-5 accuracy drop might over-fit to the test set since the authors describe their method as: "varying the degree of sparsity of each layer in turn, attempting to maintain a no more than 5% drop in inference accuracy."
- The authors argue that training data might not be accessible during pruning, which calls for pruning without re-training. However, even the training data does not exist (no label) at the time of pruning, techniques such as knowledge distillation [1] could help and enable re-training (it is feasible since we have the large model to prune).

Confidence: 3 of 3

[1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).