From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper60/Authors
comment: We thank the reviewers for the detailed comments!

1.	About the baseline model concern, we will add the baseline model’s comparison in the future version. Actually, the original motivation of our critical path distillation is DNN customization for different users, i.e. with the whole pre-trained model, as long as users need to classify a subset of classes which belongs to the original network, we don’t need to train a new small model but only need to distill the necessary critical paths from the original network and assign the distilled models to users without any retraining. But we should still compare the accuracy and overhead with training a small baseline model from scratch. Thanks for the constructive feedback!
2.	About the connections, our original aim of this work is to customize smaller DNN models with regard to subset task requirements, and the number of operation and connection reduction is not the primary focus of our work since they won’t lead to model compression effect. Due to the space limit, we didn’t mention our method’s further potential usage in the submitted paper. But we believe our findings on per-class critical paths in neural networks are an interesting and promising area which are worth more future researches.
3.	About the difference between our work and [1], [1] only analyzes the class accuracy degradation by ablating individual filters. Their main findings are still neuron-wise, in which their conclusion is “individual units specialize in subsets of the classification space indicates that the contribution of individual units towards the decisions of a CNN is both nontrivial and specific”. Our work not only verifies their neuron-wise findings by interpretable AM visualization, but also proposed a method to distill the separable per-class information flows in the neural network from both activation and gradient perspectives, which are clearly not included in [1]. 
4.	About the overhead of identifying the critical paths, we use the product of mean activation and contribution index. The whole identification process could be done by conducting one forward pass (for mean activation) and one backward pass (for contribution index) on the whole training dataset, which is fairly quick (less than 2 minutes with GTX1080 GPU support). We only use AM visualization for verification of critical path’s existence in the analysis part and it doesn’t participate in the critical path identification process.

[1] Zhou, B., Sun, Y., Bau, D. and Torralba, A., 2018. Revisiting the Importance of Individual Units in CNNs via Ablation. arXiv preprint arXiv:1806.02891.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper60/AnonReviewer1
title: Summary: The paper proposes a combination of metrics to find a critical path for each class for VGG network. This technique can decrease the number of connections and operations in the networks.
review: Key strengths: 
1-    The idea of finding a critical path for each class is interesting 
2-    Easy to read and understand 
Key weakness: 
1-    There is no baseline model for comparison which leads to difficulty in evolving  the applicability of the model
2-    The number of operation and connections in comparison to a model which does not use these new techniques are not mentioned in the paper.
3-    It is not obvious what the difference is between this paper and reference  1 in the paper  
4-    The overhead of finding a critical path with three metrics are not mentioned in the paper.

rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper60/Authors
comment: We thank the reviewer for the very constructive reviews!

1.    About the experiments of one-vs-all classifiers, we do understand that this is an impractical assumption in the real world, which also lacks a baseline model and previous researches comparison. 

But in our opinion, the novelty of our paper mainly lies in the following aspects:
     a.    the verification of the critical paths’ existence using multiple metrics;
     b.    most importantly, our method could distill the critical paths, or decouple them from each other or the original neural network without hurting per-class accuracy.

2.    From the computing perspective, after we verified the critical paths could be decoupled from other paths, one practical usage for computation reduction is to break unnecessary connections within convolutional layers, i.e. disconnecting one class’s neurons with other classes’ feature maps produced by other-class neurons in the previous layer. Suppose one layer has 512 neurons which produce 512 feature maps, in current structures, each neuron in the next layer need to conv with all 512 feature maps. While with this decoupling, neurons only need to conv with 52 feature maps (10 percent in our experiments) produced by their preferred-class neurons’, which will reduce significant computation overhead even in the all-classes scenarios. 

3.    For real-world cases, our method could be used for customization of DNNs, when centralized cloud server may want to provide APIs to different edge users with resource constraints and demands of different subset classification. Our on-going work now can successfully distill multiple classes (i.e. 2 classes, 5 classes, etc.) without hurting the classification accuracy in the subset, and still without retraining. We believe this is a more practical assumption in real word cases. 

4.   Due to the space limit, we did not mention the above details in the submitted paper, but we think our method has much potential than our paper has mentioned. And we also plan to generalize our method to large-scale ImageNet classifiers in the future.

We appreciate the constructive reviews again!

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper60/AnonReviewer2
title: Though one-vs-all experiments are not convincing enough, the existence of "critical path" is interesting.
review: Summary: This paper demonstrates the existence of class-specific "critical paths" in neural networks, using three measures of importance, namely mean activation, Taylor approximation of loss, AM visualization. For each class, the critical path's size is much smaller than the whole network, especially in higher layers.

Pros:
1. The fact that "critical paths" exist for each class, and the gradual change of them with depths, are interesting. This is the main reason why I give a 3/5 rating.
2. A relatively complete analysis of three methods for measuring importance is conducted, with good visualizations.
3. The writing and structure is fairly good.

Cons:
1. Results for one-vs-all classification are not convincing. There's no comparison with baseline methods (e.g., train a one-vs-all classifier of the same size). I couldn't see a direct practical usage of the proposed “critical path” unless it really outperforms one-vs-all classifier of the same size. Maybe the advantage here is critical path does not need retraining, but I think the baseline I mentioned can be considered. 
On the other hand, if the focus is only on network interpretability, without emphasis and more experiments on practical usage, I'm afraid the novelty is limited.
2. Given the network's classification is at the end, it is not that surprising that higher layers tend to have more class-specific neurons. In the extreme, the final logits exactly represent all classes respectively. Maybe how gradual the change is (with depths), is more interesting and more analysis could be provided.
3. A minor issue: If my understanding is right, "neuron" refers to convolutional channels, but visualizing all channels of one layer in one square (Figure 1 and Figure 2-1.2.) can cause some misunderstandings, since some people might think the square is one channel, and each point stands for a single activation value. Maybe more explanation can improve this. 




rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper60/Authors
comment: We thank the reviewers for the suggestions. 
1.    About the first concern of the reviewer, “the std values or activation values of features maps rely on the values of convolution filters and other parameters such as batch normalizations, which cannot be seen as a useful priori in my opinion”. In our experiments, we tested the STD both before the batch-norm layer and after the batch-norm layer. Based on our experiments, they show exactly the SAME trends that latter layers have the larger STD. Meanwhile, we understand that the STD could not verify the functionality differentiation alone, so our critical path does not rely on this single criterion. And that’s why we use three different neuron functionality analysis methods: per-class mean activation, contribution index, and AM visualization, to cross-verify each other to get our final conclusion that critical paths exist. And all these methods are consistent with each other， which we believe is convincing enough to verify our theory.
2.    The reviewer mentioned, “- In order to simulate the critical path of the teacher network, the proposed method using the last six layers to train the student network, which is an unfair comparison to other methods such as KD and FitNet.” Please understand that our method is NOT a Teacher-Student model. And we are NOT simulating the critical path or training the network. Instead, our method reserves the original filters and their original weights and bias on the critical path and prune all the filters. Meanwhile, our method does NOT need to “train the student network”, either. We have stated in the paper that “without any retraining” in Line 28, and “Without any retraining, the distilled model preserves the ability well to respond to the chosen class and reject all other classes” in Line 98-99.
3.    About no comparison experiments, we argue that our proposed critical path distillation is based on per-class information flow or called functionality perspective. And our distilled models conduct one-vs-all tasks. To our best knowledge, there are no previous methods talking about the critical paths of per-class information flow. If comparing to traditional compression like pruning techniques, it is more unfair since we have different tasks. Currently, there are no pruning techniques which could prune 90% filters in the latter layers in VGG16/CIFAR10 without retraining. Therefore, we did not compare our experimental results with them, which we also state in the paper that “our method proposes a new perspective on per-class information flow and thus could aggressively remove more filters without any retraining” in line 27. 

Thanks again for providing the review.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper60/AnonReviewer3
title: Overall rating: 2/5
review: - The basic idea or motivation of this paper is the observation that average STD values of higher layers are larger than those of other layers and proposed a new view for knowledge distillation. However, the std values or activation values of features maps rely on the values of convolution filters and other parameters such as batch normalizations, which cannot be seen as a useful priori in my opinion.

- In order to simulate the critical path of the teacher network, the proposed method using the last six layers to train the student network, which is an unfair comparison to other methods such as KD and FitNet.

- There are no comparison experiments in the paper, which makes readers hard to evaluate its superiority.

-----------------

Sorry for the misunderstanding :)

I have carefully read the paper again, and noticed that the paper is a neuron pruning method for compressing and speeding up deep CNNs using critical paths.

In addition, different from existing methods, the proposed method conducts the one-vs-all tasks.

Therefore, there are still some important issues:

1) It is easy to train a one-vs--all classifier model on CIFAR-10 and then compress it using state-of-the-art methods, since there are only 60K images in this dataset.
2) Does the proposed method can be effectively applied on large datasets with more categories, e.g., ImageNet.
3) Conventional methods aim to minimize the reconstruction error of feature maps of original and compressed images, which can be applied on different tasks such as detection and super-resolution. It seems that the proposed method is only effective for image classification tasks.

rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct