From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper35/Authors
comment: Thank you for your interest in our work!

1. The student blocks are trained in isolation with the IR loss, but we additionally use KD loss for classification tasks. For all tasks, we further fine-tune the student model with the KD loss (classification) or standard loss (GANs).

2. We unfortunately did not calculate this information, but we plan to open-source our code after the full paper is reviewed.

From: ~Zhendong_Zhang1
comment: Very nice work!
But I have two small questions:
1. Each student block is trained in isolation. After all student blocks are trained, is it necessary or helpful to fine-tune the model as a whole?
2. If I understand right, IR loss is the mean squared error (MSE) between intermediate representations. What is the approximate value of the MSE at the end of training?

title: Small Questions

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper35/Authors
comment: Thank you for your thoughtful review. We have put answers to your questions inline below.

- The width of the CNN greatly affects the computational cost of the network. It is possible to reduce the with of the network with the same approach?

We did not explore reducing the width of networks. However, we have preliminary results that show we can reduce the group cardinality of grouped convolutions (e.g., as used in ResNeXt). We hope to explore reducing the width of networks in future work.


- During compression, are you using the training set same as the teacher network?

For all of our experiments, we use the same training set as the teacher network. As with KD, it is possible to use different training set, but we do not explore this setting.


- In Figure 2, the result of KD almost always beats the teacher network performance. Is this result is general in KD method? Because in original KD paper, it shows that the distilled single model performs a little bit worse than the teacher 10xEnsemble model.

Recent work has shown that student models can outperform teacher models in accuracy [1]. We also have preliminary results showing the alpha parameter is critical in achieving high accuracy. The original KD paper used a fixed alpha, which may explain the difference in results.

[1] Furlanello, Tommaso, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. "Born again neural networks." arXiv preprint arXiv:1805.04770 (2018).

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper35/Authors
comment: Thank you for your thoughtful review.

We have additional experiments that support our hypothesis, which we were unable to include in the paper due to space constraints. We tried the following variations of hint training:
- One IR, with teacher input
- One IR, no teacher input (i.e., FitNets)
- Multiple IRs, no teacher input
All of these variants under-performed standard KD and LIT for ResNet and ResNeXt.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating: 4/5
Confidence: 2/3

This paper proposes a new method for model compression that extends the idea of hint training. It first emphasizes that conventional hint training and knowledge distillation (KD) method cannot perform effectively for modern deep networks. As the solution to this issue, it proposes two main ideas. First, different from conventional hint training, it trains the student network's intermediate layer with the same width teacher network intermediate layer. Second, the student network gets the previous block in the teacher model as input. These two ideas make student network more stable to learn the teacher network's knowledge and enable to get better performance than teacher network robustly. As a result, it outperforms hint training and KD in image classification problem. Also, since the intermediate representation of the teacher network is directly transferred to the student network, it is possible to compress GAN generators. Similar to the result of image classification, the performance of the student network of GAN generator outperforms the student network in both scores and generated image quality.

Pros:
- Easy to understand
- It defines the problem clearly (hint training and KD are not effective for the deep networks) and tries to solve it.
- Simple but now the idea for model compression.
- Especially, using teacher network’s intermediate output directly as an input to the student network is an interesting and good idea.
- The result of sequential compression is impressive.

Minor issues and questions:
- The width of the CNN greatly affects the computational cost of the network. It is possible to reduce the with of the network with the same approach?
- During compression, are you using the training set same as the teacher network?
- In Figure 2, the result of KD almost always beats the teacher network performance. Is this result is general in KD method? Because in original KD paper [1], it shows that the distilled single model performs a little bit worse than the teacher 10xEnsemble model.

[1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating: 4/5
Confidence: 2/3

The paper introduced a method to transfer knowledge from intermediate representation from a teacher model to a student model, outperforming [9] and compressing GANs' generators

The intuition behind the idea is solid and worth exploring. The efforts done are adequate and the amount and representation of the experiments are very sufficient.

Pros:

- Simple idea and easy to follow
- Outperforms competitive methods

Cons:

- The proof of line 36 claim ("We hypothesize that, for modern deep networks, hint training causes unstable IRs") is not clear or unproven.


[9] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. ICLR, 2015.