From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper44/Authors
comment: Dear reviewer, 

Thank you very much for your valuable comments and suggestions. We appreciate it and will address all of them in our revision. Below is our response to each of your comment. 

1. The quantization term in our loss function is just a constraint boundary of the primal objective. In order to get an optimal G(alpha), the gradient of G and the gradient of constraint (in equation 3) will eventually be parallel and opposing. Given it can’t move along the constraint boundary, it will converge.

We also measured the convergence time in our experiments. The testbed of our experimental evaluation is 1 Titan XP and the convergence time for VGG-16 on CIFAR-10 is about 4 hours. The convergence time for 2-layer CNNs with 2 layers Fully-connected layer on MNIST is about 1 hour. The above experiments have very high accuracy requirements (i.e., almost the same accuracy as the original full precision model). If the accuracy requirements are lower, the convergence time would be shorter. We will add a discussion of the convergence time in the revision.

2.-3. Thank you for pointing out these two related works and we will cite them in our revision. Compared to these two works, our key new contribution is to relax the search space of quantization bitwidth from discrete to a continuous domain and propose a new learning process to automatically find the best bitwidth for each layer without any hand tuning. 

In addition, our approach is also much more lightweight compared to R1 and R2. R1 requires calculating accuracy difference and adversarial noise and R2 requires computing the entropy of weights, both of which can drastically increase the complexity of the training procedure and encounter significant time overhead. Our method is end-to-end, i.e., updating quantization scheme and weights at the same time, so it encounters much less compression time overhead.

As for the comparison of experimental evaluation results. according to Figure 6(b) in R1, their method can compress VGG-16 from 120Mbits to around 83Mbits while our method can compress up to 4Mbits (30X) with almost no accuracy drop. Comparing with R2, our accuracy on VGG-16 is close to full precision with 30X compression rate as shown at table 1 while their method achieves 27.5X on ResNet-20 (we used different models for evaluation but the compression rate is comparable. Recall that our approach is much more lightweight and requires much less compression time).
 
We will clarify all the above in revision.

Our method can combine with [R1][R2]: once the bitwidth is identified by our method, we can apply [R1][R2] to compress them.

4. Thank you for pointing this out. This is a typo and we will fix it in revision.

Thank you again for the review, we hope this response addresses your questions well.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper44/Authors
comment: Dear reviewer, 

Thank you very much for your valuable comments and suggestions. We appreciate it and will address all of them in our revision. Below is our response to each of your comment. 

1. We use "accuracy sensitivity of quantization" for describing different accuracy drop caused by different quantization schemes. Since such term may cause confusion, we will use different wording in revision.

2.  “theta” is a tunable hyperparameter representing the tolerance of accuracy drop. Larger “theta” tolerates less accuracy drop but may also result in smaller compression rate. While smaller “theta” can potentially achieve higher compression rate, it may also potentially cause larger accuracy drop. A rule of thumb setting of “theta” is using the (expected or target) loss in full precision model. In our preliminary results, we set theta as 0.6 for VGG-16 as the VGG-16 full precision model achieves loss of 0.6. We will add a sensitivity analysis of “theta” in revision. 
 
3. Binary quantization is the most efficient quantization in terms of compression rate. However, it usually results in large accuracy drop. Higher bit quantization (e.g., the 8-bit example used in the paper) usually causes less accuracy drop but also loses some compression rate. In order to better explore the trade-off between accuracy impact and compression rate and strike a better balance, we introduce the mixed precision scheme, which determines the best quantization scheme for each layer instead of the entire model. 

We agree that 8-bit quantization can do the job of binary quantization. However, it will lose the opportunity to achieve a better compression rate. Our mixed precision refers to using different precision for different layers. This is because different quantization schemes may have different accuracy impact on different layers, so each layer has its own precision preference. We will clarify this in revision.

4. You are right, this step is not necessary. We use such retraining step is for the purpose of making sure that the model is stable at the current learned value of alpha and weights. We will clarify this in revision.

5. Our quantization method can compress VGG-16 model on CIFAR-10 data with 30X compression rate as shown at table 1. The state-of-the-art pruning result can also compress the VGG-16 on CIFAR-10 at about 30X compression rate [10][11]. Since our quantization method is complementary to pruning, we can combine our quantization approach with the pruning method. Therefore, the total compression rate is 30 times 30, which is 900. Figure 1 of [ER1] can provide a better visualized understanding. In addition, such way of calculation is adopted by several publications, such as [11]. We will clarify this in revision.

[ER1] Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv:1510.00149 (2015).

Thanks again for your valuable time and insightful comments. We will update accordingly based on your comments in the revision.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper44/Authors
comment: Dear reviewer, 

Thank you very much for your valuable comments and suggestions. We appreciate it and will address all of them in our revision. Below is our response to each of your comment. 

1. Equation 1 shows how we relax the discrete space of quantization scheme to be continuous via softmax function as it is in DARTS. Here q_{ij} denotes the quantized output for layer i, where j denotes the quantization method—0 for binary and 1 for 8-bit. We also applied batch normalization to enhance the performance, and B(q_{ij}) denotes the result after batch norm. In fact, this equation is almost the same as Equation 2 in DARTS. Since the purpose of adopting this equation is simply relaxing the search space to be continuous for the implementation of gradient descent algorithm, we think Equation 1 is correct for its capability of making continuous relaxation. It is also worth mentioning that our approach slightly differs from the continuous relaxation technique in DARTS because we incorporated batch normalization in our Equation 1. We make this change out of the reason that our 8-bit quantization includes a scaling process while binary quantization doesn’t, making the distribution of binary and 8-bit’s results different which needs to be fixed by batch normalization.

2. In our paper, we calculated the product of the parameter number of each layer and the corresponding quantization scheme bit number. We take the sum of the product of each layer as our model size and convert it into mb.

3. “theta” is a tunable hyperparameter representing the tolerance of accuracy drop. Larger “theta” tolerates less accuracy drop but may also result in smaller compression rate. While smaller “theta” can potentially achieve a higher compression rate, it may also potentially cause larger accuracy drop. A rule of thumb setting of “theta” is using the (expected or target) loss in full precision model. In our preliminary results, we set theta as 0.6 for VGG-16 as the VGG-16 full precision model achieves loss of 0.6. We will add a sensitivity analysis of “theta” in revision. 

4. It is correct that in our paper we set \zeta to 0. In our paper, we didn't try non-zero \zeta because our search space is far less than that of DARTS and a zero \zeta can produce fair results. To improve the algorithm, using non-zero is worth trying. We will do this experiment in future work.

Thanks again for your valuable time and insightful comments. We will update accordingly based on your comments in the revision.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper44/AnonReviewer2
title: Promising approach for finding quantizations
review: This paper proposes searching for quantization rates layer-by-layer using a differential cost function. This paper is very similar to DARTS algorithm used for architecture search in Liu et. al (2018) and we would argue that is an application of their ideas. We think this is fantastic application of DARTS in finding quantization rates and is a promising research direction. However the work is still in much early stages and needs further empirical studies. Some points that need additional clarification:
- As states, Equation 1 does not appear to be correct - please clarify.  
- It is not clear how the model size objective function is computed. 
- It is not clear how theta is computed
- In DARTS, when updating the model architecture, a virtual learning rate is used to update the gradient of the weights as well on the loss (see equation 5 in Liu et. al (2018)). As discussed in their paper, a first-order approximation sets \zeta to 0 and they empirically found that did not give good results. It appears in this work that \zeta is set to 0. Is this correct? If no, what are the differences between the proposed algorithm and DARTS? If yes, how do the results compare for non-zero \zeta ?

While the results at this time are limited to two datasets, the approach is promising and sheds light into building more compressed networks. 

rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper44/AnonReviewer1
title: The paper is interesting particularly because it combines the quantization problem with the loss function and tries to solve a new optimization problem. However, there are some concerns.
review: The paper aims at maintaining the accuracy of the network while compressing it, by proposing a method of weight quantization. This method uses a softmax function that takes different quantized weights and assigns different weights (alpha) to them. An optimization algorithm combines this constraint with the loss function of the network, and finds the weights alpha. The paper is interesting particularly because it combines the quantization problem with the loss function and tries to solve a new optimization problem. However, there are some concerns:

1- In lines 6 and 49, the authors talk about sensitivity of the accuracy to quantization, however, what they did was assigning different weights (alpha) to each quantized weight by optimizing a constrained function. This does not necessarily mean sensitivity to quantization, but only implies different weights. Sensitivity analysis usually needs the derivative of the output with respect to changes in the weights, however, here, alpha is the outcome of some constrained optimization.

2- How did the authors determine theta in equation 3?

3- It is not clear why the approach needs a mixed precision scheme. For example an 8-bit quantization can do the job of a binary quantization, and it is not clear why it should be combined with a binary quantization in equation 1. I see that the authors used that to get a continuous q_i, but it seems redundant to use a mixed quantization scheme. I might have missed something here. 

4- At the end of section 2, the authors retrain the network after solving for alpha. It is not clear to me why retraining is needed. After retraining, if they end up with a different set of weights, they will need to adjust alpha again, and do this iterative process until both weights and alpha's converge. However, if the weights don't change after retraining, it seems that this step is not necessary. 

5- At line 87, how did the authors come up with the factor 900? 
rating: 3: Marginally above acceptance threshold
confidence: 1: The reviewer's evaluation is an educated guess

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper44/AnonReviewer3
title: Although the paper is technically sound, the current version is lacking in several aspects and needs extensive revisions.
review: This paper proposes a differentiable mixed-precision search method for compressing deep neural networks efficiently, in which the search space of quantization bit width is relaxed from discrete to a continuous domain. The method denotes the output of layers with continuous relaxation by using the SoftMax function, and then converts the problem into a continuous and differentiable optimization problem by a trade-off between different quantization schemes. Then the authors use gradient descent to get a best trade-off parameter set. Some experimental results show the effectiveness of the proposed algorithms.

1. When converting this problem into a optimization problem, will it need a lot of time overhead or even cannot converge? especially when needing a higher accuracy (smaller maximum loss). There is no analysis of convergence time in this paper.

2. In the sections of abstract and introduction, this paper emphasizes that “We observe that different layers may have different accuracy sensitivity of quantization”. In fact, there is already some related research work, for example [R1, R2].

3. The experimental results are less convincing. Can the algorithm converge on large popular datasets? In particular, the authors should compare the proposed algorithms with more state-of-the-art methods such as the methods mentioned above.

4. In line60, does the paper mean “a network with k layers is alpha = {alpha_0, …, alpha_k-1}”?

[R1]Yiren Zhou, Seyed-Mohsen, Moosavi-Dezfooli, et al. Adaptive Quantization for Deep Neural Network (arXiv:1712.01048v1)
[R2]Xiaotian Zhu, Wengang Zhou, et al. Adaptive Layerwise Quantization for Deep Neural Network Compression, ICME 2018.

rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct