From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper34/Authors
comment: We thank the reviewer for the feedback. 

"The networks used in the experiment part are somewhat shallow. Some deeper networks and some complex datasets can be used to illustrate the abilities of the proposed models further." 

In this paper, we used CNNs with seven convolutional layers to do the experiments on CIFAR-10. While this is shallower than VGG, ResNet, etc., it achieves fairly good accuracy (>92.5%) on CIFAR-10 since even full-precision ResNet-32 only achieves 92.49% accuracy on CIFAR-10 [1]. To verify that FLightNN can also work on different types of networks, we also did experiment on wide ResNet-18 (whose largest convolutional layer has 256 filters). The accuracy results for full-precision model, LightNN-2 and FLightNN are 93.1%, 92.8%, 92.3%, respectively, while the model storage for them are 10.8MB, 2.7MB, and 1.4MB, respectively. In terms of runtime, FLightNN achieves 2x speedup compared to LightNN-2. 

We also conducted preliminary experiments on ImageNet dataset. Due to time limit, we trained LightNN-1 on AlexNet and VGG-16, and achieved 58.6% and 71.8% top-1 accuracy on these two models, respectively. This means that LightNN-1 can achieve slightly higher accuracy than full-precision model (56.6% [2]) on AlexNet, and only 1.2% lower accuracy than full-precision model (73.0% [3]) on VGG-16. 

Thanks again for providing the review! Please let us know if you have any questions. 

[1] He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In NIPS, 2012.
[3] Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper34/Authors
comment: We thank the reviewer for thoroughly understanding our paper and we are grateful that the reviewer points our the math expression problem we missed. We have updated the paper based on the comments.

"1. Between Line 59 and 60, r_{i,j} is real value (or vector)? Should that further be quantized? And what does it mean to multiply r_{i,j}? As r_{i,j}R(r_{i,j}) would possibly expand the original value."
Thanks for finding this typo. The r_{i,j} is a vector, and we have removed the extra r_{i,j} in line 59-60 in the revised version. 

"2. In Eq.(1), when computing $\partial w / \partial t_2 $, shouldn't add the partial derivative of $\partial w / \partial t_1 $ ? Note that t_2  is related with r_1, while r_1 is derived from t_1."
Thanks for raising the question, it is really a nice catch. Yes, the residual r is also a function of thresholds t. We have fixed this in the revised paper. Also, we note that our code implementation is correct since we first define the gradient for indicator function and rounding function, and used PyTorch auto-differentiation to compute the gradient. 

Thanks again for providing the review! Please let us know if you have any questions.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper34/AnonReviewer2
title: Two Concerns
review: This work extends LightNN by extra learning a threshold to choose the number of powers of 2, i.e., $k$ in the paper. Under this modification, the method attains more flexible quantization. The experiments show that it can achieve 2X speedup on FPGAs with almost no accuracy degradation.

There are two parts in the paper that are unclear to me:

1. Between Line 59 and 60, r_{i,j} is real value (or vector)? Should that further be quantized? And what does it mean to multiply r_{i,j}? As r_{i,j}R(r_{i,j}) would possibly expand the original value.
2. In Eq.(1), when computing $\partial w / \partial t_2 $, shouldn't add the partial derivative of $\partial w / \partial t_1 $ ? Note that t_2  is related with r_1, while r_1 is derived from t_1.  

It is very welcomed for the authors to clarify the above two concerns.

rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper34/Authors
comment: We thank the reviewers for the feedback. 

“It is not entirely clear from the paper how superior performance (Fig. 2) is achieved; is this simply the nature of the Pareto front given LightNN-style quantization or there is more to this particular method?  The stated motivation seems to be simply getting finer granularity in attainable solutions along the Pareto front between LightNN-1 and LightNN-2.”

Thank you for raising this insightful question. Our method is more than simply interpolating the Pareto front, which can be observed from Figure 2(a)(b) where the fastest FLightNN outperforms LightNN-1 in both accuracy and speed (which means our training algorithm creates a better Pareto front). We have re-run the experiments for several times and observed the same phenomenon. We conjecture that the accuracy improvement brought by FLightNN comes from imposing weight constraints gradually. More specifically, we set the initial value of the thresholds t to 0, and while training, the thresholds will change. We have analyzed the thresholds per iteration, and found that they are gradually increasing to some value with slight fluctuations. This means that FLightNN is gradually imposing the quantization constraint due to the nature of its differentiable training for both weights and thresholds. This phenomenon has also been observed by prior work [1,2] which shows that gradually imposing quantization constraints can achieve better accuracy than directly quantizing with a strict constraint. For completeness, we have added them in the manuscript along with a brief discussion.

In terms of runtime, FLightNN provides more continuous points between LightNN-1 and LightNN-2. However, all of them are faster than the generally studied fixed-point quantization on FPGA. A short answer for this is that shift operators are more lightweight than multipliers when implemented in hardware. More concretely, the shift operators use look-up table (LUT) units, while multipliers use digital signal processing (DSP) units on FPGA. The DSPs are much more expensive resources than LUTs in FPGA, since DSPs take much more area. A DSP contains ~1000x more gates than a LUT. For example, the FPGA (Xilinx ZC706) used in this paper has 218k LUTs, but only has 900 DSPs. Therefore, the parallelization of fixed-point DNN is bounded by DSP resources, making it slower than the FLightNN. 

“Typo on line 4: "can"”

Thank you for pointing out the typo. We have fixed it in the new version. 

[1] Zhou, Aojun, et al. "Incremental network quantization: Towards lossless cnns with low-precision weights." ICLR, 2017. 
[2] Dong, Yinpeng, et al. "Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization." BMVC, 2017.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper34/AnonReviewer3
title: No Title
review: 
Summary: 
LightNN constrains the weights of DNNs to be a sum of a limited number of powers of 2, which has the advantages on speed and storage. The model has been used in FPGA.

Pros:
1. The authors propose a novel differentiable training for quantization. The model gets good results on CIFAR10 dataset in term of runtime and storage.
2. The authors implement and synthesize the models on Xilinx FPGA and measure the runtime of batched inference. 

Cons:
The networks used in the experiment part are somewhat shallow. Some deeper networks and some complex datasets can be used to illustrate the abilities of the proposed models further.
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper34/AnonReviewer1
title:  
review: Summary: a straight-through differentiable LightNN quantization method, validated with FPGA implementation

Pros:
- Clear presentation
- Good results

Cons:
- Experiments relatively few, but okay for a short workshop paper.
- It is not entirely clear from the paper how superior performance (Fig. 2) is achieved; is this simply the nature of the Pareto front given LightNN-style quantization or there is more to this particular method?  The stated motivation seems to be simply getting finer granularity in attainable solutions along the Pareto front between LightNN-1 and LightNN-2.  

Typo on line 4: "can"
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct