Paper:
Network quantization is a model compression and acceleration technique that has become essential to neural network deployment. Most quantization methods perform fine-tuning on a pretrained network, but this sometimes results in a large loss in accuracy compared to the original network. We introduce a new technique to train quantization-friendly networks, which can be directly converted to an accurate quantized network without the need for additional fine-tuning. Our technique allows quantizing the weights and activations of all network layers down to 4 bits, achieving high efficiency and facilitating deployment in practical settings. Compared to other fully quantized networks operating at 4 bits, we show substantial improvements in accuracy, for example 66.68% top-1 accuracy on ImageNet using ResNet-18, compared to the previous state-of-the-art accuracy of 61.52% Louizos et al. (2019) and a full precision reference accuracy of 69.76%. We performed a thorough set of experiments to test the efficacy of our method and also conducted ablation studies on different aspects of the method and techniques to improve training stability and accuracy. Our codebase and trained models are available on GitHub.

Review:
This work introduces GQ-Net, a novel technique that trains quantization friendly networks that facilitate for 4 bit weights and activations. This is achieved by introducing a loss function that consists of a linear combination of two components: one that aims to minimize the error of the network on the training labels of the dataset and one that aims to minimize the discrepancy of the model output with respect to the output of the model when the weights and activations are quantized. The authors argue that this has the effect of “guiding” the optimization procedure in finding networks that can be quantized without loss of performance. For the discrepancy metric the authors use the KL divergence from the predictive distribution of the floating point model to the one of the quantized model. The authors then propose several extra techniques that boost the performance of their method: 1. scheduling the weighting coefficients of the two loss terms (something which reminisces iterative pruning methods), 2. stopping the gradient of the floating point model w.r.t. the second loss term, 3. learning the parameters of the uniform quantizer, 4. alternating optimization between the weights and the parameters of the quantizers and 5. using separate batch normalization statistics for the floating point and quantized models. The authors then evaluate their method on Imagenet classification using ResNet-18 and Mobilenet v1 / v2, while also performing an ablation study about the extra tricks that they propose.
This work is well written and in general conveys the main idea in an effective manner. Quantization friendly neural networks in an important subject in order to make deep learning tractable for real world applications. The idea seems on a high level to be interesting and simple; train floating point models that can fit the data well while also encouraging them to be robust to quantization by enforcing the predictive distributions of the fixed and floating point models to be similar in the KL-divergence sense. Nevertheless, I do have some comments that would hopefully help in improving this work:
- It does seem that GQ-Nets need extra tricks in order to perform well, and those tricks come with their own set of hyperparameters that need to be tuned. For example, at section 4.3 you mention that the top-1 accuracy of vanilla GQ-Nets is 60.95, which is lower than the RelaxedQuant baseline (that has 61.52). This raises the question whether the boost in performance is due to the several additional steps employed (which in general can be applied to other quantization techniques as well), and not due to the main idea itself. 
- Do you employ the straight-through estimator (STE) for the weights in the L_q objective? In the second paragraph of the second page you argue that due to the biased gradients of STE the performance is in general reduced, so I was wondering whether STE posed an issue there or whether you used an alternative estimator. 
- How is batch normalization handled? Do you absorb the scale and shifts in the weights / biases before you perform quantization or do you quantize the weights and then apply the BN scale and shift in full precision?
- How do you ensure and ub > lb when you learn the quantizer? In general learning the quantizer can be also done with alternative techniques (e.g. simply learning the scale and offset) so I was wondering whether you noticed benefits from using the ub, lb parametrization compared to others.
- Do you show the pre-quantization distributions at Figure 2b? In the caption you mention quantized but the resolution seems to be higher than the 16 values you should get with 4 bits. Furthermore, it should be noted that the discrepancy in BN in quantized models was, as far as I am aware, firstly noticed at [1] (and subsequently at RelaxedQuant) and both of these methods simply re-estimated the moving averages during the inference time.
Overall, I am on the fence about this work and tend to reject. Having said that, I am of course willing to revise my score after the discussions with the authors / other reviewers.
[1] Probabilistic Binary Neural Networks, Jorn W.T. Peters, Max Welling