Paper:
ADAPTIVE QUANTIZATION OF NEURAL NETWORKS
Despite the state-of-the-art accuracy of Deep Neural Networks (DNN) in various classification problems, their deployment onto resource constrained edge computing devices remains challenging due to their large size and complexity. Several recent studies have reported remarkable results in reducing this complexity through quantization of DNN models. However, these studies usually do not consider the changes in the loss function when performing quantization, nor do they take the different importances of DNN model parameters to the accuracy into account. We address these issues in this paper by proposing a new method, called adaptive quantization, which simplifies a trained DNN model by finding a unique, optimal precision for each network parameter such that the increase in loss is minimized. The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each parameter and assigns it a precision accordingly. Since this problem uses linear functions, it is computationally cheap and, as we will show, has a closed-form approximate solution. Experiments on MNIST, CIFAR, and SVHN datasets showed that the proposed method can achieve near or better than state-of-the-art reduction in model size with similar error rates. Furthermore, it can achieve compressions close to floating-point model compression methods without loss of accuracy.

Review:
Revised Review:
The authors have addressed most of my concerns with the revised manuscript. I now think the paper does just enough to warrant acceptance, although I remain a bit concerned that since the benefits are only achievable with customized hardware, the relevance/applicability of the work is somewhat limited.
Original Review:
The paper proposes a technique for quantizing the weights of a neural network, with bit-depth/precision varying on a per-parameter basis. The main idea is to minimize the number of bits used in the quantization while constraining the loss to remain below a specified upper bound. This is achieved by formulating an upper bound on the number of bits used via a set of "tolerances"; this upper bound is then minimized while estimating any increase in loss using a first order Taylor approximation.
I have a number of questions and concerns about the proposed approach. First, at a high level, there are many details that aren't clear from the text. Quantization has some bookkeeping associated with it: In a per-parameter quantization setup it will be necessary to store not just the quantized parameter, but also the number of bits used in the quantization (takes e.g. 4-5 extra bits), and there will be some metadata necessary to encode how the quantized value should be converted back to floating point (e.g., for 8-bit quantization of a layer of weights, usually the min and max are stored). From Algorithm 1 it appears the quantization assumes parameters in the range [0, 1]. Don't negative values require another bit? What happens to values larger than 1? How are even bit depths and associated asymmetries w.r.t. 0 handled (e.g., three bits can represent -1, 0, and 1, but 4 must choose to either not represent 0 or drop e.g. -1)? None of these details are clearly discussed in the paper, and it's not at all clear that the estimates of compression are correct if these bookkeeping matters aren't taken into account properly.
Additionally the paper implies that this style of quantization has benefits for compute in addition to memory savings. This is highly dubious, since the method will require converting all parameters to a standard bit-depth on the fly (probably back to floating point, since some parameters may have been quantized with bit depth up to 32). Alternatively custom GEMM/conv routines would be required which are impossible to make efficient for weights with varying bit depths. So there are likely not runtime compute or memory savings from such an approach.
I have a few other specific questions: Are the gradients used to compute \mu computed on the whole dataset or minibatches? How would this scale to larger datasets? I am confused by the equality in Equation 8: What happens for values shared by many different quantization bit depths (e.g., representing 0 presumably requires 1 bit, but may be associated with a much finer tolerance)? Should "minimization in equation 4" refer to equation 3?
In the end, while do like the general idea of utilizing the gradient to identify how sensitive the model might be to quantization of various parameters, there are significant clarity issues in the paper, I am a bit uneasy about some of the compression results claimed without clearer description of the bookkeeping, and I don't believe an approach of this kind has any significant practical relevance for saving runtime memory or compute resources.