Paper:
The tremendous memory and computational complexity of Convolutional Neural Networks (CNNs) prevents the inference deployment on resource-constrained systems. As a result, recent research focused on CNN optimization techniques, in particular quantization, which allows weights and activations of layers to be represented with just a few bits while achieving impressive prediction performance. However, aggressive quantization techniques still fail to achieve full-precision prediction performance on state-of-the-art CNN architectures on large-scale classification tasks. In this work we propose a method for weight and activation quantization that is scalable in terms of quantization levels (n-ary representations) and easy to compute while maintaining the performance close to full-precision CNNs. Our weight quantization scheme is based on trainable scaling factors and a nested-means clustering strategy which is robust to weight updates and therefore exhibits good convergence properties. The flexibility of nested-means clustering enables exploration of various n-ary weight representations with the potential of high parameter compression. For activations, we propose a linear quantization strategy that takes the statistical properties of batch normalization into account. We demonstrate the effectiveness of our approach using state-of-the-art models on ImageNet.

Review:
Summary: This paper proposes a technique for quantizing the weights and activations of a CNN. The main contribution is in replacing the heuristic to find good quantization intervals of (Zhu et al, 2016) with a different heuristic based on a hierarchical clustering algorithm, and empirically validating its effectiveness.
Strenghts:
- The proposed nested-means heuristic is simple and makes sense intuitively.
- The experiments on two modern architectures seem solid and demonstrate good empirical performance.
Weaknesses:
- The main weakness is the limited novelty of this paper. The proposed setup is almost identical to the one in (Zhu et al, 2016), except for the replacement of the heuristic to find quantization intervals with another one. While the experiments demonstrate the empirical effectiveness of the method as a whole, what is missing is a direct, controlled comparison between the original heuristic and the proposed one. Now it is hard to tell whether the accuracy increases are obtained through the proposed adaptation or because of other factors such as a better implementation or longer training.
- In section 4, it is not made clear whether the activations are quantized according to the same scheme as the weights (apart from the issue of selecting a good clipping interval, which is addressed).
- The paper is a bit short on references, considering the many recent works on quantized neural networks.
Minor comments and questions:
- The wording is sometimes imprecise, making some arguments hard to follow. Two examples:
-- "Lowering the learning rate for re-training can diminish heavy changes in the weight distribution, at the cost of longer time to converge and the risk to get stuck at plateau regions, which is especially critical for trainable scaling factors"
-- "This approach is beneficial because it defines cluster thresholds which are influenced by large weights that were shown to play a more important role than smaller weights (Han et al., 2015b)"
- The title says "for compression and inference acceleration", so it would be nice if the paper reports some compression and timing metrics in the experiments section.
- The notation in section 3.1 overly complicated, could probably be simplified a bit for readability.
- Section 3.3: "However, having an additional hyperparameter t_i for each scaling factor alpha_i renders the mandatory hyperparameter tuning infeasible." -> From section 4.2 in (Zhu et al, 2016), I believe the constant factor t is shared across all layers, making it only a single hyperparameter.
- Last paragraph of section 4: "(Cai et al., 2017) experimentally showed that the pre-activation distribution after batch normalization are all close to a Gaussian with zero mean and unit variance. Therefore, we propose to select a fixed clipping parameter gamma.". -> But what about the activations *before* the batchnorm layer where the assumption of zero mean and unit variance does not hold?