Paper:
Unlike traditional approaches that focus on the quantization at the network level, in this work we propose to minimize the quantization effect at the tensor level. We analyze the trade-off between quantization noise and clipping distortion in low precision networks. We identify the statistics of various tensors, and derive approximate analytical expressions for the mean-square-error degradation due to clipping. By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping. For example, just by choosing the accurate clipping values, more than 40% accuracy improvement is obtained for the quantization of VGG16-BN to 4-bits of precision. Our results have many applications for the quantization of neural networks at both training and inference time. One immediate application is for a rapid deployment of neural networks to low-precision accelerators without time-consuming fine tuning or the availability of the full datasets.

Review:
This paper derives a formula for finding the minimum and maximum clipping values for uniform quantization which minimize the square error resulting from quantization, for either a Laplace or Gaussian distribution over pre-quantized value. This seems like too small a contribution to warrant a paper. I wasn't convinced that appropriate baselines were used in experiments. There were a number of statements that I believed to be technically slightly incorrect. There were also some small language problems (though these didn't hinder understanding).
more specific comments:
abstract:
"derive exact expressions" -- these expressions aren't exact. they turn out to be based on a piecewise zeroth order Taylor approximation to the density.
main paper:
"allow fit bigger networks into" -> "allow bigger network to fit into"
"that we are need" -> "that need"
"introduces an additional" -> "introduces additional"
clippig -> clipping
it's not clear a-priori that information loss is the property to minimize that maximizes performance of the quantized network.
"distributions of tensors" -> "distribution of tensor elements"
this comment also applies in a number of other places, where the writing refers to the marginal distribution of values taken on by entries in a tensor as the distribution over the tensor. note that a distribution over tensors is a joint distribution over all entries in a tensor. e.g. it would capture things like eigenvalues, entry-entry covariance, rather than just marginal statistics.
"than they could have by working individually" -> "than could have been achieved by each individually"
Why the focus on small activation bit depth? I would imagine weight bit-depth was more important than activation bit depth. Especially since you're using ?32-bit? precision in the weight/activations multiplications, so activations are computed at a high bit depth anyways.
Table 1: Give absolute accuracies too! Improvement relative to what baseline?
sec 2:
sufficeint -> sufficient
\citep often used when it should instead be \citet.
"As contrast" -> "In contrast"
section 3:
uniformity -> uniformly
I don't believe the notion of p-value is being used correctly here w.r.t. the Kolmogorov-Smirnov test.
Figure 1: The mean square error should never go to 0. This suggests something is wrong. If it's just a scaling issue, consider a semilogy plot.
Figure 2: I'm unclear what baseline (no clipping) refers to in terms of clipping values. For uniform quantization there needs to be some min and max value.