Original: R ELAXED Q UANTIZATION FOR D ISCRETIZED N EURAL N ETWORKS
Revised: R ELAXED Q UANTIZATION FOR D ISCRETIZED N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices.
Revised: Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices.

Original: In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure.
Revised: In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure.

Original: Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid.
Revised: Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid.

Original: These are subsequently relaxed to continuous surrogates that can allow for efﬁcient gradient-based optimization.
Revised: These are subsequently relaxed to continuous surrogates that can allow for efﬁcient gradient-based optimization.

Original: We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent.
Revised: We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent.

Original: We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classiﬁcation.
Revised: We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classiﬁcation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks excel in a variety of large scale problems due to their highly ﬂexible parametric nature.
Revised: Neural networks excel in a variety of large scale problems due to their highly ﬂexible parametric nature.

Original: However, deploying big models on resource constrained devices, such as mobile phones, drones or IoT devices is still challenging because they require a large amount of power, memory and computation.
Revised: However, deploying big models on resource constrained devices, such as mobile phones, drones or IoT devices is still challenging because they require a large amount of power, memory and computation.

Original: Neural network compression is a means to tackle this issue and has therefore become an important research topic.
Revised: Neural network compression is a means to tackle this issue and has therefore become an important research topic.

Original: Neural network compression can be, roughly, divided into two not mutually exclusive categories: pruning and quantization.
Revised: Neural network compression can be, roughly, divided into two not mutually exclusive categories: pruning and quantization.

Original: While pruning (LeCun et al., 1990; Han et al., 2015) aims to make the model “smaller” by altering the architecture, quantization aims to reduce the precision of the arithmetic operations in the network.
Revised: While pruning (LeCun et al., 1990; Han et al., 2015) aims to make the model “smaller” by altering the architecture, quantization aims to reduce the precision of the arithmetic operations in the network.

Original: In this paper we focus on the latter.
Revised: In this paper we focus on the latter.

Original: Most network quantization methods either simulate or enforce discretization of the network during training, e.g. via rounding of the weights and activations.
Revised: Most network quantization methods either simulate or enforce discretization of the network during training, e.g. via rounding of the weights and activations.

Original: Although seemingly straighforward, the discontinuity of the discretization makes the gradient-based optimization infeasible.
Revised: Although seemingly straighforward, the discontinuity of the discretization makes the gradient-based optimization infeasible.

Original: The reason is that there is no gradient of the loss with respect to the parameters.
Revised: The reason is that there is no gradient of the loss with respect to the parameters.

Original: A workaround to the discontinuity are the “pseudo-gradients” according to the straight-through estimator (Bengio et al., 2013), which have been successfully used for training low-bit width architectures at e.g. Hubara et al.
Revised: A workaround to the discontinuity are the “pseudo-gradients” according to the straight-through estimator (Bengio et al., 2013), which have been successfully used for training low-bit width architectures at e.g. Hubara et al.

Original: ; Zhu et al.
Revised: ; Zhu et al.

Original: The purpose of this work is to introduce a novel quantization procedure, Relaxed Quantization (RQ).
Revised: The purpose of this work is to introduce a novel quantization procedure, Relaxed Quantization (RQ).

Original: RQ can bypass the non-differentiability of the quantization operation during training by smoothing it appropriately.
Revised: RQ can bypass the non-differentiability of the quantization operation during training by smoothing it appropriately.

Original: The contributions of this paper are four-fold: First , we show how to make the set of quantization targets part of the training process such that we can optimize them with gradient descent.
Revised: The contributions of this paper are four-fold: First , we show how to make the set of quantization targets part of the training process such that we can optimize them with gradient descent.

Original: Second , we introduce a way to discretize the network by converting distributions over the weights and activations to categorical distributions over the quantization grid.
Revised: Second , we introduce a way to discretize the network by converting distributions over the weights and activations to categorical distributions over the quantization grid.

Original: Third , we show that we can obtain a “smooth” quantization procedure by replacing the categorical distributions with concrete (Maddison et al., 2016; Jang et al., 2016) equivalents.
Revised: Third , we show that we can obtain a “smooth” quantization procedure by replacing the categorical distributions with concrete (Maddison et al., 2016; Jang et al., 2016) equivalents.

Original: Finally we show that stochastic rounding (Gupta et al., 2015), one of the most popular quantization techniques, can be seen as a special case of the proposed framework.
Revised: Finally we show that stochastic rounding (Gupta et al., 2015), one of the most popular quantization techniques, can be seen as a special case of the proposed framework.

Original: We present the details of our approach in Section 2, discuss related work in Section 3 and experimentally validate it in Section 4.
Revised: We present the details of our approach in Section 2, discuss related work in Section 3 and experimentally validate it in Section 4.

Original: Finally we conclude and provide fruitful directions for future research in Section 5.
Revised: Finally we conclude and provide fruitful directions for future research in Section 5.

Original: [Figure]
Revised: [Figure]

Original:  R ELAXED QUANTIZATION FOR DISCRETIZING NEURAL NETWORKS
Revised:  R ELAXED QUANTIZATION FOR DISCRETIZING NEURAL NETWORKS

Original: The central element for the discretization of weights and activations of a neural network is a quantizer q ( · ) .
Revised: The central element for the discretization of weights and activations of a neural network is a quantizer q ( · ) .

Original: The quantizer receives a (usually) continous signal as input and discretizes it to a countable set of values.
Revised: The quantizer receives a (usually) continous signal as input and discretizes it to a countable set of values.

Original: This process is inherently lossy and non-invertible: given the output of the quantizer, it is impossible to determine the exact value of the input.
Revised: This process is inherently lossy and non-invertible: given the output of the quantizer, it is impossible to determine the exact value of the input.

Original: One of the simplest quantizers is the rounding function:
Revised: One of the simplest quantizers is the rounding function:

Original: [Equation]
Revised: [Equation]

Original: where α corresponds to the step size of the quantizer.
Revised: where α corresponds to the step size of the quantizer.

Original: With α = 1 , the quantizer rounds x to its nearest integer number.
Revised: With α = 1 , the quantizer rounds x to its nearest integer number.

Original: Unfortunately, we cannot simply apply the rounding quantizer to discretize the weights and activations of a neural network.
Revised: Unfortunately, we cannot simply apply the rounding quantizer to discretize the weights and activations of a neural network.

Original: Because of the quantizers’ lossy and non-invertible nature, important information might be destroyed and lead to a decrease in accuracy.
Revised: Because of the quantizers’ lossy and non-invertible nature, important information might be destroyed and lead to a decrease in accuracy.

Original: To this end, it is preferable to train the neural network while simulating the effects of quantization during the training procedure.
Revised: To this end, it is preferable to train the neural network while simulating the effects of quantization during the training procedure.

Original: This encourages the weights and activations to be robust to quantization and therefore decreases the performance gap between a full-precision neural network and its discretized version.
Revised: This encourages the weights and activations to be robust to quantization and therefore decreases the performance gap between a full-precision neural network and its discretized version.

Original: However, the aforementioned rounding process is non-differentiable.
Revised: However, the aforementioned rounding process is non-differentiable.

Original: As a result, we cannot directly optimize the discretized network with stochastic gradient descent, the workhorse of neural network optimization.
Revised: As a result, we cannot directly optimize the discretized network with stochastic gradient descent, the workhorse of neural network optimization.

Original: In this work, we posit a “smooth” quantizer as a possible way for enabling gradient based optimization.
Revised: In this work, we posit a “smooth” quantizer as a possible way for enabling gradient based optimization.

Original: 2. L EARNING ( FIXED POINT ) QUANTIZERS VIA GRADIENT DESCENT
Revised: 2. L EARNING ( FIXED POINT ) QUANTIZERS VIA GRADIENT DESCENT

Original: The proposed quantizer comprises four elements: a vocabulary, its noise model and the resulting discretization procedure, as well as a ﬁnal relaxation step to enable gradient based optimization.
Revised: The proposed quantizer comprises four elements: a vocabulary, its noise model and the resulting discretization procedure, as well as a ﬁnal relaxation step to enable gradient based optimization.

Original: The ﬁrst element of the quantizer is the vocabulary: it is the set of (countable) output values that the quantizer can produce.
Revised: The ﬁrst element of the quantizer is the vocabulary: it is the set of (countable) output values that the quantizer can produce.

Original: In our case, this vocabulary has an inherent structure, as it is a grid of ordered scalars.
Revised: In our case, this vocabulary has an inherent structure, as it is a grid of ordered scalars.

Original: For ﬁxed point quantization the grid G is deﬁned as
Revised: For ﬁxed point quantization the grid G is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (1) where b is the number of available bits that allow for K = 2 b possible integer values.
Revised: (1) where b is the number of available bits that allow for K = 2 b possible integer values.

Original: By construction this grid of values is agnostic to the input signal x and hence suboptimal; to allow for the grid to adapt to x we introduce two free parameters, a scale α and an offset β .
Revised: By construction this grid of values is agnostic to the input signal x and hence suboptimal; to allow for the grid to adapt to x we introduce two free parameters, a scale α and an offset β .

Original: This leads to a learnable grid via ˆ G = α G + β that can adapt to the range and location of the input signal.
Revised: This leads to a learnable grid via ˆ G = α G + β that can adapt to the range and location of the input signal.

Original: The second element of the quantizer is the assumption about the input noise (cid:15) ; it determines how probable it is for a speciﬁc value of the input signal to move to each grid point.
Revised: The second element of the quantizer is the assumption about the input noise (cid:15) ; it determines how probable it is for a speciﬁc value of the input signal to move to each grid point.

Original: Adding noise to x will result in a quantizer that is, on average, a smooth function of its input.
Revised: Adding noise to x will result in a quantizer that is, on average, a smooth function of its input.

Original: In essense, this is an application of variational optimization (Staines & Barber, 2012) to the non-differentiable rounding function, which enables us to do gradient based optimization.
Revised: In essense, this is an application of variational optimization (Staines & Barber, 2012) to the non-differentiable rounding function, which enables us to do gradient based optimization.

Original: We model this form of noise as acting additively to the input signal x and being governed by a distribution p ( (cid:15) ) .
Revised: We model this form of noise as acting additively to the input signal x and being governed by a distribution p ( (cid:15) ) .

Original: This process induces a distribution p (˜ x )
Revised: This process induces a distribution p (˜ x )

Original: where ˜ x
Revised: where ˜ x

Original: = x + (cid:15) .
Revised: = x + (cid:15) .

Original: In the next step of the quantization procedure, we discretize p (˜ x ) according to the quantization grid ˆ G ; this neccesitates the evaluation of the cumulative distribution function (CDF).
Revised: In the next step of the quantization procedure, we discretize p (˜ x ) according to the quantization grid ˆ G ; this neccesitates the evaluation of the cumulative distribution function (CDF).

Original: For this reason, we will assume that the noise is distributed according to a zero mean logistic distribution with a standard deviation σ , i.e. L (0 , σ ) , hence leading to p (˜ x )
Revised: For this reason, we will assume that the noise is distributed according to a zero mean logistic distribution with a standard deviation σ , i.e. L (0 , σ ) , hence leading to p (˜ x )

Original: =
Revised: =

Original: L ( x, σ ) .
Revised: L ( x, σ ) .

Original: The CDF of the logistic distribution is the sigmoid function which is easy to evaluate and backpropagate through.
Revised: The CDF of the logistic distribution is the sigmoid function which is easy to evaluate and backpropagate through.

Original: Using Gaussian distributions proved to be less effective in preliminary experiments.
Revised: Using Gaussian distributions proved to be less effective in preliminary experiments.

Original: Other distributions are conceivable and we will brieﬂy discuss the choice of a uniform distribution in Section 2.3.
Revised: Other distributions are conceivable and we will brieﬂy discuss the choice of a uniform distribution in Section 2.3.

Original: The third element is, given the aforementioned assumptions, how the quantizer determines an appropriate assignment for each realization of the input signal x .
Revised: The third element is, given the aforementioned assumptions, how the quantizer determines an appropriate assignment for each realization of the input signal x .

Original: Due to the stochastic nature of ˜ x , a deterministic round-to-nearest operation will result in a stochastic quantizer for x .
Revised: Due to the stochastic nature of ˜ x , a deterministic round-to-nearest operation will result in a stochastic quantizer for x .

Original: Quantizing x in this manner corresponds to discretizing p (˜ x ) onto ˆ G and then sampling grid points
Revised: Quantizing x in this manner corresponds to discretizing p (˜ x ) onto ˆ G and then sampling grid points

Original: g i from it.
Revised: g i from it.

Original: More speciﬁcally, we construct a categorical distribution over the grid by adopting intervals of width equal to α centered at each of the grid points.
Revised: More speciﬁcally, we construct a categorical distribution over the grid by adopting intervals of width equal to α centered at each of the grid points.

Original: The probability of selecting that particular grid point will now be equal to the probability of ˜ x falling inside those intervals:
Revised: The probability of selecting that particular grid point will now be equal to the probability of ˜ x falling inside those intervals:

Original: [Equation]
Revised: [Equation]

Original: where ˆ x corresponds to the quantized variable, P ( · ) corresponds to the CDF and the step from
Revised: where ˆ x corresponds to the quantized variable, P ( · ) corresponds to the CDF and the step from

Original: Equation 2 to Equation 3 is due to the logistic noise assumption.
Revised: Equation 2 to Equation 3 is due to the logistic noise assumption.

Original: A visualization of the aforementioned process can be seen in Figure 1.
Revised: A visualization of the aforementioned process can be seen in Figure 1.

Original: For the ﬁrst and last grid point we will assume that they reside within ( g 0 − α/ 2 , g 0 + α/ 2] and ( g K − α/ 2 , g K + α/ 2] respectively.
Revised: For the ﬁrst and last grid point we will assume that they reside within ( g 0 − α/ 2 , g 0 + α/ 2] and ( g K − α/ 2 , g K + α/ 2] respectively.

Original: Under this assumption we will have to truncate p (˜ x ) such that it only has support within ( g 0 − α/ 2 , g K + α/ 2] .
Revised: Under this assumption we will have to truncate p (˜ x ) such that it only has support within ( g 0 − α/ 2 , g K + α/ 2] .

Original: Fortunately this is easy to do, as it corresponds to just a simple modiﬁcation of the CDF:
Revised: Fortunately this is easy to do, as it corresponds to just a simple modiﬁcation of the CDF:

Original: [Equation]
Revised: [Equation]

Original: Armed with this categorical distribution over the grid, the quantizer proceeds to assign a speciﬁc grid value to ˆ x by drawing a random sample.
Revised: Armed with this categorical distribution over the grid, the quantizer proceeds to assign a speciﬁc grid value to ˆ x by drawing a random sample.

Original: This procedure emulates quantization noise, which prevents the model from ﬁtting the data.
Revised: This procedure emulates quantization noise, which prevents the model from ﬁtting the data.

Original: This noise can be reduced in two ways: by clustering the weights and activations around the points of the grid and by reducing the logistic noise σ .
Revised: This noise can be reduced in two ways: by clustering the weights and activations around the points of the grid and by reducing the logistic noise σ .

Original: As σ → 0 , the CDF converges towards the step function, prohibiting gradient ﬂow.
Revised: As σ → 0 , the CDF converges towards the step function, prohibiting gradient ﬂow.

Original: On the other hand, if (cid:15) is too high, the optimization procedure is very noisy, prohibiting convergence.
Revised: On the other hand, if (cid:15) is too high, the optimization procedure is very noisy, prohibiting convergence.

Original: For this reason, during optimization we initialize σ in a sensible range, such that L ( x, σ ) covers a signiﬁcant portion of the grid.
Revised: For this reason, during optimization we initialize σ in a sensible range, such that L ( x, σ ) covers a signiﬁcant portion of the grid.

Original: Please confer Appendix A for details.
Revised: Please confer Appendix A for details.

Original: We then let σ be freely optimized via gradient descent such that the loss is minimized.
Revised: We then let σ be freely optimized via gradient descent such that the loss is minimized.

Original: Both effects reduce the gap between the function that the neural network computes during training time vs. test time.
Revised: Both effects reduce the gap between the function that the neural network computes during training time vs. test time.

Original: We illustrate this in Figure 2.
Revised: We illustrate this in Figure 2.

Original: The fourth element of the procedure is the relaxation of the non-differentiable categorical distribution sampling.
Revised: The fourth element of the procedure is the relaxation of the non-differentiable categorical distribution sampling.

Original: While we can use an unbiased gradient estimator via REINFORCE (Williams, 1992), we opt for a continuous relaxation due to high variances with REINFORCE.
Revised: While we can use an unbiased gradient estimator via REINFORCE (Williams, 1992), we opt for a continuous relaxation due to high variances with REINFORCE.

Original: This is achieved by replacing the categorical distribution with a concrete distribution (Maddison et al., 2016; Jang et al., 2016).
Revised: This is achieved by replacing the categorical distribution with a concrete distribution (Maddison et al., 2016; Jang et al., 2016).

Original: This relaxation procedure corresponds to adopting a “smooth” categorical distribution that can be seen as a “noisy” softmax.
Revised: This relaxation procedure corresponds to adopting a “smooth” categorical distribution that can be seen as a “noisy” softmax.

Original: Let π i be the categorical probability of sampling grid point i , i.e. π i = p
Revised: Let π i be the categorical probability of sampling grid point i , i.e. π i = p

Original: (ˆ x = g i ) ; the “smoothed” quantized value ˆ x can be obtained via:
Revised: (ˆ x = g i ) ; the “smoothed” quantized value ˆ x can be obtained via:

Original: [Equation]
Revised: [Equation]

Original: where z i is the random sample from the concrete distribution and λ is a temperature parameter that controls the degree of approximation, since as λ → 0 the concrete distribution becomes a categorical.
Revised: where z i is the random sample from the concrete distribution and λ is a temperature parameter that controls the degree of approximation, since as λ → 0 the concrete distribution becomes a categorical.

Original: We have thus deﬁned a fully differentiable “soft” quantization procedure that allows for stochastic gradients for both the quantizer parameters α, β, σ as well as the input signal x (e g. the weights or the activations of a neural network).
Revised: We have thus deﬁned a fully differentiable “soft” quantization procedure that allows for stochastic gradients for both the quantizer parameters α, β, σ as well as the input signal x (e g. the weights or the activations of a neural network).

Original: We refer to this algorithm as Relaxed Quantization (RQ).
Revised: We refer to this algorithm as Relaxed Quantization (RQ).

Original: We summarize its forward pass as performed during training in Algorithm 1.
Revised: We summarize its forward pass as performed during training in Algorithm 1.

Original: It is also worthwhile to notice that if there were no noise at the input x then the categorical distribution would have non-zero mass only at a single value, thus prohibiting gradient based optimization for x and σ .
Revised: It is also worthwhile to notice that if there were no noise at the input x then the categorical distribution would have non-zero mass only at a single value, thus prohibiting gradient based optimization for x and σ .

Original: One drawback of this approach is that the smoothed quantized values deﬁned in Equation 5 do not have to coincide with grid points, as z is not a one-hot vector.
Revised: One drawback of this approach is that the smoothed quantized values deﬁned in Equation 5 do not have to coincide with grid points, as z is not a one-hot vector.

Original: Instead, these values can lie anywhere between the smallest and largest grid point, something which is impossible with e g. stochastic rounding (Gupta et al., 2015).
Revised: Instead, these values can lie anywhere between the smallest and largest grid point, something which is impossible with e g. stochastic rounding (Gupta et al., 2015).

Original: In order to make sure that only grid-points are sampled, we propose an alternative algorithm RQ ST in which we use the variant of the straight-through (ST) estimator proposed in Jang et al.
Revised: In order to make sure that only grid-points are sampled, we propose an alternative algorithm RQ ST in which we use the variant of the straight-through (ST) estimator proposed in Jang et al.

Original: Here we sample the actual categorical distribution during the forward pass but assume a sample from the concrete distribution for the backward pass.
Revised: Here we sample the actual categorical distribution during the forward pass but assume a sample from the concrete distribution for the backward pass.

Original: While this gradient estimator is obviously biased, in practice it works as the “gradients” seem to point towards a valid direction.
Revised: While this gradient estimator is obviously biased, in practice it works as the “gradients” seem to point towards a valid direction.

Original: We perform experiments with both variants.
Revised: We perform experiments with both variants.

Original: After convergence, we can obtain a “hard” quantization procedure, i.e. select points from the grid, at test time by either reverting to a categorical distribution (instead of the continuous surrogate) or by rounding to the nearest grid point.
Revised: After convergence, we can obtain a “hard” quantization procedure, i.e. select points from the grid, at test time by either reverting to a categorical distribution (instead of the continuous surrogate) or by rounding to the nearest grid point.

Original: In this paper we chose the latter as it is more aligned with the low-resource environments in which quantized models will be deployed.
Revised: In this paper we chose the latter as it is more aligned with the low-resource environments in which quantized models will be deployed.

Original: Furthermore, with this goal in mind, we employ two quantization grids with their own learnable scalar α, σ (and potentially β ) parameters for each layer; one for the weights and one for the activations.
Revised: Furthermore, with this goal in mind, we employ two quantization grids with their own learnable scalar α, σ (and potentially β ) parameters for each layer; one for the weights and one for the activations.

Original: 2. S CALABLE QUANTIZATION VIA A LOCAL GRID
Revised: 2. S CALABLE QUANTIZATION VIA A LOCAL GRID

Original: Sampling ˆ x based on drawing K random numbers for the concrete distribution as described in
Revised: Sampling ˆ x based on drawing K random numbers for the concrete distribution as described in

Original: Equation 5 can be very expensive for larger values of K .
Revised: Equation 5 can be very expensive for larger values of K .

Original: Firstly, drawing K random numbers for every individual weight and activation in a neural network drastically increases the number of operations required in the forward pass.
Revised: Firstly, drawing K random numbers for every individual weight and activation in a neural network drastically increases the number of operations required in the forward pass.

Original: Secondly, it also requires keeping many more numbers in memory for gradient computations during the backward pass.
Revised: Secondly, it also requires keeping many more numbers in memory for gradient computations during the backward pass.

Original: Compared to a standard neural network or stochastic rounding approaches, the proposed procedure can thus be infeasible for larger models and datasets.
Revised: Compared to a standard neural network or stochastic rounding approaches, the proposed procedure can thus be infeasible for larger models and datasets.

Original: Fortunately, we can make sampling ˆ x independent of the grid size by assuming zero probability for grid-points that lie far away from the signal x .
Revised: Fortunately, we can make sampling ˆ x independent of the grid size by assuming zero probability for grid-points that lie far away from the signal x .

Original: Speciﬁcally, by only considering grid points that are within δ standard deviations away from x , we truncate p (˜ x ) such that it lies within a “localized” grid around x .
Revised: Speciﬁcally, by only considering grid points that are within δ standard deviations away from x , we truncate p (˜ x ) such that it lies within a “localized” grid around x .

Original: Require:
Revised: Require:

Original: Input x , grid ˆ G , scale of the grid α , scale of noise σ , temperature λ , fuzz param.
Revised: Input x , grid ˆ G , scale of the grid α , scale of noise σ , temperature λ , fuzz param.

Original: (cid:15) r
Revised: (cid:15) r

Original: =
Revised: =

Original: [ ˆ G − α/ 2 , g K + α/ 2] # interval points c =
Revised: [ ˆ G − α/ 2 , g K + α/ 2] # interval points c =

Original: Sigmoid (( r − x ) /σ )
Revised: Sigmoid (( r − x ) /σ )

Original: # evaluate CDF π i
Revised: # evaluate CDF π i

Original: =
Revised: =

Original: c
Revised: c

Original: [ i +1] − c
Revised: [ i +1] − c

Original: [ i ]+ (cid:15) c
Revised: [ i ]+ (cid:15) c

Original: [ K +1] − c
Revised: [ K +1] − c

Original: [1]+ K(cid:15)
Revised: [1]+ K(cid:15)

Original: # categorical distr.
Revised: # categorical distr.

Original: z ∼ Concrete ( π, λ ) return (cid:80) i
Revised: z ∼ Concrete ( π, λ ) return (cid:80) i

Original: z i g i
Revised: z i g i

Original: Require: Input x , scale and offset of the grid α, β , minimum and maximum values g 0 , g K y
Revised: Require: Input x , scale and offset of the grid α, β , minimum and maximum values g 0 , g K y

Original: =
Revised: =

Original: α · round (( x − β ) /α )
Revised: α · round (( x − β ) /α )

Original: + β return min( g K , max( g 0 , y )
Revised: + β return min( g K , max( g 0 , y )

Original: To simplify the computation required for determining the local grid elements, we choose the grid point closest to x , (cid:98) x (cid:101) , as the center of the local grid (Figure 3).
Revised: To simplify the computation required for determining the local grid elements, we choose the grid point closest to x , (cid:98) x (cid:101) , as the center of the local grid (Figure 3).

Original: Since σ is shared between all elements of the weight matrix or activation, the local grid has the same width for every element.
Revised: Since σ is shared between all elements of the weight matrix or activation, the local grid has the same width for every element.

Original: The computation of the probabilities over the localized grid is similar to the truncation happening in Equation 4 and the smoothed quantized value is obtained via a manner similar to
Revised: The computation of the probabilities over the localized grid is similar to the truncation happening in Equation 4 and the smoothed quantized value is obtained via a manner similar to

Original: Equation 5:
Revised: Equation 5:

Original: [Equation]
Revised: [Equation]

Original: 2. R ELATION TO S TOCHASTIC R OUNDING
Revised: 2. R ELATION TO S TOCHASTIC R OUNDING

Original: One of the pioneering works in neural network quantization has been the work of Gupta et al. (2015); it introduced stochastic rounding, a technique that is one of the most popular approaches for training neural networks with reduced numerical precision.
Revised: One of the pioneering works in neural network quantization has been the work of Gupta et al. (2015); it introduced stochastic rounding, a technique that is one of the most popular approaches for training neural networks with reduced numerical precision.

Original: Instead of rounding to the nearest representable value, the stochastic rounding procedure selects one of the two closest grid points with probability depending on the distance of the high precision input from these grid points.
Revised: Instead of rounding to the nearest representable value, the stochastic rounding procedure selects one of the two closest grid points with probability depending on the distance of the high precision input from these grid points.

Original: In fact, we can view stochastic rounding as a special case of RQ where p (˜ x ) =
Revised: In fact, we can view stochastic rounding as a special case of RQ where p (˜ x ) =

Original: U ( x − α 2 , x + α 2 ) .
Revised: U ( x − α 2 , x + α 2 ) .

Original: This uniform distribution centered at x of width equal to the grid width α generally has support only for the closest grid point.
Revised: This uniform distribution centered at x of width equal to the grid width α generally has support only for the closest grid point.

Original: Discretizing this distribution to a categorical over the quantization grid however assigns probabilities to the two closest grid points as in stochastic rounding, following Equation 2:
Revised: Discretizing this distribution to a categorical over the quantization grid however assigns probabilities to the two closest grid points as in stochastic rounding, following Equation 2:

Original: [Equation]
Revised: [Equation]

Original: Stochastic rounding has proven to be a very powerful quantization scheme, even though it relies on biased gradient estimates for the rounding procedure.
Revised: Stochastic rounding has proven to be a very powerful quantization scheme, even though it relies on biased gradient estimates for the rounding procedure.

Original: On the one hand, RQ provides a way to circumvent this estimator at the cost of optimizing a surrogate objective.
Revised: On the one hand, RQ provides a way to circumvent this estimator at the cost of optimizing a surrogate objective.

Original: On the other hand, RQ
Revised: On the other hand, RQ

Original: ST makes use of the unreasonably effective straight-through estimator as used in Jang et al.
Revised: ST makes use of the unreasonably effective straight-through estimator as used in Jang et al.

Original: (2016) to avoid optimizing a surrogate objective, at the cost of biased gradients.
Revised: (2016) to avoid optimizing a surrogate objective, at the cost of biased gradients.

Original: Compared to stochastic rounding, RQ ST further allows sampling of not only the two closest grid points, but also has support for more distant ones depending on the estimated input noise σ .
Revised: Compared to stochastic rounding, RQ ST further allows sampling of not only the two closest grid points, but also has support for more distant ones depending on the estimated input noise σ .

Original: Intuitively, this allows for larger steps in the input space without ﬁrst having to decrease variance at the traversion between grid sections.
Revised: Intuitively, this allows for larger steps in the input space without ﬁrst having to decrease variance at the traversion between grid sections.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this work we focus on hardware oriented quantization approaches.
Revised: In this work we focus on hardware oriented quantization approaches.

Original: As opposed to methods that focus only on weight quantization and network compression for a reduced memory footprint, quantizing all operations within the network aims to additionally provide reduced execution times.
Revised: As opposed to methods that focus only on weight quantization and network compression for a reduced memory footprint, quantizing all operations within the network aims to additionally provide reduced execution times.

Original: Within the body of work that considers quantizing weights and activations fall papers using stochastic rounding (Gupta et al., 2015; Hubara et al., 2016; Gysel et al., 2018; Wu et al., 2018).
Revised: Within the body of work that considers quantizing weights and activations fall papers using stochastic rounding (Gupta et al., 2015; Hubara et al., 2016; Gysel et al., 2018; Wu et al., 2018).

Original: (Wu et al., 2018) also consider quantized backpropagation, which is out-of-scope for this work.
Revised: (Wu et al., 2018) also consider quantized backpropagation, which is out-of-scope for this work.

Original: Furthermore, another line of work considers binarizing (Courbariaux et al., 2015;
Revised: Furthermore, another line of work considers binarizing (Courbariaux et al., 2015;

Original: Zhou et al., 2018) or ternarizing (Li et al., 2016;
Revised: Zhou et al., 2018) or ternarizing (Li et al., 2016;

Original: Zhou et al., 2018) weights and activations (Hubara et al., 2016; Rastegari et al., 2016;
Revised: Zhou et al., 2018) weights and activations (Hubara et al., 2016; Rastegari et al., 2016;

Original: Zhou et al., 2016) via the straight-through gradient estimator (Bengio et al., 2013); these allow for fast implementations of convolutions using only bit-shift operations.
Revised: Zhou et al., 2016) via the straight-through gradient estimator (Bengio et al., 2013); these allow for fast implementations of convolutions using only bit-shift operations.

Original: In a similar vein, the straight through estimator has also been used in Cai et al.
Revised: In a similar vein, the straight through estimator has also been used in Cai et al.

Original: ; Faraone et al.
Revised: ; Faraone et al.

Original: Jacob et al.
Revised: Jacob et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: Mishra & Marr (2017) for quantizing neural networks to arbitrary bit-precision.
Revised: Mishra & Marr (2017) for quantizing neural networks to arbitrary bit-precision.

Original: In these approaches, the full precision weights that are updated during training correspond to the means of the logistic distributions that are used in RQ.
Revised: In these approaches, the full precision weights that are updated during training correspond to the means of the logistic distributions that are used in RQ.

Original: Furthermore, Jacob et al.
Revised: Furthermore, Jacob et al.

Original: (2017) maintains moving averages for the minimum and maximum observed values for activations while parameterises the network’s weights’ grids via their minimum and maximum values directly.
Revised: (2017) maintains moving averages for the minimum and maximum observed values for activations while parameterises the network’s weights’ grids via their minimum and maximum values directly.

Original: This ﬁxed-point grid is therefore learned during training, however without gradient descent; unlike the proposed RQ.
Revised: This ﬁxed-point grid is therefore learned during training, however without gradient descent; unlike the proposed RQ.

Original: Alternatively, instead of discretizing real valued weights, Shayer et al.
Revised: Alternatively, instead of discretizing real valued weights, Shayer et al.

Original: (2018) directly optimize discrete distributions over them.
Revised: (2018) directly optimize discrete distributions over them.

Original: While providing promising results, this approach does not generalize straightforwardly to activation quantization.
Revised: While providing promising results, this approach does not generalize straightforwardly to activation quantization.

Original: A bayesian approach to binarized models was taken in Soudry et al.
Revised: A bayesian approach to binarized models was taken in Soudry et al.

Original: (2014), which provided encouraging results on small scale experiments with an ensemble of quantized models sampled from the approximate posterior distribution.
Revised: (2014), which provided encouraging results on small scale experiments with an ensemble of quantized models sampled from the approximate posterior distribution.

Original: Another line of work quantizes networks through regularization.
Revised: Another line of work quantizes networks through regularization.

Original: (Louizos et al., 2017a) formulate a variational approach that allows for heuristically determining the required bit-width precision for each weight of the model.
Revised: (Louizos et al., 2017a) formulate a variational approach that allows for heuristically determining the required bit-width precision for each weight of the model.

Original: Improving upon this work, (Achterhold et al., 2018) proposed a quantizing prior that encourages ternary weights during training.
Revised: Improving upon this work, (Achterhold et al., 2018) proposed a quantizing prior that encourages ternary weights during training.

Original: Similarly to RQ, this method also allows for optimizing the scale of the ternary grid.
Revised: Similarly to RQ, this method also allows for optimizing the scale of the ternary grid.

Original: In contrast to RQ, this is only done implicitly via the regularization term.
Revised: In contrast to RQ, this is only done implicitly via the regularization term.

Original: One drawback of these approaches is that the strength of the regularization decays with the amount of training data, thus potentially reducing their effectiveness on large datasets.
Revised: One drawback of these approaches is that the strength of the regularization decays with the amount of training data, thus potentially reducing their effectiveness on large datasets.

Original: Weights in a neural network are usually not distributed uniformly within a layer.
Revised: Weights in a neural network are usually not distributed uniformly within a layer.

Original: As a result, performing non-uniform quantization is usually more effective.
Revised: As a result, performing non-uniform quantization is usually more effective.

Original: (Baskin et al., 2018) employ a stochastic quantizer by ﬁrst uniformizing the weight or activation distribution through a non-linear transformation and then injecting uniform noise into this transformed space.
Revised: (Baskin et al., 2018) employ a stochastic quantizer by ﬁrst uniformizing the weight or activation distribution through a non-linear transformation and then injecting uniform noise into this transformed space.

Original: (Polino et al., 2018) propose a version of their method in which the quantizer’s code book is learned by gradient descent, resulting in a non-uniformly spaced grid.
Revised: (Polino et al., 2018) propose a version of their method in which the quantizer’s code book is learned by gradient descent, resulting in a non-uniformly spaced grid.

Original: Another line of works quantizes by clustering and therefore falls into this category; (Han et al., 2015; Ullrich et al., 2017) represent each of the weights by the centroid of its closest cluster.
Revised: Another line of works quantizes by clustering and therefore falls into this category; (Han et al., 2015; Ullrich et al., 2017) represent each of the weights by the centroid of its closest cluster.

Original: While such non-uniform techniques can be indeed effective, they do not allow for efﬁcient implementations on todays hardware.
Revised: While such non-uniform techniques can be indeed effective, they do not allow for efﬁcient implementations on todays hardware.

Original: Nevertheless, there is encouraging recent work (Zhang et al., 2018) on non-uniform grids that can be implemented with bit operations.
Revised: Nevertheless, there is encouraging recent work (Zhang et al., 2018) on non-uniform grids that can be implemented with bit operations.

Original: Within the liteterature on quantizing neural networks there are many approaches that are orthogonal to our work and could potentially be combined for additional improvements.
Revised: Within the liteterature on quantizing neural networks there are many approaches that are orthogonal to our work and could potentially be combined for additional improvements.

Original: (Mishra & Marr, 2017; Polino et al., 2018) use knowledge distrillation techniques to good effect, whereas works such as (Mishra et al., 2017) modify the architecture to compensate for lower precision computations.
Revised: (Mishra & Marr, 2017; Polino et al., 2018) use knowledge distrillation techniques to good effect, whereas works such as (Mishra et al., 2017) modify the architecture to compensate for lower precision computations.

Original: (Zhou et al., 2017; 2018; Baskin et al., 2018) perform quantization in an step-by-step manner going from input layer to output, thus allowing the later layers to more easily adapt to the rounding errors introduced.
Revised: (Zhou et al., 2017; 2018; Baskin et al., 2018) perform quantization in an step-by-step manner going from input layer to output, thus allowing the later layers to more easily adapt to the rounding errors introduced.

Original: Polino et al.
Revised: Polino et al.

Original: ; Faraone et al.
Revised: ; Faraone et al.

Original: (2018) further employ “bucketing”, where small groups of weights share a grid, instead of one grid per layer.
Revised: (2018) further employ “bucketing”, where small groups of weights share a grid, instead of one grid per layer.

Original: As an example from Polino et al.
Revised: As an example from Polino et al.

Original: (2018), a bucket size of 256 weights per grid on Resnet-18 translates to ∼ 45 .
Revised: (2018), a bucket size of 256 weights per grid on Resnet-18 translates to ∼ 45 .

Original: 7 k separate weight quantization grids as opposed to 22 in RQ.
Revised: 7 k separate weight quantization grids as opposed to 22 in RQ.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For the subsequent experiments RQ will correspond to the proposed procedure that has concrete sampling and RQ ST will correspond to the proposed procedure that uses the Gumbel-softmax straight-through estimator (Jang et al., 2016) for the gradient.
Revised: For the subsequent experiments RQ will correspond to the proposed procedure that has concrete sampling and RQ ST will correspond to the proposed procedure that uses the Gumbel-softmax straight-through estimator (Jang et al., 2016) for the gradient.

Original: We did not optimize an offset for the grids in order to be able to represent the number zero exactly, which allows for sparsity and is required for zero-padding.
Revised: We did not optimize an offset for the grids in order to be able to represent the number zero exactly, which allows for sparsity and is required for zero-padding.

Original: Furthermore we assumed a grid that starts from zero when quantizing the outputs of ReLU.
Revised: Furthermore we assumed a grid that starts from zero when quantizing the outputs of ReLU.

Original: We provide further details on the experimental settings at Appendix A. We will also provide results of our own implementation of stochastic rounding (Gupta et al., 2015) with the dynamic ﬁxed point format (Gysel et al., 2018) (SR+DR).
Revised: We provide further details on the experimental settings at Appendix A. We will also provide results of our own implementation of stochastic rounding (Gupta et al., 2015) with the dynamic ﬁxed point format (Gysel et al., 2018) (SR+DR).

Original: Here we used the same hyperparameters as for RQ.
Revised: Here we used the same hyperparameters as for RQ.

Original: All experiments were implemented with TensorFlow (Abadi et al., 2015), using the Keras library (Chollet et al., 2015).
Revised: All experiments were implemented with TensorFlow (Abadi et al., 2015), using the Keras library (Chollet et al., 2015).

Original: 4. L E N ET -5 ON MNIST AND VGG7 ON CIFAR
Revised: 4. L E N ET -5 ON MNIST AND VGG7 ON CIFAR

Original: For the ﬁrst task we considered the toy LeNet-5 network trained on MNIST with the 32C5 - MP - 64C5 - MP2 - 512FC - Softmax architecture and the VGG 2x(128C3) - MP2 - 2x(256C3) - MP - 2x(512C3) - MP2 - 1024FC - Softmax architecture on the CIFAR 10 dataset.
Revised: For the ﬁrst task we considered the toy LeNet-5 network trained on MNIST with the 32C5 - MP - 64C5 - MP2 - 512FC - Softmax architecture and the VGG 2x(128C3) - MP2 - 2x(256C3) - MP - 2x(512C3) - MP2 - 1024FC - Softmax architecture on the CIFAR 10 dataset.

Original: Details about the hyperparameter settings can be found in Appendix A.
Revised: Details about the hyperparameter settings can be found in Appendix A.

Original: By observing the results in Table 1, we see that our method can achieve competitive results that improve upon several recent works on neural network quantization.
Revised: By observing the results in Table 1, we see that our method can achieve competitive results that improve upon several recent works on neural network quantization.

Original: Considering that we achieve lower test error for 8 bit quantization than the high-precision models, we can see how RQ has a regularizing effect.
Revised: Considering that we achieve lower test error for 8 bit quantization than the high-precision models, we can see how RQ has a regularizing effect.

Original: Generally speaking we found that the gradient variance for low bit-widths (i.e.
Revised: Generally speaking we found that the gradient variance for low bit-widths (i.e.

Original: 2-4 bits) in RQ needs to be kept in check through appropriate learning rates.
Revised: 2-4 bits) in RQ needs to be kept in check through appropriate learning rates.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 65 a
Revised: 65 a

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Revised: 53 a

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Revised: 50 a

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: 53 a

Original: [Table][Table] [Table]
Revised: [Table]

Original: 50 a

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 4. R ESNET -18 AND M OBILENET ON I MAGENET
Revised: 4. R ESNET -18 AND M OBILENET ON I MAGENET

Original: In order to demonstrate the effectiveness of our proposed approach on large scale tasks we considered the task of quantizing a Resnet-18 (He et al., 2016) as well as a Mobilenet (Howard et al., 2017) trained on the Imagenet (ILSVRC2012) dataset.
Revised: In order to demonstrate the effectiveness of our proposed approach on large scale tasks we considered the task of quantizing a Resnet-18 (He et al., 2016) as well as a Mobilenet (Howard et al., 2017) trained on the Imagenet (ILSVRC2012) dataset.

Original: For the Resnet-18 experiment, we started from a pre-trained full precision model that was trained for 90 epochs.
Revised: For the Resnet-18 experiment, we started from a pre-trained full precision model that was trained for 90 epochs.

Original: We provide further details about the training procedure in Appendix B. The Mobilenet was initialized with the pretrained model available
Revised: We provide further details about the training procedure in Appendix C. The Mobilenet was initialized with the pretrained model available

Original: [Figure]
Revised: [Figure]

Original: 2 in on the tensorﬂow github repository 1 .
Revised: on the tensorﬂow github repository 1 .

Original: We quantized the weights of all layers, post ReLU activations and average pooling layer for various bit-widths via ﬁne-tuning for ten epochs.
Revised: We quantized the weights of all layers, post ReLU activations and average pooling layer for various bit-widths via ﬁne-tuning for ten epochs.

Original: Further details can be found in Appendix B.
Revised: Further details can be found in Appendix C.

Original: Some of the existing quantization works do not quantize the ﬁrst (and sometimes) last layer.
Revised: Some of the existing quantization works do not quantize the ﬁrst (and sometimes) last layer.

Original: Doing so simpliﬁes the problem but it can, depending on the model and input dimensions, signiﬁcantly increase the amount of computation required.
Revised: Doing so simpliﬁes the problem but it can, depending on the model and input dimensions, signiﬁcantly increase the amount of computation required.

Original: We therefore make use of the bit operationsper second (BOPs) metric (Baskin et al., 2018), which can be seen as a proxy for the execution speed on appropriate hardware.
Revised: We therefore make use of the bit operations (BOPs) metric (Baskin et al., 2018), which can be seen as a proxy for the execution speed on appropriate hardware.

Original: In BOPs, the impact of not quantizing the ﬁrst layer in, for example, the Resnet-18 model on Imagenet, becomes apparent: keeping the ﬁrst layer in full precision requires roughly 2 .
Revised: In BOPs, the impact of not quantizing the ﬁrst layer in, for example, the Resnet-18 model on Imagenet, becomes apparent: keeping the ﬁrst layer in full precision requires roughly 1 .

Original: 5 times as many BOPs for one forward pass through the whole network compared to quantizing all weights and activations to 5 bits.
Revised: 3 times as many BOPs for one forward pass through the whole network compared to quantizing all weights and activations to 5 bits.

Original: Figure 4 compares a wide range of methods in terms of accuracy and BOPs.
Revised: Figure 4 compares a wide range of methods in terms of accuracy and BOPs.

Original: We choose to compare only against methods that employ ﬁxed-point quantization on Resnet-18 and Mobilenet, hence do not compare with non-uniform quantization techniques, such as the one described at Baskin et al.
Revised: We choose to compare only against methods that employ ﬁxed-point quantization on Resnet-18 and Mobilenet, hence do not compare with non-uniform quantization techniques, such as the one described at Baskin et al.

Original: In addition to our own implementation of (Gupta et al., 2015) with the dynamic ﬁxed point format (Gysel et al., 2018), we also report results of “rounding”.
Revised: In addition to our own implementation of (Gupta et al., 2015) with the dynamic ﬁxed point format (Gysel et al., 2018), we also report results of “rounding”.

Original: This corresponds to simply rounding the pre-trained high-precision model followed by re-estimation of the batchnorm statistics.
Revised: This corresponds to simply rounding the pre-trained high-precision model followed by re-estimation of the batchnorm statistics.

Original: The grid in this case is deﬁned as the initial grid used for ﬁne-tuning with RQ.
Revised: The grid in this case is deﬁned as the initial grid used for ﬁne-tuning with RQ.

Original: For batchnorm re-estimation and grid initialization, please confer Appendix A.
Revised: For batchnorm re-estimation and grid initialization, please confer Appendix A.

Original: In Figure 4a we observe that on ResNet-18 the RQ variants form the “Pareto frontier” in the trade-off between accuracy and efﬁciency, along with SYQ, Apprentice and Jacob et al.
Revised: In Figure 4a we observe that on ResNet-18 the RQ variants form the “Pareto frontier” in the trade-off between accuracy and efﬁciency, along with SYQ, Apprentice and Jacob et al.

Original: SYQ, however, employs “bucketing” and Apprentice uses distillation, both of which can be combined with RQ and improve performance.
Revised: SYQ, however, employs “bucketing” and Apprentice uses distillation, both of which can be combined with RQ and improve performance.

Original: Jacob et al.
Revised: Jacob et al.

Original: (2017) does better than RQ with 8 bits, however RQ improved w.r.t.
Revised: (2017) does better than RQ with 8 bits, however RQ improved w.r.t.

Original: to its pretrained model, whereas Jacob et al.
Revised: to its pretrained model, whereas Jacob et al.

Original: (2017) decreased slightly.
Revised: (2017) decreased slightly.

Original: For experimental details with Jacob et al.
Revised: For experimental details with Jacob et al.

Original: (2017), please confer Appendix B.1.
Revised: (2017), please confer Appendix C.1.

Original: SR+DR underperforms in this setting and is worse than simple rounding for 5 to 8 bits.
Revised: SR+DR underperforms in this setting and is worse than simple rounding for 5 to 8 bits.

Original: For Mobilenet, 4b shows that RQ is competitive to existing approaches.
Revised: For Mobilenet, 4b shows that RQ is competitive to existing approaches.

Original: Simple rounding resulted in almost random chance for all of the bit conﬁgurations.
Revised: Simple rounding resulted in almost random chance for all of the bit conﬁgurations.

Original: SR+DR shows its strength for the 8 bit scenario, while in the lower bit regime, RQ outperforms competitive approaches.
Revised: SR+DR shows its strength for the 8 bit scenario, while in the lower bit regime, RQ outperforms competitive approaches.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have introduced Relaxed Quantization (RQ), a powerful and versatile algorithm for learning low-bit neural networks using a uniform quantization scheme.
Revised: We have introduced Relaxed Quantization (RQ), a powerful and versatile algorithm for learning low-bit neural networks using a uniform quantization scheme.

Original: As such, the models trained by this method can be easily transferred and executed on low-bit ﬁxed point chipsets.
Revised: As such, the models trained by this method can be easily transferred and executed on low-bit ﬁxed point chipsets.

Original: We have extensively evaluated RQ on various image classiﬁcation benchmarks and have shown that it allows for the better trade-offs between accuracy and bit operations per second.
Revised: We have extensively evaluated RQ on various image classiﬁcation benchmarks and have shown that it allows for the better trade-offs between accuracy and bit operations per second.

Original: Future hardware might enable us to cheaply do non-uniform quantization, for which this method can be easily extended.
Revised: Future hardware might enable us to cheaply do non-uniform quantization, for which this method can be easily extended.

Original: (Lai et al., 2017; Ortiz et al., 2018) for example, show the beneﬁts of low-bit ﬂoating point weights that can be efﬁciently implemented in hardware.
Revised: (Lai et al., 2017; Ortiz et al., 2018) for example, show the beneﬁts of low-bit ﬂoating point weights that can be efﬁciently implemented in hardware.

Original: The ﬂoating point quantization grid can be easily learned with RQ by redeﬁning ˆ G .
Revised: The ﬂoating point quantization grid can be easily learned with RQ by redeﬁning ˆ G .

Original: General non-uniform quantization, as described for example in (Baskin et al., 2018), is a natural extension to RQ, whose exploration we leave to future work.
Revised: General non-uniform quantization, as described for example in (Baskin et al., 2018), is a natural extension to RQ, whose exploration we leave to future work.

Original: For example, we could experiment with a base grid that is deﬁned as in Zhang et al.
Revised: For example, we could experiment with a base grid that is deﬁned as in Zhang et al.

Original: Currently, the bit-width of every quantizer is determined beforehand, but in future work we will explore learning the required bit precision within this framework.
Revised: Currently, the bit-width of every quantizer is determined beforehand, but in future work we will explore learning the required bit precision within this framework.

Original: In our experiments, batch normalization was implemented as a sequence of convolution, batch normalization and quantization.
Revised: In our experiments, batch normalization was implemented as a sequence of convolution, batch normalization and quantization.

Original: On a low-precision chip, however, batch normalization would be ”folded” (Jacob et al., 2017) into the kernel and bias of the convolution, the result of which is then rounded to low precision.
Revised: On a low-precision chip, however, batch normalization would be ”folded” (Jacob et al., 2017) into the kernel and bias of the convolution, the result of which is then rounded to low precision.

Original: In order to accurately reﬂect this folding at test time, future work on the proposed algorithm will emulate folded batchnorm at training time and learn the corresponding quantization grid of the modiﬁed kernel and bias.
Revised: In order to accurately reﬂect this folding at test time, future work on the proposed algorithm will emulate folded batchnorm at training time and learn the corresponding quantization grid of the modiﬁed kernel and bias.

Original: For fast model evaluation on low-precision hardware, quantization goes hand-in-hand with network pruning.
Revised: For fast model evaluation on low-precision hardware, quantization goes hand-in-hand with network pruning.

Original: The proposed method is orthogonal to pruning methods such as, for example, Lregularization (Louizos et al., 2017b), which allows for group sparsity and pruning of hidden units.
Revised: The proposed method is orthogonal to pruning methods such as, for example, Lregularization (Louizos et al., 2017b), which allows for group sparsity and pruning of hidden units.

Original: A E XPERIMENTAL DETAILS
Revised: A E XPERIMENTAL DETAILS

Original: The grid width α of each grid was initialized according to the bit-width b and the maximum and minimum values of the input x to the quantizer 2 .
Revised: The grid width α of each grid was initialized according to the bit-width b and the maximum and minimum values of the input x to the quantizer 2 .

Original: Since the inputs ˜ x in both cases for our approach are stochastic it makes sense to assume a width for the grid that is slightly larger than the standard width t =
Revised: Since the inputs ˜ x in both cases for our approach are stochastic it makes sense to assume a width for the grid that is slightly larger than the standard width t =

Original: (max( x ) − min( x )) / 2 b ; for the activations, whenever b > 4 , we initialize α = t + 3 t/ 2 b , for 4 ≥ b > 2 we used α = t + 3 t/ 2 b +1 and ﬁnally for b = 2 we used α = t .
Revised: (max( x ) − min( x )) / 2 b ; for the activations, whenever b > 4 , we initialize α = t + 3 t/ 2 b , for 4 ≥ b > 2 we used α = t + 3 t/ 2 b +1 and ﬁnally for b = 2 we used α = t .

Original: Since with ReLU activations the magnitude can become quite large (thus leading to increased quantization noise for smaller bit widths), this scheme keeps the noise injected to the network in check.
Revised: Since with ReLU activations the magnitude can become quite large (thus leading to increased quantization noise for smaller bit widths), this scheme keeps the noise injected to the network in check.

Original: For the weights we 2 For activations we computed the minimum and maximum on a random minibatch of inputs.
Revised: For the weights we 2 For activations we computed the minimum and maximum on a random minibatch of inputs.

Original: always used an initial α = t + 3 t/ 2 b .
Revised: always used an initial α = t + 3 t/ 2 b .

Original: The standard deviation of the logistic noise σ was initialized to be three times smaller than the width α , i.e. σ = α/ 3 .
Revised: The standard deviation of the logistic noise σ was initialized to be three times smaller than the width α , i.e. σ = α/ 3 .

Original: Under this speciﬁcation, most of the probability mass of the logistic distribution is initially (roughly) in the bins containing the closest grid point and its’ two neighbors.
Revised: Under this speciﬁcation, most of the probability mass of the logistic distribution is initially (roughly) in the bins containing the closest grid point and its’ two neighbors.

Original: The moving averages of layer statistics that are aggregated during the training phase for the batch normalization do not necessarily reﬂect the statistics of the quantized model accurately.
Revised: The moving averages of layer statistics that are aggregated during the training phase for the batch normalization do not necessarily reﬂect the statistics of the quantized model accurately.

Original: Even though
Revised: Even though

Original: RQ aims to minimize the gap between training and testing phase, we found that the aggregated statistics in combination with the learned scale and shift parameters of batch normalization lead to decreased test performance.
Revised: RQ aims to minimize the gap between training and testing phase, we found that the aggregated statistics in combination with the learned scale and shift parameters of batch normalization lead to decreased test performance.

Original: In order to avoid this drop in accuracy, we apply the insights from (Peters & Welling, 2018) and recompute the statistics of the quantized model before reporting the ﬁnal test error rate.
Revised: In order to avoid this drop in accuracy, we apply the insights from (Peters & Welling, 2018) and recompute the statistics of the quantized model before reporting the ﬁnal test error rate.

Original: The ﬁnal models were determined through early stopping using the validation loss computed with minibatch statistics, in case the model uses batch normalization.
Revised: The ﬁnal models were determined through early stopping using the validation loss computed with minibatch statistics, in case the model uses batch normalization.

Original: For the MNIST experiment we rescaled the input to the [-1, 1] range, employed no regularization and the network was trained with Adam (Kingma & Ba, 2014) and a batch size of 128.
Revised: For the MNIST experiment we rescaled the input to the [-1, 1] range, employed no regularization and the network was trained with Adam (Kingma & Ba, 2014) and a batch size of 128.

Original: We used a local grid whenever the bit width was larger than 2 for both, weights and biases (shared grid parameters), as well as for the ouputs of the ReLU, with δ = 3 .
Revised: We used a local grid whenever the bit width was larger than 2 for both, weights and biases (shared grid parameters), as well as for the ouputs of the ReLU, with δ = 3 .

Original: For the 8 and 4 bit networks we used a temperature λ of 2 whereas for the 2 bit models we used a temperature of 1 for RQ.
Revised: For the 8 and 4 bit networks we used a temperature λ of 2 whereas for the 2 bit models we used a temperature of 1 for RQ.

Original: We trained the 8 and 4 bit networks for 100 epochs using a learning rate of 1e-3 and the 2 bit networks for 200 epochs with a learning rate of 5e-4.
Revised: We trained the 8 and 4 bit networks for 100 epochs using a learning rate of 1e-3 and the 2 bit networks for 200 epochs with a learning rate of 5e-4.

Original: In all of the cases the learning rate was annealed to zero during the lastepochs.
Revised: In all of the cases the learning rate was annealed to zero during the lastepochs.

Original: For the CIFAR 10 experiment, the hyperparameters were chosen identically to the LeNet-5 experiments except a few differences.
Revised: For the CIFAR 10 experiment, the hyperparameters were chosen identically to the LeNet-5 experiments except a few differences.

Original: We chose a learning rate ot 1e-4 instead of 1e-3 for 8 and 4 bit networks and trained for 300 epochs with a batch size of 100.
Revised: We chose a learning rate ot 1e-4 instead of 1e-3 for 8 and 4 bit networks and trained for 300 epochs with a batch size of 100.

Original: We also included a weight decay term of 1e-4 for the 8 bit networks.
Revised: We also included a weight decay term of 1e-4 for the 8 bit networks.

Original: For the 2 bit model we started with a learning rate of 1e-3.
Revised: For the 2 bit model we started with a learning rate of 1e-3.

Original: The VGG model contains a batch normalization layer after every convolutional layer, but preceeded by max pooling, if present.
Revised: The VGG model contains a batch normalization layer after every convolutional layer, but preceeded by max pooling, if present.

Revised: B C ONVERGENCE SPEED OF VGG ON CIFAR

Revised: Training a neural network with RQ imposes an additional sampling burden for every weight and activation in the network.

Revised: Here, we investigate whether the extra “noise” that is introduced hampers the convergence speed of the network when we train from a random initialization.

Original: B I MAGENET DETAILS
Revised: We recorded the learning curves for a 2/2 bit RQ-VGG network on CIFAR 10 (as this quantization level exhibits the largest amount of noise) and compare it to the full precision baseline. The results can be seen in Figure 5. As we can observe, the 2/2 bit network has qualitatively similar trends to the full precision baseline.

Revised: Therefore we can conclude that the noise is not detrimental for the task at hand, at least for this particular model.

Revised: In terms of wall-clock time, training the RQ model with a full (4 elements) grid took approximately 15 times as long as the high-precision baseline with an implementation in

Revised: Tensorﬂow v1.11.0 and running on a Titan-X Nvidia GPU.

Revised: C I MAGENET DETAILS

Original: Each channel of the input images was preprocessed by subtracting the mean and dividing by the standard deviation of that channel across the training set.
Revised: Each channel of the input images was preprocessed by subtracting the mean and dividing by the standard deviation of that channel across the training set.

Original: We then resized the images such that the shorter side is set to 256 and then applied random 224x224 crops and random horizontal ﬂips for data augmentation.
Revised: We then resized the images such that the shorter side is set to 256 and then applied random 224x224 crops and random horizontal ﬂips for data augmentation.

Original: For evaluation we consider the center 224x224 crop of the images.
Revised: For evaluation we consider the center 224x224 crop of the images.

Original: We trained the base Resnet-18 model with stochastic gradient descent, a batch size of 128, nesterov momentum of 0.9 and a learning rate of 0.1 which was multiplied by 0.1 at the 30th and 60th epoch.
Revised: We trained the base Resnet-18 model with stochastic gradient descent, a batch size of 128, nesterov momentum of 0.9 and a learning rate of 0.1 which was multiplied by 0.1 at the 30th and 60th epoch.

Original: We also applied weight decay with a strength of 1e-4.
Revised: We also applied weight decay with a strength of 1e-4.

Original: For the quantized model ﬁne-tuning phase, we used Adam with a learning rate of 5 e − 6 , a batch size of 24 and a momentum of 0.99.
Revised: For the quantized model ﬁne-tuning phase, we used Adam with a learning rate of 5 e − 6 , a batch size of 24 and a momentum of 0.99.

Original: We used a temperature of 2 for both RQ variants.
Revised: We used a temperature of 2 for both RQ variants.

Original: Following the strategy in (Jacob et al., 2017), we did not quantize the biases.
Revised: Following the strategy in (Jacob et al., 2017), we did not quantize the biases.

Original: Table 2 contains the error rates for Resnet-18 and Mobilenet on which Figure 1 is based on.
Revised: Table 2 contains the error rates for Resnet-18 and Mobilenet on which Figure 1 is based on.

Original: Algorithm and architecture speciﬁc changes are mentioned explicitly through footnotes.
Revised: Algorithm and architecture speciﬁc changes are mentioned explicitly through footnotes.

Original: B. J ACOB ET AL . (2017) FOR R ESNET 
Revised: C.

Revised: [Figure]

Original: We used the code provided at https://github.com/tensorflow/models/tree/ master/official/resnet and modiﬁed the construction of the training and evaluation graph by inserting quantization operations provided by the tensorflow.contrib.quantize package.
Revised: We used the code provided at https://github.com/tensorflow/models/tree/ master/official/resnet and modiﬁed the construction of the training and evaluation graph by inserting quantization operations provided by the tensorflow.contrib.quantize package.

Original: In a ﬁrst step, the unmodiﬁed code was used to train a high-precision Resnet18 model using the hyper-parameter settings for the learning rate scheduling that are provided in the github repository.
Revised: In a ﬁrst step, the unmodiﬁed code was used to train a high-precision Resnet18 model using the hyper-parameter settings for the learning rate scheduling that are provided in the github repository.

Original: More speciﬁcally, the model was trained for 90 epochs with a batch size of 128 .
Revised: More speciﬁcally, the model was trained for 90 epochs with a batch size of 128 .

Original: The learning rate scheduling involved a ”warm up” period in which the learning rate was annealed from zero to 0 .over
Revised: The learning rate scheduling involved a ”warm up” period in which the learning rate was annealed from zero to 0 .over

Original: the ﬁrst 50 k steps, after which it was divided by 10 after epochs 30 , 60 and 80 respectively.
Revised: the ﬁrst 50 k steps, after which it was divided by 10 after epochs 30 , 60 and 80 respectively.

Original: Gradients were modiﬁed using a momentum of 0 .
Revised: Gradients were modiﬁed using a momentum of 0 .

Original: Final test performance under this procedure is 29 .
Revised: Final test performance under this procedure is 29 .

Original: 53% top-1 error and 10 .
Revised: 53% top-1 error and 10 .

Original: 44% top-5 error.
Revised: 44% top-5 error.

Original: From the high-precision model checkpoint, the ﬁnal quantized model was then ﬁne-tuned for 10 epochs using a constant learning rate of 1 e − 4 and mo- [Table] [Table]
Revised: From the high-precision model checkpoint, the ﬁnal quantized model was then ﬁne-tuned for 10 epochs using a constant learning rate of 1 e − 4 and momentum of 0 .

Original: DoReFa (Zhou et al., 2016) eb 1 /40 .18 .- -

Original: ELQ (Zhou et al., 2018) 1 /35 .13 .- - 2 /32 .11 .- -

Original: SYQ (Faraone et al., 2018)

Original: [-f 1 /
-]
Revised: [-f 1 /
-]

Original: Apprentice (Mishra & Marr, 2017)

Original: [Table]

Original: 2 / − − − 4 /29 .−

Original: − −

Original: RQ

Original: [Table]

Original: RQ ST

Original: [Table]

Original: [Table]

Original: mentum of 0 .

Original: We did not freeze the moving averages of the batch normalization layers.
Revised: We did not freeze the moving averages of the batch normalization layers.

Original: Finally, we found that re-estimating the batchnorm statistics was harmful for this algorithm.
Revised: Finally, we found that re-estimating the batchnorm statistics was harmful for this algorithm.

Original: We hypothesise that this is due to the usage of folded batch normalization, which incorporates the statistics into the construction of the grid at training time.
Revised: We hypothesise that this is due to the usage of folded batch normalization, which incorporates the statistics into the construction of the grid at training time.

Original: B. J ACOB ET AL . (2017) FOR M OBILENET
Revised: C. J ACOB ET AL . (2017) FOR M OBILENET

Original: The 8 / 8 bit results for quantizing Mobilenet provided in table 2 are read off from Figure 4.1 in Jacob et al.
Revised: The 8 / 8 bit results for quantizing Mobilenet provided in table 2 are read off from Figure 4.1 in Jacob et al.

Original: The pre-trained models published at https://github.com/tensorflow/ models/blob/master/research/slim/nets/mobilenet_v1.md originally reﬂected that number up until commit 4415c2613b0c74032a7c631769ef9fa7f5477d88 , but have since been updated to improved error rates of 29 . 9 and 11 . 1 respectively.
Revised: The pre-trained models published at https://github.com/tensorflow/ models/blob/master/research/slim/nets/mobilenet_v1.md originally reﬂected that number up until commit 4415c2613b0c74032a7c631769ef9fa7f5477d88 , but have since been updated to improved error rates of 29 . 9 and 11 . 1 respectively.

Original: Unfortunately, there are several conﬂicting sources for quantized Mobilenet results and pretrainedmodels within the tensorﬂow github repository.
Revised: Unfortunately, there are several conﬂicting sources for quantized Mobilenet results and pretrainedmodels within the tensorﬂow github repository.

Original: https://github.com/tensorflow/ tensorflow/blob/master/tensorflow/contrib/lite/g3doc/models.md# image-classification-quantized-models , for example, reports error rates of 30 . 0 and 11 . 0 , whereas at https://github.com/tensorflow/tensorflow/tree/master/ tensorflow/contrib/quantize the reported top-1 error rate is 30 . 3 . 
Revised: https://github.com/tensorflow/ tensorflow/blob/master/tensorflow/contrib/lite/g3doc/models.md# image-classification-quantized-models , for example, reports error rates of 30 . 0 and 11 . 0 , whereas at https://github.com/tensorflow/tensorflow/tree/master/ tensorflow/contrib/quantize the reported top-1 error rate is 30 . 3 . [Table] [Table] DoReFa (Zhou et al., 2016) be 1 /40 .18 .- -

Revised: ELQ (Zhou et al., 2018) 1 /35 .13 .- - 2 /32 .11 .- -

Revised: SYQ (Faraone et al., 2018)

Original: We attempted to use the provided training scripts in the https://github.com/tensorflow/ models/blob/master/research/slim repository to train lower-bit mobilenet variants, but did not succeed in doing so.{+f 1 /
 Apprentice [Table] [Table]+}
Revised: {+f 1 /
 Apprentice [Table] [Table]+}

Revised: 2 / − − − 4 /29 .−

Revised: − −

Revised: RQ

Revised: [Table]

Original:  We experimented with learning rates in the range of [5 e − 6 , 5 e − 5 , 1 e − 4 ] for 5 / 5 , 6 / 6 and 8 / 8 bit-width variants, but could not achieve signiﬁcant accuracy improvements within the ﬁrst 10 epochs of ﬁne-tuning of the high-precision model published at https://github.com/tensorflow/models/blob/master/research/ slim/nets/mobilenet_v1.md .
Revised: [Table] [Table] We attempted to use the provided training scripts in the https://github.com/tensorflow/ models/blob/master/research/slim repository to train lower-bit mobilenet variants, but did not succeed in doing so. We experimented with learning rates in the range of [5 e − 6 , 5 e − 5 , 1 e − 4 ] for 5 / 5 , 6 / 6 and 8 / 8 bit-width variants, but could not achieve signiﬁcant accuracy improvements within the ﬁrst 10 epochs of ﬁne-tuning of the high-precision model published at https://github.com/tensorflow/models/blob/master/research/ slim/nets/mobilenet_v1.md .

Original: After 10 epochs, the 8 / 8 version achieved 31 .
Revised: After 10 epochs, the 8 / 8 version achieved 31 .

Original: 39 top-1 error with a learning rate of 1 e − 4 and as such is worse than the published results.
Revised: 39 top-1 error with a learning rate of 1 e − 4 and as such is worse than the published results.

Original: We therefore chose to only include the published numbers for the 8 / 8 bit model and leave addition hyperparameter tuning to future work.
Revised: We therefore chose to only include the published numbers for the 8 / 8 bit model and leave addition hyperparameter tuning to future work.