Original: A DDITIVE P OWERS - OF -T WO Q UANTIZATION : A N ONUNIFORM D ISCRETIZATION FOR N EURAL N ETWORKS
Revised: A DDITIVE P OWERS - OF -T WO Q UANTIZATION : A N ONUNIFORM D ISCRETIZATION FOR N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We proposed Additive Powers-of-Two (APoT) quantization, an efﬁcient nonuniform quantization scheme that attends to the bell-shaped and long-tailed distribution of weights in neural networks.
Revised: We proposed Additive Powers-of-Two (APoT) quantization, an efﬁcient nonuniform quantization scheme that attends to the bell-shaped and long-tailed distribution of weights in neural networks.

Original: By constraining all quantization levels as a sum of several Powers-of-Two terms, APoT quantization enjoys  overwhelming efﬁciency of computation and a good match with weights’ distribution.
Revised: By constraining all quantization levels as a sum of several Powers-of-Two terms, APoT quantization enjoys the overwhelming efﬁciency of computation and a good match with weights’ distribution.

Original: A simple reparameterization on  clipping function is applied to generate better-deﬁned gradient for updating of optimal clipping threshold.
Revised: A simple reparameterization on the clipping function is applied to generate a betterdeﬁned gradient for updating of optimal clipping threshold.

Original: Moreover, weight normalization is presented to reﬁne the input distribution of weights to be more stable and consistent.
Revised: Moreover, weight normalization is presented to reﬁne the input distribution of weights to be more stable and consistent.

Original: Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision models demonstrating the effectiveness of our proposed APoT quantization.
Revised: Experimental results show that our proposed method out- performs state -of-the-art methods , and is even competitive with the full-precision models demonstrating the effectiveness of our proposed APoT quantization.

Original: For example, our 3-bit quantized ResNet-34 on ImageNet only drops 0.3% Top-1 and 0.2%
Revised: For example our 5-bit quantized ResNet-50 on ImageNet achieves

Original: Top-5 accuracy without bells and whistles, while the computation of our model is approximately 2 × less than uniformly quantized neural networks.
Revised: 8% top-1 accuracy without bells and whistles meanwhile our model is capable to decrease % ﬁxed point computation overhead in uniformly quantized counterpart.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Neural Networks (DNNs) have made a signiﬁcant improvement for various real-world applications.
Revised: Deep Neural Networks (DNNs) have made a signiﬁcant improvement for various real-world applications.

Original: However, the huge memory and computational cost impede the mass deployment of DNNs, e.g., on resource-constrained devices.
Revised: However, the huge memory and computational cost impede the mass deployment of DNNs, e.g., on resource-constrained devices.

Original: To the reduce memory footprint and computational burden, several model compression methods such as quantization (Zhou et al., 2016), pruning (Han et al., 2015) and low-rank decomposition (Denil et al., 2013) have been widely explored.
Revised: To the reduce memory footprint and computational burden, several model compression methods such as quantization (Zhou et al., 2016), pruning (Han et al., 2015) and low-rank decomposition (Denil et al., 2013) have been widely explored.

Original: In this paper, we focus on the neural network quantization for efﬁcient inference.
Revised: In this paper, we focus on the neural network quantization for efﬁcient inference.

Original: Two operations are involved in the quantization process, namely clipping and projection.
Revised: Two operations are involved in the quantization process, namely clipping and projection.

Original: The clipping operation sets a full precision number to the range boundary if it is outside of the range; the projection operation maps each number (after clipping) into a predeﬁned quantization level (a ﬁxed number).
Revised: The clipping operation sets a full precision number to the range boundary if it is outside of the range; the projection operation maps each number (after clipping) into a predeﬁned quantization level (a ﬁxed number).

Original: We can see that both operations incur information loss.
Revised: We can see that both operations incur information loss.

Original: A good quantization method should resolve the two following questions/challenges, which correspond to two contradictions respectively.
Revised: A good quantization method should resolve the two following questions/challenges, which correspond to two contradictions respectively.

Original: How to determine the optimal clipping threshold to balance clipping range and projection resolution?
Revised: How to determine the optimal clipping threshold to balance clipping range and projection resolution?

Original: The resolution indicates the interval between two quantization levels; the smaller the interval, the higher the resolution.
Revised: The resolution indicates the interval between two quantization levels; the smaller the interval, the higher the resolution.

Original: The ﬁrst contradiction is that given a ﬁxed number of bits to represent weights, the range and resolution are inversely proportional.
Revised: The ﬁrst contradiction is that given a ﬁxed number of bits to represent weights, the range and resolution are inversely proportional.

Original: For example, a larger range can clip fewer weights; however, the resolution becomes smaller and thus damage the projection.
Revised: For example, a larger range can clip fewer weights; however, the resolution becomes smaller and thus damage the projection.

Original: Note that slipshod clipping of outliers can jeopardize the network a lot (Zhao et al., 2019) although they may only take 1-2% of all weights in one layer.
Revised: Note that slipshod clipping of outliers can jeopardize the network a lot (Zhao et al., 2019) although they may only take 1-2% of all weights in one layer.

Original: Previous works have tried either pre-deﬁned (Cai et al., 2017;
Revised: Previous works have tried either pre-deﬁned (Cai et al., 2017;

Original: Zhou et al., 2016) or trainable (Choi et al., 2018b) clipping thresholds, but how to ﬁnd the optimal threshold during training automatically is still not resolved.
Revised: Zhou et al., 2016) or trainable (Choi et al., 2018b) clipping thresholds, but how to ﬁnd the optimal threshold during training automatically is still not resolved.

Original: How to design quantization levels with consideration for both the computational efﬁciency and the distribution of weights?
Revised: How to design quantization levels with consideration for both the computational efﬁciency and the distribution of weights?

Original: Most of the existing quantization approaches (Cai et al., 2017; Gong et al., 2019), use uniform quantization although non-uniform quantization can usually achieve better accuracy (Zhu et al., 2016).
Revised: Most of the existing quantization approaches (Cai et al., 2017; Gong et al., 2019), use uniform quantization although non-uniform quantization can usually achieve better accuracy (Zhu et al., 2016).

Original: The reason is that projection against uniform quantization levels are much more hardware-friendly (Zhou et al., 2016).
Revised: The reason is that projection against uniform quantization levels are much more hardware-friendly (Zhou et al., 2016).

Original: However, empirical study (Han et al., 2015) has shown that weights in a layer of DNN follow a bell-shaped and long-tailed distribution instead of a uniform distribution.
Revised: However, empirical study (Han et al., 2015) has shown that weights in a layer of DNN follow a bell-shaped and long-tailed distribution instead of a uniform distribution.

Original: In other words, a fair percentage of weights concentrate around the mean; and a few weights are of relatively high magnitude and out of the quantization range (called outliers).
Revised: In other words, a fair percentage of weights concentrate around the mean; and a few weights are of relatively high magnitude and out of the quantization range (called outliers).

Original: The second contradiction is: considering the bell-shaped distribution of weight, it is well-motivated to assign higher resolution (i.e. smaller quantization interval) around the mean; however, such non-uniform quantization levels will introduce high computational overhead.
Revised: The second contradiction is: considering the bell-shaped distribution of weight, it is well-motivated to assign higher resolution (i.e. smaller quantization interval) around the mean; however, such non-uniform quantization levels will introduce high computational overhead.

Original: Powers-of-Two quantization levels (Miyashita et al., 2016;
Revised: Powers-of-Two quantization levels (Miyashita et al., 2016;

Original: Zhou et al., 2017) are then proposed because of its cheap multiplication implemented by shift operations on hardware, and super high resolution around the mean.
Revised: Zhou et al., 2017) are then proposed because of its cheap multiplication implemented by shift operations on hardware, and super high resolution around the mean.

Original: However, the vanilla powers-of-two quantization method only increases the resolution near the mean and ignores other regions at all when the bit-width is increased.
Revised: However, the vanilla powers-of-two quantization method only increases the resolution near the mean and ignores other regions at all when the bit-width is increased.

Original: Consequently, it assigns inordinate quantization levels for a tiny range around the mean.
Revised: Consequently, it assigns inordinate quantization levels for a tiny range around the mean.

Original: To resolve the two challenges above, we propose multiple novel quantization techniques as follows:
Revised: To resolve the two challenges above, we propose multiple novel quantization techniques as follows:

Original: We introduce the Additive Powers-of-Two (APoT) quantization scheme for the weights of DNNs.
Revised: We introduce the Additive Powers-of-Two (APoT) quantization scheme for the weights of DNNs.

Original: APoT is a non-uniform quantization scheme, in which the quantization levels is a sum of several PoT terms and can adapt well to the bell-shaped distribution of weights.
Revised: APoT is a non-uniform quantization scheme, in which the quantization levels is a sum of several PoT terms and can adapt well to the bell-shaped distribution of weights.

Original: APoT quantization enjoys a 2 ×  speed-up compared with uniform quantization on both general and speciﬁc hardware.
Revised: APoT quantization enjoys a 2 × multiplication speed-up compared with uniform quantization on both general and speciﬁc hardware.

Original: We propose a Reparameterized Clipping Function (RCF) that can compute a more accurate gradient for the clipping threshold and thus facilitate the optimization of the clipping threshold.
Revised: We propose a Reparameterized Clipping Function (RCF) that can compute a more accurate gradient for the clipping threshold and thus facilitate the optimization of the clipping threshold.

Original: We introduce weight normalization for neural network quantization.
Revised: We introduce weight normalization for neural network quantization.

Original: Normalized weights in the forward pass are more stable and consistent for clipping and projection.
Revised: Normalized weights in the forward pass are more stable and consistent for clipping and projection.

Original: Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision implementation with higher computational efﬁciency.
Revised: Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision implementation with higher computational efﬁciency.

Original: Speciﬁcally, our 3-bit quantized ResNet-34 on ImageNet only drops 0.3% Top-1 and 0.2% Topaccuracy.
Revised: Speciﬁcally, our 5-bit quantized ResNet-50 on ImageNet achieve 76.8% Top-1 and 93.3% Topaccuracy.

Revised: Compared with uniform quantization, our method can further decrease 31% computation, demonstrating the proposed algorithm is hardware-friendly.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we ﬁrst give some preliminaries in uniform quantization and PoT quantization.
Revised: In this section, we ﬁrst give some preliminaries in uniform quantization and PoT quantization.

Original: Then we introduce our APoT quantization and RCF.
Revised: Then we introduce our APoT quantization and RCF.

Original: Lastly, we present a weights normalization technique for quantized neural networks.
Revised: Lastly, we present a weights normalization technique for quantized neural networks.

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Suppose kernels in a convolutional layer are represented by a 4D tensor W ∈ R C out × C in × K × K , where C out and C in are the number of output and input channels respectively, and K is the kernel size.
Revised: Suppose kernels in a convolutional layer are represented by a 4D tensor W ∈ R C out × C in × K × K , where C out and C in are the number of output and input channels respectively, and K is the kernel size.

Original: We denote the quantization of the weights as
Revised: We denote the quantization of the weights as

Original: [Equation]
Revised: [Equation]

Original: where α is clipping threshold and the clipping function (cid:98)· , α (cid:101) clips weights into [ − α, α ] .
Revised: where α is clipping threshold and the clipping function (cid:98)· , α (cid:101) clips weights into [ − α, α ] .

Original: After clipping, each element of W is projected by Π( · ) onto the quantization levels.
Revised: After clipping, each element of W is projected by Π( · ) onto the quantization levels.

Original: We denote Q ( α, b ) for a set of quantization levels, where b is the bit-width.
Revised: We denote Q ( α, b ) for a set of quantization levels, where b is the bit-width.

Original: For uniform quantization, the quantization levels are deﬁned as
Revised: For uniform quantization, the quantization levels are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For every ﬂoating-point number, uniform quantization maps it to a b -bit ﬁxed-point representation (quantization levels) in Q u ( α, b ) .
Revised: For every ﬂoating-point number, uniform quantization maps it to a b -bit ﬁxed-point representation (quantization levels) in Q u ( α, b ) .

Original: Note that α is stored separately as a full-precision ﬂoating-point number for each whole W .
Revised: Note that α is stored separately as a full-precision ﬂoating-point number for each whole W .

Original: Convolution is done against the quantization levels ﬁrst and the results are then multiplied by α .
Revised: Convolution is done against the quantization levels ﬁrst and the results are then multiplied by α .

Original: Arithmetical computation, e.g., convolution, can be implemented using low-precision ﬁxed point operations on hardware, which are substantially cheaper than their ﬂoating-point contradictory (Goldberg, 1991).
Revised: Arithmetical computation, e.g., convolution, can be implemented using low-precision ﬁxed point operations on hardware, which are substantially cheaper than their ﬂoating-point contradictory (Goldberg, 1991).

Original: Nevertheless, uniform quantization does not match the distribution of weights, which is typically bell-shaped.
Revised: Nevertheless, uniform quantization does not match the distribution of weights, which is typically bell-shaped.

Original: A straightforward solution is to assign more quantization levels (higher resolution) for the peak of the distribution and fewer levels (lower resolution) for the tails.
Revised: A straightforward solution is to assign more quantization levels (higher resolution) for the peak of the distribution and fewer levels (lower resolution) for the tails.

Original: However, it is difﬁcult to implement the arithmetical operations for the non-uniform quantization levels efﬁciently.1.0 Float 0. Q u a n t i z e d (a) Uniform Quantization 0.
Revised: However, it is difﬁcult to implement the arithmetical operations for the non-uniform quantization levels efﬁciently.

Original: 1.0 Float 0.

Original: Q u a n t i z e d 0.00 0.02 0.

Original: [Figure]
Revised: [Figure]

Original: Quantization 0.

Original: 1.0 Float 0.

Original: Q u a n t i

Original: z

Original: e d

Original: [Figure]

Original: 2. A DDITIVE P OWERS - OF -T WO Q UANTIZATION
Revised: 2. A DDITIVE P OWERS - OF -T WO Q UANTIZATION

Original: To solve the contradiction between non-uniform resolution and hardware efﬁciency, Powers-ofTwo (PoT) quantization (Miyashita et al., 2016; Zhou et al., 2017) is proposed by constraining quantization levels to be powers-of-two values or zero, i.e.,
Revised: To solve the contradiction between non-uniform resolution and hardware efﬁciency, Powers-ofTwo (PoT) quantization (Miyashita et al., 2016; Zhou et al., 2017) is proposed by constraining quantization levels to be powers-of-two values or zero, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Apparently, as a non-uniform quantizer, PoT has a higher resolution for the value range with denser weights because of its exponential property.
Revised: Apparently, as a non-uniform quantizer, PoT has a higher resolution for the value range with denser weights because of its exponential property.

Original: Furthermore, multiplication between a Powers-of-two number 2 k and the other operand x can be implemented by bit-wise shift instead of bulky digital multipliers, i.e., 
Revised: Furthermore, multiplication between a Powers-of-two number 2 x and the other operand r can be implemented by bit-wise shift instead of bulky digital multipliers, i.e., [Equation] if

Original: [Equation]
Revised: [Equation]

Original: where >> denotes the right shift operation and is computationally cheap, which only takes 1 clock cycle in modern CPU architectures.
Revised: where >> denotes the right shift operation and is computationally cheap, which only takes 1 clock cycle in modern CPU architectures.

Original: However, we ﬁnd that in PoT quantization does not beneﬁt from more bits.
Revised: However, we ﬁnd that in PoT quantization does not beneﬁt from more bits.

Original: Assume α is 1, as shown in Equation (3), when we increase the bit-width from b to b + 1 , the interval [0 , ± 2 − 2 b − 1 +1 ] will be split into 2 b  sub-intervals, whereas all other intervals remain unchanged.
Revised: Assume α is 1, as shown in Equation (3), when we increase the bit-width from b to b + 1 , the interval [0 , ± 2 − 2 b − 1 +1 ] will be split into 2 b − 1 − 1 sub-intervals, whereas all other intervals remain unchanged.

Original: In other words, by increasing the bit-width, the resolution will increase only for [ − 2 − 2 b − 1 +1 , 2 − 2 b − 1 +1 ] .
Revised: In other words, by increasing the bit-width, the resolution will increase only for [ − 2 − 2 b − 1 +1 , 2 − 2 b − 1 +1 ] .

Original: We refer this phenomenon as the rigid resolution of PoT quantization.
Revised: We refer this phenomenon as the rigid resolution of PoT quantization.

Original: Take Q p (1 , 5) as an example, the two smallest positive levels are 2 − 15 and 2 − 14 , which is excessive ﬁne-grained.
Revised: Take Q p (1 , 5) as an example, the two smallest positive levels are 2 − 15 and 2 − 14 , which is excessive ﬁne-grained.

Original: In contrast, the two largest levels are 2 − 1 and 2 0 , whose interval is large enough to incur high projection error for weights between [2 − 1 , 2 0 ] , e.g., 0.75.
Revised: In contrast, the two largest levels are 2 − 1 and 2 0 , whose interval is large enough to incur high projection error for weights between [2 − 1 , 2 0 ] , e.g., 0.75.

Original: To tackle the rigid resolution problem, we propose Additive Powers-of-Two (APoT) quantization.
Revised: To tackle the rigid resolution problem, we propose Additive Powers-of-Two (APoT) quantization.

Original: Without loss of generality, in this section, we only consider unsigned numbers for simplicity 1 .
Revised: Without loss of generality, in this section, we only consider unsigned numbers for simplicity 1 .

Original: In
Revised: In

Original: APoT quantization, levels are viewed as a sum of several PoT terms as shown below,
Revised: APoT quantization, levels are viewed as a sum of several PoT terms as shown below,

Original: [Equation]
Revised: [Equation]

Original: where γ is a scaling coefﬁcient to make sure the maximum level in Q a is α .
Revised: where γ is a scaling coefﬁcient to make sure the maximum level in Q a is α .

Original: k is called the base bit-width, which is the bit-width for each additive term, and n is the number of additive terms.
Revised: k is called the base bit-width, which is the bit-width for each additive term, and n is the number of additive terms.

Original: When the bit-width b and the base bit-width k is set, n can be calculated by n = bk .
Revised: When the bit-width b and the base bit-width k is set, n can be calculated by n = bk .

Original: The number of additive terms in APoT quantization can increase with bit-width b , which provides ﬂexible resolution for the non-uniform levels.
Revised: The number of additive terms in APoT quantization can increase with bit-width b , which provides ﬂexible resolution for the non-uniform levels.

Original: We use b = 4 and k = 2 as an example to illustrate how APoT resolves the rigid resolution problem.
Revised: We use b = 4 and k = 2 as an example to illustrate how APoT resolves the rigid resolution problem.

Original: For this example, we have p 1 ∈ { 0 , 2 0 , 2 − 2 , 2 − 4 } , p 2 ∈ { 0 , 2 − 1 , 2 − 3 , 2 − 5 } and γ = 2 α/ 3 .
Revised: For this example, we have p 1 ∈ { 0 , 2 0 , 2 − 2 , 2 − 4 } , p 2 ∈ { 0 , 2 − 1 , 2 − 3 , 2 − 5 } and γ = 2 α/ 3 .

Original: Firstly, we can see the smallest positive quantization level in Q a (1 , 4) is 2 − 4 / 3 . 
Revised: Firstly, we can see the smallest positive quantization level in Q a (1 , 4) is 2 − 4 / 3 . Compared with [Figure] Uniform

Original: Compared with the original PoT levels, APoT allocates quantization levels prudently for the central area.
Revised: APoT PoT (k=1) (k=2) (k=b) [Figure] Activations Buffer

Revised: Weights Buffer

Revised: [Figure]

Revised: +

Revised: Accumulator

Revised: Shiftxbit

Revised: Output

Revised: Sign α × the original PoT levels, APoT allocates quantization levels prudently for the central area.

Original: Secondly,
Revised: Secondly,

Original: APoT generates 3 new quantization levels between 2 0 and 2 − 1 , to properly increase the resolution.
Revised: APoT generates 3 new quantization levels between 2 0 and 2 − 1 , to properly increase the resolution.

Original: Figure 1 compares the 3 quantization methods using 4 bits for range [0, 1].
Revised: Figure 1 compares the 3 quantization methods using 4 bits for range [0, 1].

Original: APoT quantization has a reasonable distribution of quantization levels, with more levels in the peak area (near 0) and relatively higher resolution than the vanilla PoT quantization at the tail (near 1).
Revised: APoT quantization has a reasonable distribution of quantization levels, with more levels in the peak area (near 0) and relatively higher resolution than the vanilla PoT quantization at the tail (near 1).

Original: Relation to other quantization methods.
Revised: Relation to other quantization methods.

Original: On the one hand, the ﬁxed-point number representations used in the uniform quantization is a special case of APoT. When k = 1 in Equation (5), the quantization levels is a sum of b PoT terms or 0.
Revised: On the one hand, the ﬁxed-point number representations used in the uniform quantization is a special case of APoT. When k = 1 in Equation (5), the quantization levels is a sum of b PoT terms or 0.

Original: In the ﬁxed-point representations, each bit indicates one speciﬁc choice of the additive terms.
Revised: In the ﬁxed-point representations, each bit indicates one speciﬁc choice of the additive terms.

Original: On the other hand, when k = b , there is only one PoT term and Q a ( α, b ) becomes Q p ( α, b ) , i.e., PoT quantization.
Revised: On the other hand, when k = b , there is only one PoT term and Q a ( α, b ) becomes Q p ( α, b ) , i.e., PoT quantization.

Original: We can conclude that when k decreases,
Revised: We can conclude that when k decreases,

Original: APoT levels are decomposed into more PoT terms, and the distribution of levels becomes more uniform.
Revised: APoT levels are decomposed into more PoT terms, and the distribution of levels becomes more uniform.

Original: Our experiments use k = 2 , which is an intermediate choice between the uniform case ( k = 1 ) and the vanilla PoT case ( k = b ).
Revised: Our experiments use k = 2 , which is an intermediate choice between the uniform case ( k = 1 ) and the vanilla PoT case ( k = b ).

Original: Computation.
Revised: Computation.

Original: Multiplication for ﬁxed-point numbers can be implemented by shifting the multiplicand (i.e., the activations) and adding the partial product.
Revised: Multiplication for ﬁxed-point numbers can be implemented by shifting the multiplicand (i.e., the activations) and adding the partial product.

Original: The n in Equation (5) denotes the number of additive PoT terms in the multiplier (weights), and control the speed of computation.
Revised: The n in Equation (5) denotes the number of additive PoT terms in the multiplier (weights), and control the speed of computation.

Original: Since n = bk , either decreasing b or increasing k can accelerate the multiplication.
Revised: Since n = bk , either decreasing b or increasing k can accelerate the multiplication.

Original: Compared with uniform quantization ( k = 1 ), our method ( k = 2 ) is approximately 2 × faster.
Revised: Compared with uniform quantization ( k = 1 ), our method ( k = 2 ) is approximately 2 × faster.

Original: As for the full precision α , it is a coefﬁcient for all weights in a layer and can be multiplied only once after the multiply-accumulate operation is ﬁnished.
Revised: As for the full precision α , it is a coefﬁcient for all weights in a layer and can be multiplied only once after the multiply-accumulate operation is ﬁnished.

Revised: Figure 2 shows the hardware accelerator, the weights buffer takes k -bit as a PoT term and shift-adds the activations.

Original: 2. R EPARAMETERIZED C LIPPING F UNCTION
Revised: 2. R EPARAMETERIZED C LIPPING F UNCTION

Original: Besides the projection operation, the clipping operation (cid:98) W , α (cid:101) is also important for quantization.
Revised: Besides the projection operation, the clipping operation (cid:98)W , α (cid:101) is also important for quantization.

Original: α is a threshold that determines the value range of weights in a quantized layer.
Revised: α is a threshold that determines the value range of weights in a quantized layer.

Original: Tuning the clipping threshold α is a key challenge because of the long-tail distribution of the weights.
Revised: Tuning the clipping threshold α is a key challenge because of the long-tail distribution of the weights.

Original: Particularly, if α is too large (e.g., the maximum absolute value of W ), Q ( α, b ) would have a wide range and then the projection will lead to large error as a result of insufﬁcient resolution for the weights in the central area; if α is too small, more outliers will be clipped slipshodly.
Revised: Particularly, if α is too large (e.g., the maximum absolute value of W ), Q ( α, b ) would have a wide range and then the projection will lead to large error as a result of insufﬁcient resolution for the weights in the central area; if α is too small, more outliers will be clipped slipshodly.

Original: Considering the distribution of weights can be complex and differs across layers and training steps, a static clipping threshold α for all layers is not optimal.
Revised: Considering the distribution of weights can be complex and differs across layers and training steps, a static clipping threshold α for all layers is not optimal.

Original: To jointly optimize the clipping threshold α and weights via SGD during training, Choi et al.
Revised: To jointly optimize the clipping threshold α and weights via SGD during training, Choi et al.

Original: (2018b) apply the Straight-Through Estimator (STE) (Bengio et al., 2013) to do the backward propagation for the projection operation.
Revised: (2018b) apply the Straight-Through Estimator (STE) (Bengio et al., 2013) to do the backward propagation for the projection operation.

Original: According to STE, we have
Revised: According to STE, we have

Original: [Equation]
Revised: [Equation]

Original: where the weights outside of the range cannot contribute to the gradients, which results in inaccurate gradient approximation.
Revised: where the weights outside of the range cannot contribute to the gradients, which results in inaccurate gradient approximation.

Original: To provide a reﬁned gradient for the clipping threshold, we design a
Revised: To provide a reﬁned gradient for the clipping threshold, we design a

Original: Reparameterized Clipping Function (RCF) as
Revised: Reparameterized Clipping Function (RCF) as

Original: [Equation]
Revised: [Equation]

Original:  Instead of directly clipping them to [ − α, α ] , RCF outputs a constant clipping range and rescales weights back after the projection, which is mathematically equivalent to Equation (1) during forward.
Revised: [Figure] Instead of directly clipping them to [ − α, α ] , RCF outputs a constant clipping range and rescales weights back after the projection, which is mathematically equivalent to Equation (1) during forward.

Original: In backpropagation, STE is adopted for the projection operation and the gradients of α are calculated by
Revised: In backpropagation, STE is adopted for the projection operation and the gradients of α are calculated by

Original: [Equation]
Revised: [Equation]

Original: Compared with the normal clipping function, RCF provides more accurate gradient signals for the optimization because both weights inside ( | W | ≤ α ) and out of ( | W | > α ) the range can contribute to the gradient for the clipping threshold.
Revised: Compared with the normal clipping function, RCF provides more accurate gradient signals for the optimization because both weights inside ( |W| ≤ α ) and out of ( |W| > α ) the range can contribute to the gradient for the clipping threshold.

Original: Particularly, the outliers are responsible for the clipping and the weights in [ − α, α ] are for projection.
Revised: Particularly, the outliers are responsible for the clipping and the weights in [ − α, α ] are for projection.

Original: Therefore, the update of α considers both clipping and projection and ﬁnds a sweet spot between them.
Revised: Therefore, the update of α considers both clipping and projection and ﬁnds a sweet spot between them.

Original: [Figure] In experiments, we observe that the clipping threshold will become universally smaller when bitwidth is reduced to guarantee sufﬁcient resolution, which further validates the effectiveness of the gradient in RCF.
Revised: In experiments, we observe that the clipping threshold will become universally smaller when bitwidth is reduced to guarantee sufﬁcient resolution, which further validates the effectiveness of the gradient in RCF.

Original: 2. W EIGHT N ORMALIZATION
Revised: 2. W EIGHT N ORMALIZATION

Original: In practice, we ﬁnd that learning α for weights is quite an arduous process because the distribution of weights is pretty steep and changes frequently during training.
Revised: In practice, we ﬁnd that learning α for weights is quite an arduous process because the distribution of weights is pretty steep and changes frequently during training.

Original: As a result, jointly training the clipping threshold and weights parameters is hard to converge.
Revised: As a result, jointly training the clipping threshold and weights parameters is hard to converge.

Original: Inspired by the crucial role of batch normalization (BN) (Ioffe & Szegedy, 2015) in activation quantization (Cai et al., 2017), we propose weight normalization (WN) to reﬁne the distribution of weights with zero mean and unit variance,˜ W = W − µ σ + (cid:15) , where µ = 1 I I (cid:88) i = W i , [Equation]
Revised: Inspired by the crucial role of batch normalization (BN) (Ioffe & Szegedy, 2015) in activation quantization (Cai et al., 2017), we propose weight normalization (WN) to reﬁne the distribution of weights with zero mean and unit variance,

Original: = (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is a small number (typically 10 − 5 ) for numerical stability, and I denotes the number of weights in one layer.
Revised: where (cid:15) is a small number (typically 10 − 5 ) for numerical stability, and I denotes the number of weights in one layer.

Original: Note that quantization of weights is applied right after this normalization.
Revised: Note that quantization of weights is applied right after this normalization.

Original: Normalization is important to provide a relatively consistent and stable input distribution to both clipping and projection functions for smoother optimization of α over different layers and iterations during training.
Revised: Normalization is important to provide a relatively consistent and stable input distribution to both clipping and projection functions for smoother optimization of α over different layers and iterations during training.

Original: Besides, making the mean of weights to be zero can help make full use of the symmetric design of the quantization levels.
Revised: Besides, making the mean of weights to be zero can reap the beneﬁts of the symmetric design of the quantization levels.

Original: Here, we conduct a case study of ResNet-20 on CIFAR10 to illustrate how normalization for weights can help quantization.
Revised: Here, we conduct a case study of ResNet-20 on CIFAR10 to illustrate how normalization for weights can help quantization.

Original: For a certain layer (at a certain training step) in ResNet-18, We ﬁrstly ﬁx the weights,  let α go from 0 to max ( | W | ) and plot the curve of  clipping ratio (i.e. the proportion of clipped weights).
Revised: For a certain layer (at a certain training step) in ResNet-18, We ﬁrstly ﬁx the weights, and let α go from 0 to max |W| to plot the curve of the clipping ratio (i.e. the proportion of clipped weights).

Original: As shown in Figure 2a, the change of clipping ratio is much smoother after quantization.
Revised: As shown in Figure 3a, the change of clipping ratio is much smoother after quantization.

Original: As a result, the optimization of α will be signiﬁcantly smoother.
Revised: As a result, the optimization of α will be signiﬁcantly smoother.

Original: In addition, normalization also makes the distribution of weights quite more consistent over training iterations.
Revised: In addition, normalization also makes the distribution of weights quite more consistent over training iterations.

Original: We ﬁx the value of α , and visualize clipping ratio over training iterations in Figure 2b.
Revised: We ﬁx the value of α , and visualize clipping ratio over training iterations in Figure 3b.

Original: After normalization, the same α will result in almost the same clipping ratio, which improves the consistency of optimization goal for α .
Revised: After normalization, the same α will result in almost the same clipping ratio, which improves the consistency of optimization goal for α .

Original: More experimental analysis demonstrating the effectiveness of the normalization on weights can be found in Appendix A.
Revised: More experimental analysis demonstrating the effectiveness of the normalization on weights can be found in Appendix A.

Original: 2. T RAINING AND D EPLOYING
Revised: 2. T RAINING AND D EPLOYING

Original: For activation, we use  uniform quantization as we observe APoT does not outperform uniform quantization signiﬁcantly for activation.
Revised: For activations, we use the uniform quantization as we observe APoT does not outperform uniform quantization signiﬁcantly for activation.

Original: During backpropagation, we use STE when computing the gradients of weights, i.e. ∂ ˆ W ∂ ˜ W = 1 .
Revised: During backpropagation, we use STE when computing the gradients of weights, i.e. ∂ ˆ W ∂ ˜ W = 1 .

Original: The detailed training procedure is shown in Algorithm 1.
Revised: The detailed training procedure is shown in Algorithm 1.

Original: For efﬁcient inference, we discard the full precision weights W and only store the quantized weights ˆ
Revised: For efﬁcient inference, we discard the full precision weights W and only store the quantized weights ˆ

Original: W .
Revised: W .

Original: Which means our proposed normalization on weights does not occupy any memory or computation resources during inference.
Revised: Which means our proposed normalization on weights does not occupy any memory or computation resources during inference.

Original: Compared with other uniform quantization methods, APoT quantization is more efﬁcient and effective during inference.[Table] [Table] [Table]
Revised: Compared with other uniform quantization methods, APoT quantization is more efﬁcient and effective during inference.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Non-Uniform Quantization.
Revised: Non-Uniform Quantization.

Original: Several methods are proposed to adapt to the non-uniform distribution of weights.
Revised: Several methods are proposed to adapt to the non-uniform distribution of weights.

Original: LQ-Nets(Zhang et al., 2018) learns quantization levels based on the quantization error minimization (QEM) algorithm.
Revised: LQ-Nets(Zhang et al., 2018) learns quantization levels based on the quantization error minimization (QEM) algorithm.

Original: Distillation (Polino et al., 2018) optimizes the quantization levels directly to minimize the task loss which reﬂects the behavior of their teacher network.
Revised: Distillation (Polino et al., 2018) optimizes the quantization levels directly to minimize the task loss which reﬂects the behavior of their teacher network.

Original: These methods use ﬁnite ﬂoating-point numbers to quantize weights (and activations), bringing extra computation overhead compared with linear quantization.
Revised: These methods use ﬁnite ﬂoating-point numbers to quantize weights (and activations), bringing extra computation overhead compared with linear quantization.

Original: Logarithmic quantizers (Zhou et al., 2017;
Revised: Logarithmic quantizers (Zhou et al., 2017;

Original: Miyashita et al., 2016) leverage powers-of-2 values to accelerate the computation by shift operations; however, they suffer from the rigid resolution problem.
Revised: Miyashita et al., 2016) leverage powers-of-2 values to accelerate the computation by shift operations; however, they suffer from the rigid resolution problem.

Original: Jointly Training.
Revised: Jointly Training.

Original: Many works have been explored to optimize the quantization parameters (e.g., α ) and the weights parameters simultaneously.
Revised: Many works have been explored to optimize the quantization parameters (e.g., α ) and the weights parameters simultaneously.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2016) learns positive and negative scaling coefﬁcients respectively.
Revised: (2016) learns positive and negative scaling coefﬁcients respectively.

Original: LQ-Nets jointly train these parameters to minimize the quantization error.
Revised: LQ-Nets jointly train these parameters to minimize the quantization error.

Original: QIL (Jung et al., 2019) introduces a learnable transformer to change the quantization intervals and optimize them based on the task loss.
Revised: QIL (Jung et al., 2019) introduces a learnable transformer to change the quantization intervals and optimize them based on the task loss.

Original: PACT (Choi et al., 2018b) parameterizes the clipping threshold in activations and optimize it through gradient descent.
Revised: PACT (Choi et al., 2018b) parameterizes the clipping threshold in activations and optimize it through gradient descent.

Original: However, in PACT, the gradient of α is not accurate, which only includes the contribution from outliers and ignores the contribution from other weights.
Revised: However, in PACT, the gradient of α is not accurate, which only includes the contribution from outliers and ignores the contribution from other weights.

Original: Weight Normalization.
Revised: Weight Normalization.

Original: Previous works on weights normalization mainly focus on addressing the limitations of BatchNorm (Ioffe & Szegedy, 2015).
Revised: Previous works on weight normalization mainly focus on addressing the limitations of BatchNorm (Ioffe & Szegedy, 2015).

Original: Salimans & Kingma (2016); Hoffer et al.
Revised: Salimans & Kingma (2016); Hoffer et al.

Original: (2018) decouple direction from magnitude to accelerate the training procedure.
Revised: (2018) decouple direction from magnitude to accelerate the training procedure.

Original: Weight Standardization (Qiao et al., 2019) normalizes weights to zero mean and unit variance during  forward pass.
Revised: Weight Standardization (Qiao et al., 2019) normalizes weights to zero mean and unit variance during the forward pass.

Original: However, there is limited literature that studies the normalization on weights for neural network quantization.
Revised: However, there is limited literature that studies the normalization of weights for neural network quantization.

Original: (Zhu et al., 2016) uses feature-scaling to normalize weights by dividing the maximum absolute value.
Revised: (Zhu et al., 2016) uses feature-scaling to normalize weights by dividing the maximum absolute value.

Revised: Weight Normalization based Quantization (Cai & Li, 2019)

Revised: also uses this feature scaling and derive the gradient to eliminate the outliers in the weights tensor.

Revised: [Table]

Revised: [Table]

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we validate our proposed method on ImageNet-ILSVRC

Revised: In this section [Table] we validate our proposed method on ImageNet-ILSVRC

Original: (Russakovsky et al., 2015)
Revised: (Russakovsky et al., 2015)

Original: and CIFAR10 (Krizhevsky et al., 2009) datasets.
Revised: and CIFAR10 (Krizhevsky et al., 2009).

Original: We also conducta number of ablation studies investigating different quantizers.
Revised: We also conduct ablation studies investigating different quantizers and our proposed RCF as well as Weight Normalization

Original: The checkpoints of quantized models are released anonymously.
Revised: [Table]

Original: 4. E VALUATION ON I MAGE N ET
Revised: 4. E VALUATION ON I MAGE N ET

Original: We compare our methods with several strong state-of-the-art methods on ResNet-18 and ResNet34 (He et al., 2016), including DoReFa-Net (Zhou et al., 2016), PACT (Choi et al., 2018b), LQ-
Revised: We compare our methods with several strong state-of-the-art methods on ResNet architectures (He et al., 2016), including DoReFa-Net (Zhou et al., 2016), PACT (Choi et al., 2018b), LQ-Net (Zhang et al., 2018), DSQ (Gong et al., 2019), QIL (Jung et al., 2019). Both weights and activations of

Original: [Figure]
Revised: [Table]

Original: Net (Zhang et al., 2018), DSQ
Revised: ET AL . , 2018 A ) [Table]

Original: (Gong et al., 2019), QIL (Jung et al., 2019). Both weights and activations of the networks are quantized for comparisons. 
Revised: the networks are quantized for comparisons. All state-of-the-art methods use full precision (32 bits) for the ﬁrst and the last layer, which may introduce more memory cost. In our implementation, we employ 8-bit quantization for them to balance the accuracy drop and the hardware overhead.

Original: For our proposed APoT quantization algorithm, two conﬁgurations of the bit-width, i.e., 3 and( k = 2 and b = 1 or 2 in Equation (5)) 3 are tested.
Revised: For our proposed APoT quantization algorithm, two conﬁgurations of the bit-width, i.e., 3 and( k = 2 and b = 1 or 2 in Equation (5)), where one bit is used for the sign of the weights, are tested.

Original: We also test the 2-bit quantized network.
Revised: We also test the 2-bit quantized network.

Original: For the 2-bit symmetric quantization levels for weights, Q ( α, 2) = {± α, 0 } , therefore only RCF and WN are used in this setting.
Revised: For the 2-bit symmetric quantization levels for weights, Q ( α, 2) = {± α, 0 } , therefore only RCF and WN are used in this setting.

Original: To obtain a reasonable initialization, we follow Lin et al.
Revised: To obtain a reasonable initialization, we follow Lin et al.

Original: ; Jung et al.
Revised: ; Jung et al.

Original: (2019) to use a progressive way to initialize our model.
Revised: (2019) to use a progressive way to initialize our model.

Original: For instance, the 5-bit quantized model is initialized from the pre-trained full precision one 4 , while the 3-bit networks is initialized from the trained 5-bit model.
Revised: For instance, the 5-bit quantized model is initialized from the pre-trained full precision one 2 , while the 3-bit network is initialized from the trained 5-bit model.

Revised: We compare the accuracy, memory cost, and the ﬁxed point operations under different bit-width.

Revised: For the operation count, we assume the one F IX OP is one operation between 8-bit weights and activations.

Original: {+An operation of k 1 n 1 -bit weights and k 2 n 2 -bit activations is counted as n 1 × n 
+}
Revised: {+An operation of k 1 n 1 -bit weights and k 2 n 2 -bit activations is counted as n 1 × n 
+}

Revised: F IX OP.

Revised: Meanwhile, we omit the operations produced by Batch Normalization, as they are usually folded into convolution in hardware deployment.

Original: More details of the implementation are in the Appendix.
Revised: More details of the implementation are in the Appendix.

Original: Overall results are shown in Figure 3.
Revised: Overall results are shown in Table 1 − 3.

Original: The results of DoReFa-Net are taken from Choi et al.
Revised: The results of DoReFa-Net are taken from Choi et al.

Original: (2018b), and the other results are quoted from the original papers.
Revised: (2018b), and the other results are quoted from the original papers.

Original: It can be observed that our 5-bit quantized network achieves even higher accuracy than the full precision model (0.7% Top-1 improvement on ResNet-18 and 0.2% Top-1 improvement on ResNet-34).
Revised: It can be observed that our 5bit quantized network achieves even higher accuracy than the full precision baselines (0.7% Topimprovement on ResNet-18 and 0.2% Top-1 improvement on ResNet-34 and ResNet-50), which means quantization may serve the purpose of regularization.

Revised: Along with the accuracy performance, our APoT quantization can achieve better hardware performance on model size and F IX OPS when k is set to 2.

Original: Our 3-bit quantized networks are also approaching full-precision accuracy and only drop 0.5% and 0.3% accuracy on ResNet-18 and ResNet34.
Revised: Our 3-bit quantized networks are also approaching full-precision accuracy and only drop 0.5% and 0.3% accuracy on ResNet-18 and ResNet-34.

Original: When b is further reduced to 2, our model still outperforms baselines, which demonstrates the effectiveness of RCF and WN.
Revised: When b is further reduced to 2, our model still outperforms baselines, which demonstrates the effectiveness of RCF and WN.

Revised: Note that Choi et al.

Revised: (2018a) use a full precision shortcut in the model, reaching higher accuracy on ResNet-50 however suffering from the hardware performance, especially the different precision between the main path and the residual path may result in greater latency in a pipelined implementation.

Revised: html

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. E VALUATION ON CIFAR 
Revised: E VALUATION ON CIFAR

Original: We quantize ResNet-20 and ResNet-56 (He et al., 2016) on CIFAR10 for evaluation.
Revised: We quantize ResNet-20 and ResNet-56 (He et al., 2016) on CIFAR10 for evaluation.

Original: Again, we adopt progressive initialization and choose the quantization bit as 2, 3 and 5.
Revised: Again, we adopt progressive initialization and choose the quantization bit as 2, 3 and 5.

Original: More implementations and the test error curves can be found in the Appendix.
Revised: More implementations and the test error curves can be found in the Appendix.

Original: html

Original: Table.
Revised: Table.

Original: 1 summarizes the accuracy of our APoT in comparison with baselines.
Revised: 4 summarizes the accuracy of our APoT in comparison with baselines.

Original: For 3 bit and 5 bit models, APoT quantization surpasses the full precision accuracy by 0.6 to 0.7%.
Revised: For 3-bit and 5-bit models, APoT quantization has reached comparable results with the full precision baselines.

Original: It is worthwhile to note that all state-of-the-arts methods in the table use 4 levels to quantize weights into 2 bits.
Revised: It is worthwhile to note that all state-of-the-arts methods in the table use 4 levels to quantize weights into 2-bit.

Original: Our model only employs ternary weights for 2-bit representation and still achieves the highest accuracy.
Revised: Our model only employs ternary weights for 2-bit representation and still outstrips existing quantization methods. 4. A BLATION S TUDY

Original: [Figure]
Revised: [Table]

Revised: Our proposed algorithm consists of three techniques, APoT quantization levels to ﬁt the bell-shaped distribution, RCF to learn the clipping threshold and WN to avoid the perturbation of the distribution of weights during training.

Revised: In this section, we conduct an ablation study for these three techniques.

Revised: We compare the APoT quantizer, the vanilla PoT quantizer, uniform quantizer and a non-uniform quantizer using Lloyd algorithm

Revised: (Cai et al., 2017) to quantize the weights.

Revised: And we either apply

Revised: RCF to learn the optimal clipping range or do not clip any weights (i.e. α = max |W| ).

Revised: Weight

Revised: Normalization is also adopted or discarded to justify the effectiveness of these techniques.

Revised: Table 5 summarizes the results of ResNet-18 using different techniques.

Revised: Quantizer using Lloyd achieves the highest accuracy, however, the irregular non-uniform quantized weights cannot utilize the ﬁxed point arithmetic to accelerate the inference time.

Revised: APoT quantization attends to the distribution of weights, which reaches the same accuracy in 5-bit and only decreases 0.2% accuracy in 3-bit quantization compared with Lloyd, and shares a better tradeoff between task performance and hardware performance.

Revised: We also observe that the vanilla PoT quantization suffers from the rigid resolution , and has the lowest accuracy in the 5-bit model.

Revised: Clipping range also matters in quantization, the comparison in Table 5 shows that a proper clipping range can help improve the robustness of the network.

Revised: Especially when the network is quantized to

Revised: [Figure]

Revised: 3-bit, the accuracy will drop signiﬁcantly because of the quantization interval increases.

Revised: Applying

Revised: RCF to learn the optimal clipping range could improve at most 1.6% accuracy.

Revised: As we mentioned before, normalization of weight is important to learn the clipping range, and the network diverges if RCF is applied without WN.

Revised: We refer to the Appendix for more details of weight normalization during training.

Original: 4. R EVISITING Q UANTIZATION E RROR
Revised: 4. R EVISITING Q UANTIZATION E RROR

Original: Typically, quantization error  is deﬁned as the mean squared error between weights ˜ W andand ˆ W before and after quantization respectively.
Revised: Typically, quantization error ( ∆ ) is deﬁned as the mean squared error between weights ˜ W and ˆ W before and after quantization respectively, deﬁned as ∆ = E [ ˜ W − ˆ W ] 2 .

Original: The quantization error can be viewed as a sum of  clipping error  and  projection error, deﬁned as
Revised: This quantization error is composed of two errors, the clipping error ∆ clip produced by (cid:98)· , α (cid:101) and the projection error ∆ proj produced by Π Q . I.e.

Original: [Equation]
Revised: [Equation]

Original: Previous methods (Zhang et al., 2018; Cai et al., 2017) seek to minimize the quantization error(i.e. min( E clip + E proj ) ) to obtain the optimal clipping threshold, while RCF is directly optimized by the ﬁnal training loss to balance projection error and clipping error.
Revised: Previous methods (Zhang et al., 2018; Cai et al., 2017) seek to minimize the quantization error to obtain the optimal clipping threshold (i.e. α = arg min α (∆ clip + ∆ proj ) ), while RCF is directly optimized by the ﬁnal training loss to balance projection error and clipping error.

Original: We compare theQuantization
Revised: We compare the

Original:  Error Minimization (QEM) method with our RCF on the quantized ResNet-18 model.
Revised: Quantization Error Minimization (QEM) method with our RCF on the quantized ResNet-18 model.

Original: Figuregives an overview of  clipping error and projection error using RCF or QEM.
Revised: Figure 4 gives an overview of the clipping error and projection error using RCF or QEM.

Original: For the 5-bit quantized model, RCF has a much higher quantization error.
Revised: For the 5-bit quantized model, RCF has a much higher quantization error.

Original: The projection error obtained by RCF is lower than QEM and QEM signiﬁcantly reduces the clipping error.
Revised: The projection error obtained by RCF is lower than QEM and QEM signiﬁcantly reduces the clipping error.

Original: Based on the truth that RCF has better ﬁnal accuracy, we can infer that projection error has a higher priority in RCF.
Revised: Therefore, we can infer that projection error has a higher priority in RCF.

Original: When quantizing to 3-bit, the clipping error in RCF still exceeds QEM except for the ﬁrst quantized layer.
Revised: When quantizing to 3-bit, the clipping error in RCF still exceeds QEM except for the ﬁrst quantized layer.

Original: This means RCF can identify whether the projection is more important than the clipping over different layers and bit-width.
Revised: This means RCF can identify whether the projection is more important than the clipping over different layers and bit-width.

Original: Generally, the insight behind is that simply minimizing the quantization error may not be the best choice and it is more direct to optimize threshold with respect to training loss.
Revised: Generally, the insight behind is that simply minimizing the quantization error may not be the best choice and it is more direct to optimize threshold with respect to training loss.

Original: 4. Q UANTIZER C OMPARISON

Original: [Table]

Original: Since other quantization methods do not use RCF or normalization on weights, the effectiveness of our APoT quantization is not well evaluated.

Original: Hence we implement our algorithm with different quantization levels Q ( a, b ) while keeping other settings unchanged.

Original: Forbit quantized models, we evaluate uniform quantization, APoT quantization, and the vanilla PoT quantization.

Original: As for 3 bit, APoT resembles PoT, therefore we evaluate APoT quantization and uniform quantization.

Original: Table 2 shows the Top-1 accuracy of ResNet-18 on

Original: ImageNet with the above quantization methods.

Original: For both 3-bit and 5-bit setting, APoT achieves higher accuracy than uniform quantization as well as higher hardware efﬁciency.

Original: A H OW D OES N ORMALIZATION H ELP Q UANTIZATION
Revised: A H OW D OES N ORMALIZATION H ELP Q UANTIZATION

Original: In this section, we show some experimental results to illustrate the effect of our weights normalization in quantization neural networks.
Revised: In this section, we show some experimental results to illustrate the effect of our weights normalization in quantization neural networks.

Original: A. W EIGHTS D ISTRIBUTION
Revised: A. W EIGHTS D ISTRIBUTION

Original: We visualize the density distribution of weights before normalization W and after normalization ˜ W during training to demonstrate its effectiveness.
Revised: We visualize the density distribution of weights before normalization W and after normalization ˜ W during training to demonstrate its effectiveness.

Original: Figure 5a demonstrates the density distribution of the ﬁfth layer of the 5-bit quantized ResNet-18, form which we can see that the density of the unnormalized weights could be extensive high ( > 8) in the centered area.
Revised: Figure 5a demonstrates the density distribution of the ﬁfth layer of the 5-bit quantized ResNet-18, from which we can see that the density of the unnormalized weights could be extensive high ( > 8) in the centered area.

Original: Such distribution indicates that even a tiny change of clipping threshold would bring a signiﬁcant effect on clipping when α is small, as shown in Figure 2a. , which means a small learning rate for α is needed.
Revised: Such distribution indicates that even a tiny change of clipping threshold would bring a signiﬁcant effect on clipping when α is small, as shown in Figure 3a. , which means a small learning rate for α is needed.

Original: However, if the learning rate is too small, the change of α cannot follow the change of weights distribution because weights are also updated according to Figure 5a.
Revised: However, if the learning rate is too small, the change of α cannot follow the change of weights distribution because weights are also updated according to Figure 5a.

Original: Thus it is unfavourable to train the clipping threshold for unnormalized weights, while Figure 5b shows that the normalized weights can have a stable distribution.
Revised: Thus it is unfavorable to train the clipping threshold for unnormalized weights, while Figure 5b shows that the normalized weights can have a stable distribution.

Original: Furthermore, the dashed line in the ﬁgure indicates W usually do not have zero mean, which may not utilize the symmetric design of quantization levels.
Revised: Furthermore, the dashed line in the ﬁgure indicates W usually do not have zero mean, which may not utilize the symmetric design of quantization levels.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  T RAINING B EHAVIOR
Revised:  T RAINING B EHAVIOR

Original: The above experiments use normalization during training to compare the distribution of weights.
Revised: The above experiments use normalization during training to compare the distribution of weights.

Original: In this section, we compare the training of quantization neural networks with and without normalization to investigate the real effect of WN.
Revised: In this section, we compare the training of quantization neural networks with and without normalization to investigate the real effect of WN.

Original: Here, we train a 3-bit quantized (full precision for activations)
Revised: Here, we train a 3-bit quantized (full precision for activations)

Original: ResNet-20 from scratch, and compare the results under different learning rate for α .
Revised: ResNet-20 from scratch, and compare the results under different learning rate for α .

Original: The results is shown in Table 3, from which we can ﬁnd that if weights are normalized during training, the network can converge to descent performances and is robust to the learning rate of clipping threshold.
Revised: The results are shown in Table 6, from which we can ﬁnd that if weights are normalized during training, the network can converge to descent performances and is robust to the learning rate of clipping threshold.

Original: However, if the weights are not normalized, the network would diverge if the learning rate for α is too high.
Revised: However, if the weights are not normalized, the network would diverge if the learning rate for α is too high.

Original: Even if the learning rate is set to a lower value, the network do not outperform the normalized one.
Revised: Even if the learning rate is set to a lower value, the network does not outperform the normalized one.

Original: Based on the training behaviors,we can conclude that the learning  for clipping threshold without WN in QNNs need a careful choice.
Revised: Based on the training behaviors, the learning rate for clipping threshold without WN in QNNs need a careful choice.

Original: [Table]
Revised: [Table]

Original: B E XPERIMENTS D ETAILS 
Revised: B E XPERIMENTS D ETAILS B. I MPLEMENTATIONS D ETAILS

Original: [Table] B. I MPLEMENTATIONS D ETAILS

Original: The ImageNet dataset consists of 1.2M training and 50K validation images.
Revised: The ImageNet dataset consists of 1.2M training and 50K validation images.

Original: We use a standard data preprocess in the original paper (He et al., 2016).
Revised: We use a standard data preprocess in the original paper (He et al., 2016).

Original: For training images, they are randomly cropped and resized to 224 × 224 .
Revised: For training images, they are randomly cropped and resized to 224 × 224 .

Original: Validation images are center-cropped to the same size.
Revised: Validation images are center-cropped to the same size.

Original: We use the Pytorch ofﬁcial code 5 to construct ResNet-18 and ResNet-34, and they are initialized from the released pretrained model.
Revised: We use the Pytorch ofﬁcial code 3 to construct ResNet-18 and ResNet-34, and they are initialized from the released pretrained model.

Original: We use stochastic gradient descent (SGD) with  momentum of 0.9 to optimize both weight parameters and the clipping threshold simultaneously.
Revised: We use stochastic gradient descent (SGD) with the momentum of 0.9 to optimize both weight parameters and the clipping threshold simultaneously.

Original: Batch size is set to 1024 and  learning rate starts from 0.1 with a decay factor of 0.1 at epoch 30,60,90.
Revised: Batch size is set to 1024 and the learning rate starts from 0.1 with a decay factor of 0.1 at epoch 30,60,90.

Original: The network is trained up toepochs and weight decay is set to 5 × 10 − 5 for ResNet-18 and 10 − 4 for ResNet-34.
Revised: The network is trained up to 120 epochs and weight decay is set to 5 × 10 − 5 for ResNet-18 and 10 − 4 for ResNet-34.

Original: We do not quantize the ﬁrst layer and the last layer as other methods did (Jung et al., 2019; Zhang et al., 2018;

Original: Choi et al., 2018b).

Original: The CIFAR10 dataset contains 50K training and 10K test images with 32 × 32 pixel.
Revised: The CIFAR10 dataset contains 50K training and 10K test images with 32 × 32 pixels.

Original: The ResNet architectures for CIFAR

Revised: The ResNet architectures for CIFAR

Original: (He et al., 2016) contains a convlutional layer followed by 3 residual blocks and a ﬁnal FC layer.
Revised: (He et al., 2016) contains a convolutional layer followed by 3 residual blocks and a ﬁnal FC layer.

Original: We train full precision ResNet-20 and ResNet-56 ﬁrstly and use them as initialization for quantized models.
Revised: We train full precision ResNet-20 and ResNet-56 ﬁrstly and use them as initialization for quantized models.

Original: All networks were trained for 200 epochs with a mini-batch size of 128.
Revised: All networks were trained for 200 epochs with a mini-batch size of 128.

Original: SGD with momentum of 0.9 was adopted to optimize the parameters.
Revised: SGD with momentum of 0.9 was adopted to optimize the parameters.

Original: Learning rate started from 0.1 and was scaled by 0.1 at epoch 80,120.
Revised: Learning rate started at 0.1 and was scaled by 0.1 at epoch 80,120.

Original: Weight decay was set to 10 − 4 and we do not quantize the ﬁrst and last layers.
Revised: Weight decay was set to 10 − 4 .

Original: For clipping threshold α , we set 8.0 for activations and 3.0 for weights as initial value when training a 5-bit quantized model.
Revised: For clipping threshold α , we set 8.0 for activations and 3.0 for weights as initial value when training a 5-bit quantized model.

Original: The learning rate of α is set to 0.01 and 0.03 for weights and activations, respectively.
Revised: The learning rate of α is set to 0.01 and 0.03 for weights and activations, respectively.

Original: During practice, we found that the learning rate of α merely do not inﬂuencethe network performance.
Revised: During practice, we found that the learning rate of α merely does not inﬂuence network performance.

Original: Different from PACT (Choi et al., 2018b), the update of α in our works already consider the projection error, so we do not require a relatively large L2-regularization for it.
Revised: Different from PACT (Choi et al., 2018b), the update of α in our works already consider the projection error, so we do not require a relatively large L2-regularization for it.

Original: In practice, the network works ﬁne when the weight decay for α is set to 1e − 5 .
Revised: In practice, the network works ﬁne when the weight decay for α is set to 10 − 5 .

Original: We report the top-1 and top-5 accuracy in Table 4 and the test error (and training error) of our quantized model in Figure 6.

Original: Our method achieves state-of-the-art accuracy.

Original: Though Choi et al.

Original: (2018a) only has 0.1% top-1 accuracy difference in 2-bit quantized ResNet-18, their method uses full precision shortcut to increase the network performance.

Original: Our method is more effective and efﬁcient compared with the above uniform or non-uniform counterparts.

Original: [Figure]

Original: B. AP O T Q UANTIZATION FOR A CTIVATIONS
Revised: B. AP O T Q UANTIZATION FOR A CTIVATIONS

Original: In this section, we apply APoT quantization to activations.
Revised: In this section, we apply APoT quantization to activations.

Original: We also use two conﬁgurations ( k = 2 , n = 1 or n = 2 ).
Revised: We also use two conﬁgurations ( k = 2 , n = 1 or n = 2 ).

Original: Note that activations after the ReLU function is positive, therefore we do not need additional sign bit for activations.
Revised: Note that activations after the ReLU function is positive, therefore we do not need additional sign bit for activations.

Original: For weights quantization, we use uniform quantization.
Revised: For weights quantization, we use uniform quantization.

Original: Other settings are kept the same.
Revised: Other settings are kept the same.

Original: [Table]
Revised: [Table]

Original: Table 5 summarizes the top-1 accuracy comparison of APoT and uniform quantization for activations.
Revised: Table 7 summarizes the top-1 accuracy comparison of APoT and uniform quantization for activations.

Original: For 4-bit quantization, APoT achieves the same accuracy with uniform quantization, while the uniform quantization can surpass APoT in the 2-bit scenario.
Revised: For 4-bit quantization, APoT achieves the same accuracy with uniform quantization, while the uniform quantization can surpass APoT in the 2-bit scenario.

Original: Our proposed APoT quantization do not have signiﬁcant advantages when quantizing activations.
Revised: Our proposed APoT quantization do not have signiﬁcant advantages when quantizing activations.