Original: W AVE Q: G RADIENT -B ASED D EEP Q UANTIZATION OF N EURAL N ETWORKS THROUGH S INUSOIDAL R EGULARIZATION
Revised: W AVE Q: G RADIENT -B ASED D EEP Q UANTIZATION OF N EURAL N ETWORKS THROUGH S INUSOIDAL R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep quantization of neural networks below eight bits can lead to superlinear beneﬁts in storage and compute efﬁciency.
Revised: Deep quantization of neural networks below eight bits can lead to superlinear beneﬁts in storage and compute efﬁciency.

Original: However, homogeneously quantizing all the layers to the same level does not account for the distinction of the layers and their individual properties.
Revised: However, homogeneously quantizing all the layers to the same level does not account for the distinction of the layers and their individual properties.

Original: Heterogeneous assignment of bitwidths to the layers is attractive but opens an exponentially large non-contiguous hyperparameter space (Available Bitwidths # Layers ).
Revised: Heterogeneous assignment of bitwidths to the layers is attractive but opens an exponentially large non-contiguous hyperparameter space (Available Bitwidths # Layers ).

Original: Thus, ﬁnding the bitwidth while also quantizing the network to those levels becomes a major challenge.
Revised: Thus, ﬁnding the bitwidth while also quantizing the network to those levels becomes a major challenge.

Original: This paper addresses this challenge through a sinusoidal regularization mechanism, dubbed WaveQ .
Revised: This paper addresses this challenge through a sinusoidal regularization mechanism, dubbed WaveQ .

Original: Adding our parametrized sinusoidal regularizer enables WaveQ to not only ﬁnd the quantized weights, but also learn the bitwidth of the layers by making the period of the sinusoidal regularizer a trainable parameter.
Revised: Adding our parametrized sinusoidal regularizer enables WaveQ to not only ﬁnd the quantized weights, but also learn the bitwidth of the layers by making the period of the sinusoidal regularizer a trainable parameter.

Original: In addition, the sinusoidal regularizer itself is designed to align its minima on the quantization levels.
Revised: In addition, the sinusoidal regularizer itself is designed to align its minima on the quantization levels.

Original: With these two innovations, during training, stochastic gradient descent uses the form of the sinusoidal regularizer and its minima to push the weights to the quantization levels while it is also learning the period which will determine the bitwidth of each layer separately.
Revised: With these two innovations, during training, stochastic gradient descent uses the form of the sinusoidal regularizer and its minima to push the weights to the quantization levels while it is also learning the period which will determine the bitwidth of each layer separately.

Original: As such WaveQ is a gradient-based mechanism that jointly learns the quantized weights as well as the heterogeneous bitwidths.
Revised: As such WaveQ is a gradient-based mechanism that jointly learns the quantized weights as well as the heterogeneous bitwidths.

Original: We show that WaveQ balances compute efﬁciency and accuracy, and provides a heterogeneous bitwidth assignment for quantization of a large variety of deep networks (AlexNet, CIFAR10, MobileNet, ResNet-18, ResNet-20, SVHN, and VGG-11) that virtually preserves the accuracy.
Revised: We show that WaveQ balances compute efﬁciency and accuracy, and provides a heterogeneous bitwidth assignment for quantization of a large variety of deep networks (AlexNet, CIFAR10, MobileNet, ResNet-18, ResNet-20, SVHN, and VGG-11) that virtually preserves the accuracy.

Original: WaveQ is versatile and can also be used with predetermined bitwidths by ﬁxing the period of the sinusoidal regularizer.
Revised: WaveQ is versatile and can also be used with predetermined bitwidths by ﬁxing the period of the sinusoidal regularizer.

Original: In this case, WaveQ , on average, improves the accuracy of quantized training algorithms (DoReFa and
Revised: In this case, WaveQ , on average, improves the accuracy of quantized training algorithms (DoReFa and

Original: WRPN) by ∼ 4 . 8% , and outperforms multiple state-of-the-art techniques.
Revised: WRPN) by ∼ 4 . 8% , and outperforms multiple state-of-the-art techniques.

Original: Finally,
Revised: Finally,

Original: WaveQ is applicable to quantizing transformers and yields signiﬁcant beneﬁts.
Revised: WaveQ is applicable to quantizing transformers and yields signiﬁcant beneﬁts.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Quantization, in general, and deep quantization (below eight bits) (Krishnamoorthi, 2018), in particular, aims to not only reduce the compute requirements of DNNs but also reduce their memory footprint (Zhou et al., 2016; Judd et al., 2016b; Hubara et al., 2017; Mishra et al., 2018; Sharma et al., 2018).
Revised: Quantization, in general, and deep quantization (below eight bits) (Krishnamoorthi, 2018), in particular, aims to not only reduce the compute requirements of DNNs but also reduce their memory footprint (Zhou et al., 2016; Judd et al., 2016b; Hubara et al., 2017; Mishra et al., 2018; Sharma et al., 2018).

Original: Nevertheless, without specialized training algorithms, quantization can diminish the accuracy.
Revised: Nevertheless, without specialized training algorithms, quantization can diminish the accuracy.

Original: As such, the practical utility of quantization hinges upon addressing two fundamental challenges: (1) discovering the appropriate bitwidth of quantization for each layer while considering the accuracy; and (2) learning weights in the quantized domain for a given set of bitwidths.
Revised: As such, the practical utility of quantization hinges upon addressing two fundamental challenges: (1) discovering the appropriate bitwidth of quantization for each layer while considering the accuracy; and (2) learning weights in the quantized domain for a given set of bitwidths.

Original: This paper formulates both of these challenges as a gradient-based joint optimization problem by introducing an additional novel sinusoidal regularization term in the training loss, called WaveQ .
Revised: This paper formulates both of these challenges as a gradient-based joint optimization problem by introducing an additional novel sinusoidal regularization term in the training loss, called WaveQ .

Original: The following two main insights drive this work.
Revised: The following two main insights drive this work.

Original: (1) Sinusoidal functions ( sin 2 ) have inherent periodic minima and by adjusting the period, the minima can be positioned on quantization levels corresponding to a bitwidth at per-layer granularity.
Revised: (1) Sinusoidal functions ( sin 2 ) have inherent periodic minima and by adjusting the period, the minima can be positioned on quantization levels corresponding to a bitwidth at per-layer granularity.

Original: (2) As such, sinusoidal period becomes a direct and continuous representation of the bitwidth.
Revised: (2) As such, sinusoidal period becomes a direct and continuous representation of the bitwidth.

Original: Therefore, WaveQ incorporates this continuous variable (i.e., period) as a differentiable part of the training loss in the form of a regularizer.
Revised: Therefore, WaveQ incorporates this continuous variable (i.e., period) as a differentiable part of the training loss in the form of a regularizer.

Original: Hence, WaveQ is a differentiable regularization mechanism, it piggy backs on the stochastic gradient descent that trains the neural network to also learn the bitwidth (the period).
Revised: Hence, WaveQ is a differentiable regularization mechanism, it piggy backs on the stochastic gradient descent that trains the neural network to also learn the bitwidth (the period).

Original: Simultaneously this parametric sinusoidal regularizer pushes the weights to the quantization levels ( sin 2 minima).
Revised: Simultaneously this parametric sinusoidal regularizer pushes the weights to the quantization levels ( sin 2 minima).

Original: By adding our parametric sinusoidal regularizer to the original training objective function, our method automatically yields the bitwidths for each layer along with nearly quantized weights for those bitwidths.
Revised: By adding our parametric sinusoidal regularizer to the original training objective function, our method automatically yields the bitwidths for each layer along with nearly quantized weights for those bitwidths.

Original: In fact, the original optimization procedure itself is harnessed for this purpose, which is enabled by the differentiability of the sinusoidal regularization term.
Revised: In fact, the original optimization procedure itself is harnessed for this purpose, which is enabled by the differentiability of the sinusoidal regularization term.

Original: As such, quantized training algorithms (Zhou et al., 2016; Mishra et al., 2018) that still use some form of backpropagation (Rumelhart et al., 1986) can effectively utilize the proposed mechanism by modifying their loss.
Revised: As such, quantized training algorithms (Zhou et al., 2016; Mishra et al., 2018) that still use some form of backpropagation (Rumelhart et al., 1986) can effectively utilize the proposed mechanism by modifying their loss.

Original: Moreover, the proposed technique is ﬂexible as it enables heterogeneous quantization across the layers.
Revised: Moreover, the proposed technique is ﬂexible as it enables heterogeneous quantization across the layers.

Original: The WaveQ regularization can also be applied for training a model from scratch, or for ﬁne-tuning a pretrained model.
Revised: The WaveQ regularization can also be applied for training a model from scratch, or for ﬁne-tuning a pretrained model.

Original: In contrast to the prior inspiring works (Uhlich et al., 2019; Esser et al., 2019), WaveQ is the only technique that casts ﬁnding the bitwidths and the corresponding quantized weights as a simultaneous gradient-based optimization through sinusoidal regularization during the training process.
Revised: In contrast to the prior inspiring works (Uhlich et al., 2019; Esser et al., 2019), WaveQ is the only technique that casts ﬁnding the bitwidths and the corresponding quantized weights as a simultaneous gradient-based optimization through sinusoidal regularization during the training process.

Original: We also prove a theoretical result to provide an insight on why the proposed approach leads to solutions preserving the original accuracy during quantization.
Revised: We also prove a theoretical result to provide an insight on why the proposed approach leads to solutions preserving the original accuracy during quantization.

Original: We evaluate WaveQ using different bitwidth assignments across different DNNs (AlexNet, CIFAR-10, MobileNet, ResNet-18, ResNet20, SVHN, and VGG-11).
Revised: We evaluate WaveQ using different bitwidth assignments across different DNNs (AlexNet, CIFAR-10, MobileNet, ResNet-18, ResNet20, SVHN, and VGG-11).

Original: To show the versatility of WaveQ , it is used with two different quantized training algorithms, DoReFa (Zhou et al., 2016) and WRPN (Mishra et al., 2018).
Revised: To show the versatility of WaveQ , it is used with two different quantized training algorithms, DoReFa (Zhou et al., 2016) and WRPN (Mishra et al., 2018).

Original: Over all the bitwidth assignments, the proposed regularization, on average, improves the top-1 accuracy of
Revised: Over all the bitwidth assignments, the proposed regularization, on average, improves the top-1 accuracy of

Original: DoReFa by 4.8%.
Revised: DoReFa by 4.8%.

Original: The reduction in the bitwidth, on average, leads to 77.5% reduction in the energy consumed during the execution of these networks.
Revised: The reduction in the bitwidth, on average, leads to 77.5% reduction in the energy consumed during the execution of these networks.

Original: Finally, we apply WaveQ to Transformer
Revised: Finally, we apply WaveQ to Transformer

Original: DNNs citeppby augmenting their loss with WaveQ parametric sinusiodal regularization.
Revised: DNNs citeppby augmenting their loss with WaveQ parametric sinusiodal regularization.

Original: In this case, the conventional stochastic gradient descent plus WaveQ regularization is used to quantize the big
Revised: In this case, the conventional stochastic gradient descent plus WaveQ regularization is used to quantize the big

Original: Transformer model from (Ott et al., 2018) for machine translation on the IWSLT14 German-English dataset (IWS).
Revised: Transformer model from (Ott et al., 2018) for machine translation on the IWSLT14 German-English dataset (IWS).

Original: For 5, 6, and 7-bit quantization, training with WaveQ yields 0.46, 0.14, 0.04 improved BiLingual Evaluation Understudy (BLEU) score, respectively.
Revised: For 5, 6, and 7-bit quantization, training with WaveQ yields 0.46, 0.14, 0.04 improved BiLingual Evaluation Understudy (BLEU) score, respectively.

Original: As a point of reference, the original big Transformer model from (Ott et al., 2018) improves the BLEU by only 0.1 over the state-of-the-art.
Revised: As a point of reference, the original big Transformer model from (Ott et al., 2018) improves the BLEU by only 0.1 over the state-of-the-art.

Original: Code available at https://github.com/waveq-reg/waveq
Revised: Code available at https://github.com/waveq-reg/waveq

Original:  J OINT L EARNING OF L AYER B ITWIDTHS AND Q UANTIZED P ARAMETERS
Revised:  J OINT L EARNING OF L AYER B ITWIDTHS AND Q UANTIZED P ARAMETERS

Original: Our proposed method WaveQ exploits weight regularization in order to automatically quantize a neural network while training.
Revised: Our proposed method WaveQ exploits weight regularization in order to automatically quantize a neural network while training.

Original: To that end, Section 2.1 describes the role of regularization in neural networks and then Section 2.2 explains WaveQ in more details.
Revised: To that end, Section 2.1 describes the role of regularization in neural networks and then Section 2.2 explains WaveQ in more details.

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Quantizer.
Revised: Quantizer.

Original: We discuss how quantization of weight works.
Revised: We discuss how quantization of weight works.

Original: Consider a ﬂoating-point variable w f to be mapped into a quantized domain using ( b + 1) bits.
Revised: Consider a ﬂoating-point variable w f to be mapped into a quantized domain using ( b + 1) bits.

Original: Let Q be a set of (2 k + 1) quantized values, where k = 2 b − 1 .
Revised: Let Q be a set of (2 k + 1) quantized values, where k = 2 b − 1 .

Original: Considering linear quantization, Q can be represented as (cid:8) − 1 , − k − 1 k , ..., − 1 k , 0 , 1 k , ..., k − 1 k , 1 (cid:9) , where 1 k is the size of the quantization bin.
Revised: Considering linear quantization, Q can be represented as (cid:8) − 1 , − k − 1 k , ..., − 1 k , 0 , 1 k , ..., k − 1 k , 1 (cid:9) , where 1 k is the size of the quantization bin.

Original: Now, w f can be mapped to the b -bit quantization Zhou et al.
Revised: Now, w f can be mapped to the b -bit quantization Zhou et al.

Original: (2016) space as follows.
Revised: (2016) space as follows.

Original: [Equation]
Revised: [Equation]

Original: In Equation 2.1, quantize b ( x ) = 1 2 b − 1 round((2 b − 1) x ) , w f is a scalar, W f is a vector, and w qo is a scalar in the range [ − 1 , 1] .
Revised: In Equation 2.1, quantize b ( x ) = 1 2 b − 1 round((2 b − 1) x ) , w f is a scalar, W f is a vector, and w qo is a scalar in the range [ − 1 , 1] .

Original: Then, a scaling factor c is determined per layer to map the ﬁnal quantized weight w q into the range [ − c, + c ] .
Revised: Then, a scaling factor c is determined per layer to map the ﬁnal quantized weight w q into the range [ − c, + c ] .

Original: As such, w q takes the form cw qo , where c > 0 , and w qo ∈ Q .
Revised: As such, w q takes the form cw qo , where c > 0 , and w qo ∈ Q .

Original: [Figure]
Revised: [Figure]

Original: Soft constraints through regularization and the loss landscape of neural networks.
Revised: Soft constraints through regularization and the loss landscape of neural networks.

Original: Neural networks’ loss landscapes are known to be highly non-convex and it has been empirically veriﬁed that loss surfaces for large networks have many local minima that essentially provide equivalent test errors Choromanska et al.
Revised: Neural networks’ loss landscapes are known to be highly non-convex and it has been empirically veriﬁed that loss surfaces for large networks have many local minima that essentially provide equivalent test errors Choromanska et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: This opens up the possibility of adding soft constrains as extra custom objectives during the training process, in addition to the original objective (i.e., to minimize the accuracy loss).
Revised: This opens up the possibility of adding soft constrains as extra custom objectives during the training process, in addition to the original objective (i.e., to minimize the accuracy loss).

Original: The added constraint could be with the purpose of increasing generalization or imposing some preference on the weights values
Revised: The added constraint could be with the purpose of increasing generalization or imposing some preference on the weights values

Original: P ena l t y
Revised: P ena l t y

Original: Weight
Revised: Weight

Original: Quant.
Revised: Quant.

Original: Step
Revised: Step

Original: Quantization Levels
Revised: Quantization Levels

Original: P ena l t y
Revised: P ena l t y

Original: Weight W
Revised: Weight W

Original: P ena l t y
Revised: P ena l t y

Original: Bitwidth 𝛽 (a) (e)    6 bits
Revised: Bitwidth 𝛽 (a) (e)    6 bits

Original: R eg .
Revised: R eg .

Original: Lo ss
Revised: Lo ss

Original: Weight W
Revised: Weight W

Original: R egu l a r i z a t i on s t r eng t h
Revised: R egu l a r i z a t i on s t r eng t h

Original: Iterations (b) (d) (c)
Revised: Iterations (b) (d) (c)

Original: B i t w i d t h M i n i m i
Revised: B i t w i d t h M i n i m i

Original: z a t i on
Revised: z a t i on

Original: B i t w i d t h 𝛽 0.
Revised: B i t w i d t h 𝛽 0.

Original: #  3 Phase
Revised: #  3 Phase

Original: Phase Phase
Revised: Phase Phase

Original: Regularization in action.
Revised: Regularization in action.

Original: Regularization effectively constrains weight parameters by adding a corresponding term (regularizer) to the objective loss function.
Revised: Regularization effectively constrains weight parameters by adding a corresponding term (regularizer) to the objective loss function.

Original: A classical example is Weight Decay Krogh & Hertz (1991) that aims to reduce the network complexity by limiting the growth of the weights.
Revised: A classical example is Weight Decay Krogh & Hertz (1991) that aims to reduce the network complexity by limiting the growth of the weights.

Original: This soft constraint is realized by adding a term, proportional to the L 2 norm of the weights to the objective function as the regularizer that penalizes large weight values.
Revised: This soft constraint is realized by adding a term, proportional to the L 2 norm of the weights to the objective function as the regularizer that penalizes large weight values.

Original: WaveQ , on the other hand, uses regularization to push the weights to the quantization levels.
Revised: WaveQ , on the other hand, uses regularization to push the weights to the quantization levels.

Original: For the sake of simplicity and clarity, Figure 1(a) and (b) illustrate a geometrical sketch for a hypothetical loss surface (original objective function to be minimized) and an extra regularization term in 2-D weight space, respectively.
Revised: For the sake of simplicity and clarity, Figure 1(a) and (b) illustrate a geometrical sketch for a hypothetical loss surface (original objective function to be minimized) and an extra regularization term in 2-D weight space, respectively.

Original: For weight decay regularization (Figure 1 (a)), the faded circle shows that as we get closer to the origin, the regularization loss is minimized.
Revised: For weight decay regularization (Figure 1 (a)), the faded circle shows that as we get closer to the origin, the regularization loss is minimized.

Original: The point w opt is the optimum just for the loss function alone and the overall optimum solution is achieved by striking a balance between the original loss term and the regularization loss term.
Revised: The point w opt is the optimum just for the loss function alone and the overall optimum solution is achieved by striking a balance between the original loss term and the regularization loss term.

Original: Similarly, Figure 1(b) shows a representation of the proposed periodic regularization for a ﬁxed bitwidth β .
Revised: Similarly, Figure 1(b) shows a representation of the proposed periodic regularization for a ﬁxed bitwidth β .

Original: A periodic pattern of minima pockets are seen surrounding the original optimum point.
Revised: A periodic pattern of minima pockets are seen surrounding the original optimum point.

Original: The objective of the optimization problem is to ﬁnd the best solution that is the closest to one of those minima pockets where weight values are nearly matching the desired quantization levels, hence the name quantization-friendly.
Revised: The objective of the optimization problem is to ﬁnd the best solution that is the closest to one of those minima pockets where weight values are nearly matching the desired quantization levels, hence the name quantization-friendly.

Original: 2. W AVE Q R EGULARIZATION
Revised: 2. W AVE Q R EGULARIZATION

Original: The proposed regularizer is formulated in Equation (2.2) where the ﬁrst term pushes the weights to the quantization levels and the second correlated term aims to reduce the bitwidth of each individual layer heterogeneously.
Revised: The proposed regularizer is formulated in Equation (2.2) where the ﬁrst term pushes the weights to the quantization levels and the second correlated term aims to reduce the bitwidth of each individual layer heterogeneously.

Original: [Equation]
Revised: [Equation]

Original: In Equation (2.2), λ w is the weights quantization regularization strength which governs how strongly weight quantization errors are penalized, and λ β is the bitwidth regularization strength.
Revised: In Equation (2.2), λ w is the weights quantization regularization strength which governs how strongly weight quantization errors are penalized, and λ β is the bitwidth regularization strength.

Original: The parameter β i is proportional to the quantization bitwidth which is elaborated later in this section.
Revised: The parameter β i is proportional to the quantization bitwidth which is elaborated later in this section.

Original: Figure(a) shows a 3-D visualization of our regularizer, R .
Revised: Figure(a) shows a 3-D visualization of our regularizer, R .

Original: Figure 2 (b), (c) show a 2-D proﬁle w.r.t weights ( w ), while (d) shows a 2-D proﬁle w.r.t the bitwidth ( β ) .
Revised: Figure 2 (b), (c) show a 2-D proﬁle w.r.t weights ( w ), while (d) shows a 2-D proﬁle w.r.t the bitwidth ( β ) .

Original: Periodic sinusoidal regularization.
Revised: Periodic sinusoidal regularization.

Original: As shown in Equation equation 2.2, the ﬁrst regularization term is based on a periodic function (sinusoidal) that adds a smooth and differentiable term to the original objective, Figure 2(b).
Revised: As shown in Equation equation 2.2, the ﬁrst regularization term is based on a periodic function (sinusoidal) that adds a smooth and differentiable term to the original objective, Figure 2(b).

Original: The periodic regularizer induces a periodic pattern of minima that correspond to the desired quantization levels.
Revised: The periodic regularizer induces a periodic pattern of minima that correspond to the desired quantization levels.

Original: Such correspondence is achieved by matching the period to the quantization step ( 1 / (2 β i − 1) ) based on a particular number of bits ( β i ) for a given layer i .
Revised: Such correspondence is achieved by matching the period to the quantization step ( 1 / (2 β i − 1) ) based on a particular number of bits ( β i ) for a given layer i .

Original: Learning the sinusoidal period.
Revised: Learning the sinusoidal period.

Original: The parameter β i in (Equation 2.2) controls the period of the sinusoidal regularizer.
Revised: The parameter β i in (Equation 2.2) controls the period of the sinusoidal regularizer.

Original: Thereby β
Revised: Thereby β

Original: i is directly proportional to the actual quantization bitwidth ( b i ) of layer i as follows:
Revised: i is directly proportional to the actual quantization bitwidth ( b i ) of layer i as follows:

Original: [Equation]
Revised: [Equation]

Original: In Equation (B.2) where α i ∈ R + is a scaling factor.
Revised: In Equation (B.2) where α i ∈ R + is a scaling factor.

Original: Note that b i ∈
Revised: Note that b i ∈

Original: Z is the only discrete parameter, while β i ∈ R
Revised: Z is the only discrete parameter, while β i ∈ R

Original: + is a continuous real valued variable, and (cid:100)∗(cid:101) .
Revised: + is a continuous real valued variable, and (cid:100)∗(cid:101) .

Original: is the ceiling operator.
Revised: is the ceiling operator.

Original: While the ﬁrst term in Equation equation 2.2 is only responsible for promoting quantized weights, the second term ( λ β (cid:80) i β i ) aims to reduce the number of bits for each layer i individually while the overall loss is aiming to maximize accuracy.
Revised: While the ﬁrst term in Equation equation 2.2 is only responsible for promoting quantized weights, the second term ( λ β (cid:80) i β i ) aims to reduce the number of bits for each layer i individually while the overall loss is aiming to maximize accuracy.

Original: As such, this term is a soft constraint that yields heterogeneous bitwidths for different layers.
Revised: As such, this term is a soft constraint that yields heterogeneous bitwidths for different layers.

Original: The main insight here is that β i , which also controls the period of the sinusoidal term, is a continuous valued parameter by deﬁnition.
Revised: The main insight here is that β i , which also controls the period of the sinusoidal term, is a continuous valued parameter by deﬁnition.

Original: As such, β i acts as an ideal optimization objective constraint and a proxy to minimize the actual quantization bitwidth b i .
Revised: As such, β i acts as an ideal optimization objective constraint and a proxy to minimize the actual quantization bitwidth b i .

Original: Therefore, WaveQ avoids the issues of gradient-based optimization for discrete valued parameters.
Revised: Therefore, WaveQ avoids the issues of gradient-based optimization for discrete valued parameters.

Original: Furthermore, the beneﬁt of learning the sinusoidal period is two-fold.
Revised: Furthermore, the beneﬁt of learning the sinusoidal period is two-fold.

Original: First, it provides a smooth differentiable objective for ﬁnding minimal bitwidths.
Revised: First, it provides a smooth differentiable objective for ﬁnding minimal bitwidths.

Original: Second, simultaneously learning the scaling factor ( α i ) associated with the found bitwidth.
Revised: Second, simultaneously learning the scaling factor ( α i ) associated with the found bitwidth.

Original: Leveraging the sinusoidal properties, WaveQ learns the following two quantization parameters simultaneously: (1) a per-layer quantization bitwidth ( b i ) along with (2) a scaling factor ( α i ) through learning the period of the sinusoidal function.
Revised: Leveraging the sinusoidal properties, WaveQ learns the following two quantization parameters simultaneously: (1) a per-layer quantization bitwidth ( b i ) along with (2) a scaling factor ( α i ) through learning the period of the sinusoidal function.

Original: Additionally, by exploiting the periodicity, differentiability, and the local convexity proﬁle in sinusoidal functions,
Revised: Additionally, by exploiting the periodicity, differentiability, and the local convexity proﬁle in sinusoidal functions,

Original: WaveQ automatically propels network weights towards values that are inherently closer to quantization levels according to the jointly learned quantizer’s parameters
Revised: WaveQ automatically propels network weights towards values that are inherently closer to quantization levels according to the jointly learned quantizer’s parameters

Original: b i , α i as follows.
Revised: b i , α i as follows.

Original: F i r s t
Revised: F i r s t

Original: D e r
Revised: D e r

Original: i v a t
Revised: i v a t

Original: i v e
Revised: i v e

Original: S e c ond
Revised: S e c ond

Original: D e r
Revised: D e r

Original: i v a t
Revised: i v a t

Original: i v e
Revised: i v e

Original: Bitwidth 𝜷 Bitwidth 𝜷
Revised: Bitwidth 𝜷 Bitwidth 𝜷

Original: Bitwidth 𝜷
Revised: Bitwidth 𝜷

Original: Bitwidth 𝜷 Bitwidth 𝜷 Bitwidth 𝜷 (a) (b) (c)
Revised: Bitwidth 𝜷 Bitwidth 𝜷 Bitwidth 𝜷 (a) (b) (c)

Original: z z z 𝝏 𝒃 𝑹 𝟎 𝒘;𝜷 𝐔𝐧𝐛𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃
Revised: z z z 𝝏 𝒃 𝑹 𝟎 𝒘;𝜷 𝐔𝐧𝐛𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃

Original: 𝑹 𝟏 𝒘;𝜷 𝐁𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃
Revised: 𝑹 𝟏 𝒘;𝜷 𝐁𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃

Original: 𝑹 𝟐 𝒘;𝜷 𝐁𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃𝟐 𝑹 𝟎 𝒘;𝜷 𝐔𝐧𝐛𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃𝟐 𝑹 𝟏 𝒘;𝜷 𝐔𝐧𝐛𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃𝟐 𝑹 𝟐 𝒘;𝜷 𝐁𝐨𝐮𝐧𝐝𝐞𝐝
Revised: 𝑹 𝟐 𝒘;𝜷 𝐁𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃𝟐 𝑹 𝟎 𝒘;𝜷 𝐔𝐧𝐛𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃𝟐 𝑹 𝟏 𝒘;𝜷 𝐔𝐧𝐛𝐨𝐮𝐧𝐝𝐞𝐝 𝝏 𝒃𝟐 𝑹 𝟐 𝒘;𝜷 𝐁𝐨𝐮𝐧𝐝𝐞𝐝

Original: B i t w i d t h 𝜷
Revised: B i t w i d t h 𝜷

Original: B i t w i d t h 𝜷
Revised: B i t w i d t h 𝜷

Original: B i t w i d t h 𝜷
Revised: B i t w i d t h 𝜷

Original: 𝑹 𝟎 (𝒘;𝜷) 𝑹 𝟏 (𝒘;𝜷)
Revised: 𝑹 𝟎 (𝒘;𝜷) 𝑹 𝟏 (𝒘;𝜷)

Original: 𝑹 𝟐 (𝒘;𝜷)
Revised: 𝑹 𝟐 (𝒘;𝜷)

Original: These learned parameters ( b, α ), as explained above, can be mapped to the quantizer parameters explained in Equation equation 2.1.
Revised: These learned parameters ( b, α ), as explained above, can be mapped to the quantizer parameters explained in Equation equation 2.1.

Original: For ( b +1) bits quantization (the extra bit is the sign bit):
Revised: For ( b +1) bits quantization (the extra bit is the sign bit):

Original: [Equation]
Revised: [Equation]

Original: Bounding the gradients.
Revised: Bounding the gradients.

Original: The denominator in the ﬁrst term of Equation equation 2.
Revised: The denominator in the ﬁrst term of Equation equation 2.

Original: [Equation]
Revised: [Equation]

Original: ) is used to control the range derivatives of the proposed regularization term with respect to β .
Revised: ) is used to control the range derivatives of the proposed regularization term with respect to β .

Original: This denominator is chosen to limit vanishing and exploding gradients during training.
Revised: This denominator is chosen to limit vanishing and exploding gradients during training.

Original: To this end, we compared three variants of equation equation 2.
Revised: To this end, we compared three variants of equation equation 2.

Original: with different normalization deﬁned, for k = 0 , 1 , and 2 , as:
Revised: with different normalization deﬁned, for k = 0 , 1 , and 2 , as:

Original: [Equation]
Revised: [Equation]

Original: Figure 3 (a), (b), (c) provide a visualization on how each of the proposed scaled variants affect the ﬁrst and second derivatives.
Revised: Figure 3 (a), (b), (c) provide a visualization on how each of the proposed scaled variants affect the ﬁrst and second derivatives.

Original: For R k =0 and R k =2 , there are regions of vanishing or exploding gradients.
Revised: For R k =0 and R k =2 , there are regions of vanishing or exploding gradients.

Original: Only the regularization R k =

Revised: Only the regularization R k =

Original: (the proposed one) is free of such issues.
Revised: (the proposed one) is free of such issues.

Original: Setting the regularization strengths.
Revised: Setting the regularization strengths.

Original: The convergence behavior depends on the setting of the regularization strengths λ w and λ β .
Revised: The convergence behavior depends on the setting of the regularization strengths λ w and λ β .

Original: Our proposed objective seeks to learn multiple quantization parameterization in conjunction.
Revised: Our proposed objective seeks to learn multiple quantization parameterization in conjunction.

Original: As such, the learning process can be portrayed as three phases (Figure 2(e)).
Revised: As such, the learning process can be portrayed as three phases (Figure 2(e)).

Original: In Phase ( 1 ), the learning process optimizes for the original task loss E 0 .
Revised: In Phase ( 1 ), the learning process optimizes for the original task loss E 0 .

Original: Initially, the small λ w and λ β values allow the gradient descent to explore the optimization surface freely.
Revised: Initially, the small λ w and λ β values allow the gradient descent to explore the optimization surface freely.

Original: As the training process moves forward, we transition to phase ( 2 ) where the larger λ w and λ β gradually engage both the weights quantization regularization and the bitwidth regularization, respectively.
Revised: As the training process moves forward, we transition to phase ( 2 ) where the larger λ w and λ β gradually engage both the weights quantization regularization and the bitwidth regularization, respectively.

Original: Note that, for this to work, the strength of the weights quantization regularization λ w should be higher than the strength of the bitwidth regularization λ β such that a bitwidth per layer could be properly evaluated and eventually learned during this phase.
Revised: Note that, for this to work, the strength of the weights quantization regularization λ w should be higher than the strength of the bitwidth regularization λ β such that a bitwidth per layer could be properly evaluated and eventually learned during this phase.

Original: After the bitwidth regularizer converges to a bitwidth for each layer, we transition to phase ( 3 ), where we ﬁx the learned bitwidths and gradually decay λ β while we keep λ w high.
Revised: After the bitwidth regularizer converges to a bitwidth for each layer, we transition to phase ( 3 ), where we ﬁx the learned bitwidths and gradually decay λ β while we keep λ w high.

Original: The criterion for choosing λ w and λ β is to balance the magnitude of regularization loss to be smaller than the magnitude of accuracy loss.
Revised: The criterion for choosing λ w and λ β is to balance the magnitude of regularization loss to be smaller than the magnitude of accuracy loss.

Original: The mathematical formula used to generate λ w and λ β proﬁles can be found in the supplementary material.
Revised: The mathematical formula used to generate λ w and λ β proﬁles can be found in the supplementary material.

Original: (Figure 8).
Revised: (Figure 8).

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: The results of this section are motivated as follows.
Revised: The results of this section are motivated as follows.

Original: Intuitively, we would like to show that the global minima of E
Revised: Intuitively, we would like to show that the global minima of E

Original: = E 

Revised: = E

Original: + R are very close to the minima of E 0 that minimizes R .
Revised: + R are very close to the minima of E 0 that minimizes R .

Original: In other words, we expect to extract among the original solutions, the ones that are most prone to be quantized.
Revised: In other words, we expect to extract among the original solutions, the ones that are most prone to be quantized.

Original: To establish such result, we will not consider the minima of E
Revised: To establish such result, we will not consider the minima of E

Original: = E 0 + R , but the sequence S n of minima of E n = E 

Revised: = E 0 + R , but the sequence S n of minima of E n = E

Original: + δ n R deﬁned for any sequence δ n of real positive numbers.
Revised: + δ n R deﬁned for any sequence δ n of real positive numbers.

Original: The next theorem shows that our intuition holds true, at least asymptotically with n provided δ n → 0 .
Revised: The next theorem shows that our intuition holds true, at least asymptotically with n provided δ n → 0 .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let E 0 , R : R n → [0 , ∞ ) be continuous and assume that the set S E 0 of the global minima of E 0 is non-empty and compact.
Revised: Let E 0 , R : R n → [0 , ∞ ) be continuous and assume that the set S E 0 of the global minima of E 0 is non-empty and compact.

Original: As S E 0 is compact, we can also deﬁne S E 0 ,R ⊆
Revised: As S E 0 is compact, we can also deﬁne S E 0 ,R ⊆

Original: S E 0 as the set of minima of E 0 which minimizes R .
Revised: S E 0 as the set of minima of E 0 which minimizes R .

Original: Let δ n be a sequence of real positive numbers, deﬁne
Revised: Let δ n be a sequence of real positive numbers, deﬁne

Original: E n = E 0 + δ n R and the sequence S n = S E n of the global minima of E n .
Revised: E n = E 0 + δ n R and the sequence S n = S E n of the global minima of E n .

Original: Then, the following holds true
Revised: Then, the following holds true

Original: If δ n → 0 and S n → S ∗ , then S ∗ ⊆
Revised: If δ n → 0 and S n → S ∗ , then S ∗ ⊆

Original: S E 0 ,R , 2.
Revised: S E 0 ,R , 2.

Original: If δ n → 0 then there is a subsequence δ n k → 0 and a non-empty set S ∗ ⊆
Revised: If δ n → 0 then there is a subsequence δ n k → 0 and a non-empty set S ∗ ⊆

Original: S E 0 ,R so that S n k
Revised: S E 0 ,R so that S n k

Original: → S ∗ ,
Revised: → S ∗ ,

Original: where the convergence of sets, denoted by S n → S ∗ , is deﬁned as the convergence to 0 of their
Revised: where the convergence of sets, denoted by S n → S ∗ , is deﬁned as the convergence to 0 of their

Original: Haussdorff distance, i.e., lim n →∞ d H ( S n , S ∗ ) = 0 .
Revised: Haussdorff distance, i.e., lim n →∞ d H ( S n , S ∗ ) = 0 .

Original: Proof.
Revised: Proof.

Original: For the ﬁrst statement, assume that S n → S ∗ .
Revised: For the ﬁrst statement, assume that S n → S ∗ .

Original: We wish to show that S ∗ ⊆
Revised: We wish to show that S ∗ ⊆

Original: S E 0 ,R .
Revised: S E 0 ,R .

Original: Assume that x n is a sequence of global minima of F + δ n G converging to x ∗ .
Revised: Assume that x n is a sequence of global minima of F + δ n G converging to x ∗ .

Original: It sufﬁces to show that x ∗ ∈ S E 0 ,R .
Revised: It sufﬁces to show that x ∗ ∈ S E 0 ,R .

Original: First let us observe that x ∗ ∈ S E 0 .
Revised: First let us observe that x ∗ ∈ S E 0 .

Original: Indeed, let
Revised: Indeed, let

Original: [Equation]
Revised: [Equation]

Original: and assume that x ∈ S E 0 .
Revised: and assume that x ∈ S E 0 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Thus, since E 0 is continuous and x n → x ∗ we have that E 0 ( x ∗ ) =
Revised: Thus, since E 0 is continuous and x n → x ∗ we have that E 0 ( x ∗ ) =

Original: λ which implies x ∗ ∈ S E 0 .
Revised: λ which implies x ∗ ∈ S E 0 .

Original: Next, deﬁne
Revised: Next, deﬁne

Original: [Equation]
Revised: [Equation]

Original: Let ˆ x ∈ S E 0 ,R so that R (ˆ x ) = µ .
Revised: Let ˆ x ∈ S E 0 ,R so that R (ˆ x ) = µ .

Original: Now observe that, by the minimality of x n we have that
Revised: Now observe that, by the minimality of x n we have that

Original: [Equation]
Revised: [Equation]

Original: Thus, R ( x n ) ≤ µ for all n .
Revised: Thus, R ( x n ) ≤ µ for all n .

Original: Since R is continuous and x n → x ∗ we have that R ( x ∗ ) ≤ µ which implies that R ( x ∗ ) = µ since x ∗ ∈ S E 0 .
Revised: Since R is continuous and x n → x ∗ we have that R ( x ∗ ) ≤ µ which implies that R ( x ∗ ) = µ since x ∗ ∈ S E 0 .

Original: Thus, x ∗ ∈ S E 0 ,R .
Revised: Thus, x ∗ ∈ S E 0 ,R .

Original: The second statement follows from the standard theory of Hausdorff distance on compact metric spaces and the ﬁrst statement.
Revised: The second statement follows from the standard theory of Hausdorff distance on compact metric spaces and the ﬁrst statement.

Original: Theorem 1 implies that by decreasing the strength of R , one recovers the subset of the original solutions that achives the smallest quantization loss.
Revised: Theorem 1 implies that by decreasing the strength of R , one recovers the subset of the original solutions that achives the smallest quantization loss.

Original: In practice, we are not interested in global minima, and we should not decrease much the strength of R .
Revised: In practice, we are not interested in global minima, and we should not decrease much the strength of R .

Original: In our context, Theorem 1 should then be understood as a proof of concept on why the proposed approach leads the expected result.
Revised: In our context, Theorem 1 should then be understood as a proof of concept on why the proposed approach leads the expected result.

Original: Experiments carried out in the next section will support this claim.
Revised: Experiments carried out in the next section will support this claim.

Original: Additionally, note that while the theorem is stated in terms of a limit as the regularization parameter vanishes, the proof in fact gives a corresponding stability result.
Revised: Additionally, note that while the theorem is stated in terms of a limit as the regularization parameter vanishes, the proof in fact gives a corresponding stability result.

Original: Namely, if the regularization parameter is sufﬁciently small relative to the main loss then the minimizers will be “almost” quantized.
Revised: Namely, if the regularization parameter is sufﬁciently small relative to the main loss then the minimizers will be “almost” quantized.

Original: For the interested reader, we provide a more detailed version of the above analysis in the supplementary material.
Revised: For the interested reader, we provide a more detailed version of the above analysis in the supplementary material.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: To demonstrate the effectiveness of WaveQ , we evaluated it on several deep neural networks with different Image Classiﬁcation datasets (CIFAR10, SVHN, and ImageNet), and one Transformerbased network, which is the big Transformer model from(Ott et al., 2018) on the IWSLT14 GermanEnglish dataset (IWS).
Revised: To demonstrate the effectiveness of WaveQ , we evaluated it on several deep neural networks with different Image Classiﬁcation datasets (CIFAR10, SVHN, and ImageNet), and one Transformerbased network, which is the big Transformer model from(Ott et al., 2018) on the IWSLT14 GermanEnglish dataset (IWS).

Original: We provide results for two different types of quantization.
Revised: We provide results for two different types of quantization.

Original: First, we show quantization results for learned heterogeneous bitwidths using WaveQ and we provide different analyses to asses the quality of these learned bitwidth assignments.
Revised: First, we show quantization results for learned heterogeneous bitwidths using WaveQ and we provide different analyses to asses the quality of these learned bitwidth assignments.

Original: Second, we further provide results assuming a preset homogeneous bitwidth assignment as a special setting of WaveQ .
Revised: Second, we further provide results assuming a preset homogeneous bitwidth assignment as a special setting of WaveQ .

Original: This is, in some cases, a practical assumption that might stem from particular hardware requirements or constraints.
Revised: This is, in some cases, a practical assumption that might stem from particular hardware requirements or constraints.

Original: Table 1 provides a summary of the evaluated networks and datasets for both learned heterogeneous bitwidths, and the special case of training preset homogeneous bitwidth assignments.
Revised: Table 1 provides a summary of the evaluated networks and datasets for both learned heterogeneous bitwidths, and the special case of training preset homogeneous bitwidth assignments.

Original: We compare our proposed WaveQ method with PACT (Choi et al., 2018a), LQ-Nets (Zhang et al., 2018),
Revised: We compare our proposed WaveQ method with PACT (Choi et al., 2018a), LQ-Nets (Zhang et al., 2018),

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Experimental setup.
Revised: Experimental setup.

Original: We implemented
Revised: We implemented

Original: WaveQ on top of DoReFa inside Distiller (Zmora et al., 2018), an open source framework for quantization by Intel that implements various quantization techniques.
Revised: WaveQ on top of DoReFa inside Distiller (Zmora et al., 2018), an open source framework for quantization by Intel that implements various quantization techniques.

Original: The reported accuracies for DoReFa and
Revised: The reported accuracies for DoReFa and

Original: WRPN are with the built-in implementations in Distiller, which may not exactly match the accuracies reported in their respective papers.
Revised: WRPN are with the built-in implementations in Distiller, which may not exactly match the accuracies reported in their respective papers.

Original: However, an independent implementation from a major company provides an unbiased foundation for comparison.
Revised: However, an independent implementation from a major company provides an unbiased foundation for comparison.

Original: We quantize all convolution and fully connected layers, except for the ﬁrst and last layers which use 8-bits.
Revised: We quantize all convolution and fully connected layers, except for the ﬁrst and last layers which use 8-bits.

Original: This assumption is commensurate with the previous techniques.
Revised: This assumption is commensurate with the previous techniques.

Original: 4. L EARNED H ETEROGENEOUS B ITWIDTHS
Revised: 4. L EARNED H ETEROGENEOUS B ITWIDTHS

Original: As for quantizing both weights and activations, Table 1 shows that incorporating WaveQ into the quantized training process yields best accuracy results outperforming PACT, LQ-Net, DSQ, and
Revised: As for quantizing both weights and activations, Table 1 shows that incorporating WaveQ into the quantized training process yields best accuracy results outperforming PACT, LQ-Net, DSQ, and

Original: DoReFa with signiﬁcant margins.
Revised: DoReFa with signiﬁcant margins.

Original: Furthermore, it can be seen that the learned heterogeneous bitwidths yield better accuracy as compared to the preset 4-bit homogeneous assignments, with lower, on average, bitwidth (3.85, 3.57, and 3.95 bits for AlexNet, ResNet-18, and MobileNet, respectively).
Revised: Furthermore, it can be seen that the learned heterogeneous bitwidths yield better accuracy as compared to the preset 4-bit homogeneous assignments, with lower, on average, bitwidth (3.85, 3.57, and 3.95 bits for AlexNet, ResNet-18, and MobileNet, respectively).

Original: Figure 4(a),(b) (bottom bar graphs) show the learned heterogeneous weight bitwidths over layers for AlexNet and ResNet-18, respectively.
Revised: Figure 4(a),(b) (bottom bar graphs) show the learned heterogeneous weight bitwidths over layers for AlexNet and ResNet-18, respectively.

Original: As seen, WaveQ parametric regularization yields a spectrum of varying bitwidth assignments to the layers which vary from 2 bits to 8 bits with an irregular pattern.
Revised: As seen, WaveQ parametric regularization yields a spectrum of varying bitwidth assignments to the layers which vary from 2 bits to 8 bits with an irregular pattern.

Original: These results demonstrate that the proposed regularization, WaveQ , automatically distinguishes different layers and their varying importance with respect to accuracy while learning their respective bitwidths.
Revised: These results demonstrate that the proposed regularization, WaveQ , automatically distinguishes different layers and their varying importance with respect to accuracy while learning their respective bitwidths.

Original: To assess the quality of these bitwidths assignments, we conduct a sensitivity analysis on the relatively big networks (see next subsection).
Revised: To assess the quality of these bitwidths assignments, we conduct a sensitivity analysis on the relatively big networks (see next subsection).

Original: Beneﬁts of heterogeneous quantization.
Revised: Beneﬁts of heterogeneous quantization.

Original: Figure 4(a),(b) (top graphs) show various comparisons and sensitivity results for learned heterogeneous bitwidth assignments for bigger networks (AlexNet and ResNet-18).
Revised: Figure 4(a),(b) (top graphs) show various comparisons and sensitivity results for learned heterogeneous bitwidth assignments for bigger networks (AlexNet and ResNet-18).

Original: It is infeasible to enumerate these networks’ respective quantization spaces.
Revised: It is infeasible to enumerate these networks’ respective quantization spaces.

Original: Compared to 4-bit homogeneous quantization, learned heterogeneous assignments achieve better accuracy with lower, on average, bitwidth 3 .
Revised: Compared to 4-bit homogeneous quantization, learned heterogeneous assignments achieve better accuracy with lower, on average, bitwidth 3 .

Original: 85 bits for AlexNet and 3 .
Revised: 85 bits for AlexNet and 3 .

Original: 57 bits for ResNet-18.
Revised: 57 bits for ResNet-18.

Original: This demonstrates that a homogeneous (uniform) assignment of the bits is not always the desired choice to minimize accuracy loss.
Revised: This demonstrates that a homogeneous (uniform) assignment of the bits is not always the desired choice to minimize accuracy loss.

Original: Furthermore, Figure 4 also shows that decrementing the learned bitwidth for any single layer at a time results in 0 .
Revised: Furthermore, Figure 4 also shows that decrementing the learned bitwidth for any single layer at a time results in 0 .

Original: 44% and 0 .
Revised: 44% and 0 .

Original: 24% average reduction in accuracy for AlexNet and ResNet-18, respectively.
Revised: 24% average reduction in accuracy for AlexNet and ResNet-18, respectively.

Original: The dotted blue line with (cid:4) markers shows how differently decrementing the bitwidth of various individual layers affect the accuracy.
Revised: The dotted blue line with (cid:4) markers shows how differently decrementing the bitwidth of various individual layers affect the accuracy.

Original: This trend further demonstrates the effectiveness of learning with WaveQ to ﬁnd the lowest bitwidth that maximizes the accuracy.
Revised: This trend further demonstrates the effectiveness of learning with WaveQ to ﬁnd the lowest bitwidth that maximizes the accuracy.

Original: Energy savings.
Revised: Energy savings.

Original: To demonstrate the energy savings of the solutions found by WaveQ , we evaluate it on Stripes (Judd et al., 2016a), a custom accelerator designed for DNNs, which exploits bit-serial computation to support ﬂexible bitwidths for DNN operations.
Revised: To demonstrate the energy savings of the solutions found by WaveQ , we evaluate it on Stripes (Judd et al., 2016a), a custom accelerator designed for DNNs, which exploits bit-serial computation to support ﬂexible bitwidths for DNN operations.

Original: As shown in Table 1, the reduction in the bitwidth, on average, leads to 77.5% reduction in the energy consumed during the execution of these networks.
Revised: As shown in Table 1, the reduction in the bitwidth, on average, leads to 77.5% reduction in the energy consumed during the execution of these networks.

Original: 4. P RESET H OMOGENOUS B ITWIDTH Q UANTIZATION
Revised: 4. P RESET H OMOGENOUS B ITWIDTH Q UANTIZATION

Original: We also consider a preset homogeneous bitwidth quantization which can also be supported by WaveQ under special settings where we ﬁx β (to a preset bitwidth).
Revised: We also consider a preset homogeneous bitwidth quantization which can also be supported by WaveQ under special settings where we ﬁx β (to a preset bitwidth).

Original: Hence, only the ﬁrst regularization term is engaged for propelling the weights to the quantization levels.
Revised: Hence, only the ﬁrst regularization term is engaged for propelling the weights to the quantization levels.

Original: Table 2 shows accuracies of different networks (SimpleNet-5, ResNet-20, VGG-11, and
Revised: Table 2 shows accuracies of different networks (SimpleNet-5, ResNet-20, VGG-11, and

Original: SVHN-8) using plain WRPN, plain DoReFa and DoReFa + WaveQ for 3, 4, and 5 bitwdiths.
Revised: SVHN-8) using plain WRPN, plain DoReFa and DoReFa + WaveQ for 3, 4, and 5 bitwdiths.

Original: As depicted, the results concretely show the effect of incorporating WaveQ into existing quantized training techniques and how it outperforms the previously reported accuracies.
Revised: As depicted, the results concretely show the effect of incorporating WaveQ into existing quantized training techniques and how it outperforms the previously reported accuracies.

Original: Weight distributions during training.
Revised: Weight distributions during training.

Original: Fig- ure 5 shows the evolution of weights distributions over ﬁne-tuning epochs for different layers of CIFAR10, SVHN, AlexNet, and ResNet18 networks.
Revised: Fig- ure 5 shows the evolution of weights distributions over ﬁne-tuning epochs for different layers of CIFAR10, SVHN, AlexNet, and ResNet18 networks.

Original: The high-precision weights form clusters and gradually converge around the quantization centroids as regularization loss is minimized along with the main accuracy loss.
Revised: The high-precision weights form clusters and gradually converge around the quantization centroids as regularization loss is minimized along with the main accuracy loss.

Original: 4. W AVE Q FOR T RANSFORMER Q UANTIZATION
Revised: 4. W AVE Q FOR T RANSFORMER Q UANTIZATION

Original: [Table]
Revised: [Table]

Original: Transformers (encoder-decoder architectures) have been shown to achieve best results for NLP tasks including machine translation (Vaswani et al., 2017) and automatic speech recognition (Mohamed et al., 2019).
Revised: Transformers (encoder-decoder architectures) have been shown to achieve best results for NLP tasks including machine translation (Vaswani et al., 2017) and automatic speech recognition (Mohamed et al., 2019).

Original: A Transformer layer relies on a key-value self-attention mechanism for learning relationships between distant concepts, rather than relying on recurrent connections and memory cells.
Revised: A Transformer layer relies on a key-value self-attention mechanism for learning relationships between distant concepts, rather than relying on recurrent connections and memory cells.

Original: Herein, we extend the application of WaveQ regularization to improve the accuracy of deeply quantized (below 8 bits) Transformer models.
Revised: Herein, we extend the application of WaveQ regularization to improve the accuracy of deeply quantized (below 8 bits) Transformer models.

Original: We run our experiments on IWSLT 2014 GermanEnglish (DE-EN) dataset.
Revised: We run our experiments on IWSLT 2014 GermanEnglish (DE-EN) dataset.

Original: We use the Transformer model implemented in the fairseq-py toolkit (Ott et al., 2019).
Revised: We use the Transformer model implemented in the fairseq-py toolkit (Ott et al., 2019).

Original: All experiments are based on the big transformer model with 6 blocks in the encoder and decoder networks.
Revised: All experiments are based on the big transformer model with 6 blocks in the encoder and decoder networks.

Original: Table 3 shows the effect of applying WaveQ regularization into the training process for 5, 6, and 7-bit quantization on the ﬁnal accuracy (BLEU score).
Revised: Table 3 shows the effect of applying WaveQ regularization into the training process for 5, 6, and 7-bit quantization on the ﬁnal accuracy (BLEU score).

Original: WaveQ consistently improves the BLEU score of quantized models at various quantization bitwidths (7bits).
Revised: WaveQ consistently improves the BLEU score of quantized models at various quantization bitwidths (7bits).

Original: Moreover, higher improvements are obtained at lower bitwidths.
Revised: Moreover, higher improvements are obtained at lower bitwidths.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We conduct an experiment that uses WaveQ for training from scratch.
Revised: We conduct an experiment that uses WaveQ for training from scratch.

Original: For the sake of clarity, we are considering in this experiment the case of preset bitwidth assignments (i.e., λ β = 0 ).
Revised: For the sake of clarity, we are considering in this experiment the case of preset bitwidth assignments (i.e., λ β = 0 ).

Original: WaveQ is used with a constant λ w .
Revised: WaveQ is used with a constant λ w .

Original: As the Figure illustrates, using a constant λ w results in the weights being stuck in a region close to their initialization, (i.e., quantization objective dominates the accuracy objective).
Revised: As the Figure illustrates, using a constant λ w results in the weights being stuck in a region close to their initialization, (i.e., quantization objective dominates the accuracy objective).

Original: However, if we dynamically change the λ w following the exponential curve in Figure 6Row(III)-Col(I)) during the from-scratch training, the weights no longer get stuck.
Revised: However, if we dynamically change the λ w following the exponential curve in Figure 6Row(III)-Col(I)) during the from-scratch training, the weights no longer get stuck.

Original: Instead, the weights traverse the space ( i.e., jump from wave to wave ) as illustrated in Figure 6-Cols(II) and (III) for CIFAR and SVHN, respectively.
Revised: Instead, the weights traverse the space ( i.e., jump from wave to wave ) as illustrated in Figure 6-Cols(II) and (III) for CIFAR and SVHN, respectively.

Original: In these two columns, Rows (I), (II), (III), correspond to quantization with 3, 4, 5 bits, respectively.
Revised: In these two columns, Rows (I), (II), (III), correspond to quantization with 3, 4, 5 bits, respectively.

Original: citepInitially, the smaller λ w values allow the gradient descent to explore the optimization surface freely, as the training process moves forward, the larger λ w gradually engages the sinusoidal regularizer, and eventually pushes the weights close to the quantization levels.
Revised: citepInitially, the smaller λ w values allow the gradient descent to explore the optimization surface freely, as the training process moves forward, the larger λ w gradually engages the sinusoidal regularizer, and eventually pushes the weights close to the quantization levels.

Original: Further convergence analysis is provided in the supplementary material.
Revised: Further convergence analysis is provided in the supplementary material.

Original: Next, we provide results comparing two proﬁles of the regularization strength ( λ w ).
Revised: Next, we provide results comparing two proﬁles of the regularization strength ( λ w ).

Original: Proﬁle 1 : λ w gradually increases as training proceeds then gradually decays towards the end of training.
Revised: Proﬁle 1 : λ w gradually increases as training proceeds then gradually decays towards the end of training.

Original: (Figure 7(a)).
Revised: (Figure 7(a)).

Original: Proﬁle 2 : λ w gradually increases as training proceeds and remains high (Figure 7(c)).
Revised: Proﬁle 2 : λ w gradually increases as training proceeds and remains high (Figure 7(c)).

Original: Figure 7(a,c) depicts different loss components and Figure 7(b,d) visualizes weights trajectories.
Revised: Figure 7(a,c) depicts different loss components and Figure 7(b,d) visualizes weights trajectories.

Original: Both proﬁles show that accuracy loss is unimpededly minimized along with WaveQ loss.
Revised: Both proﬁles show that accuracy loss is unimpededly minimized along with WaveQ loss.

Original: Our theoretical results align with
Revised: Our theoretical results align with

Original: Proﬁle 1 (Figure 7(b))–what reviewer insightfully pointed out.
Revised: Proﬁle 1 (Figure 7(b))–what reviewer insightfully pointed out.

Original: Although λ w decays back towards end of training, the weights mostly remain tied to their quantization levels except for a few deﬂections that cause slight increase of the regularization loss towards end of training.
Revised: Although λ w decays back towards end of training, the weights mostly remain tied to their quantization levels except for a few deﬂections that cause slight increase of the regularization loss towards end of training.

Original: In terms of test accuracy, both proﬁles yield similar results ( Proﬁle 1 , 74.95%) vs ( Proﬁle 2 , 74.45%).
Revised: In terms of test accuracy, both proﬁles yield similar results ( Proﬁle 1 , 74.95%) vs ( Proﬁle 2 , 74.45%).

Original: Note that while the theorem is stated in terms of a limit as the regularization parameter vanishes, the proof in fact gives a corresponding stability result.
Revised: Note that while the theorem is stated in terms of a limit as the regularization parameter vanishes, the proof in fact gives a corresponding stability result.

Original: Namely, if the regularization parameter is sufﬁciently small relative to the main loss then the minimizers will be “almost” quantized.
Revised: Namely, if the regularization parameter is sufﬁciently small relative to the main loss then the minimizers will be “almost” quantized.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: This research lies at the intersection of (1) quantized training algorithms and (2) techniques that discover bitwidth for quantization.
Revised: This research lies at the intersection of (1) quantized training algorithms and (2) techniques that discover bitwidth for quantization.

Original: The following diuscusses the most related works in both directions.
Revised: The following discusses the most related works in both directions.

Original:  In contrast, WaveQ modiﬁes the loss function of the  training  to simultaneously learn the period of an adaptive sinusoidal regularizer through the same stochastic gradient descent that trains the network.
Revised: Most related methods deﬁne a new optimization problem and use a special method for solving it. For example, (Bai et al., 2019) uses a proximal gradient method (adds a prox step after each stochas- tic gradient step), (Yang et al., 2020) uses ADMM, and (Tung & Mori, 2018) takes a Bayesian approach. This only makes the training more difﬁcult, slower and increases the computational complexity. In contrast, WaveQ exploits the conventional stochastic gradient descent method while jointly optimizing for the original training loss while softly constraining it to simultaneously learn the quantized parameters, and more importantly bitwidths.

Original: The differentiability of the adaptive sinusoidal regularizer enables simultaneously learning both the bitwidths and pushing the weight values to the quantization levels.
Revised: The differentiability of the adaptive sinusoidal regularizer enables simultaneously learning both the bitwidths and pushing the weight values to the quantization levels.

Original: As such, WaveQ can be used as a complementary method to some of these efforts, which is demonstrated by experiments with both DoReFa-Net 
Revised: As such, WaveQ can be used as a complementary method to some of these efforts, which is demonstrated by experiments with both DoReFa-Net (Zhou et al., 2016) and

Original: (Zhou et al., 2016) and WRPN (Mishra et al., 2018).
Revised: WRPN (Mishra et al., 2018).

Original: Our preliminary efforts (Anonymous) and another concurrent work (Naumov et al., 2018) use a sinusoidal regularization to push the weights closer to the quantization levels.
Revised: Our preliminary efforts (Anonymous) and another concurrent work (Naumov et al., 2018) use a sinusoidal regularization to push the weights closer to the quantization levels.

Original: However, neither of these two works make the period a differentiable parameter nor ﬁnd bitwidths during training.
Revised: However, neither of these two works make the period a differentiable parameter nor ﬁnd bitwidths during training.

Original: Quantized training algorithms There have been several techniques (Zhou et al., 2016; Zhu et al., 2017; Mishra et al., 2018) that train a neural network in a quantized domain after the bitwidth of the layers is determined manually.
Revised: Quantized training algorithms There have been several techniques (Zhou et al., 2016; Zhu et al., 2017; Mishra et al., 2018) that train a neural network in a quantized domain after the bitwidth of the layers is determined manually.

Original: DoReFa-Net
Revised: DoReFa-Net

Original: (Zhou et al., 2016) uses straight through estimator (Bengio et al., 2013) for quantization and extends it for any arbitrary k bit quantization in weights, activations, and gradients.
Revised: (Zhou et al., 2016) uses straight through estimator (Bengio et al., 2013) for quantization and extends it for any arbitrary k bit quantization in weights, activations, and gradients.

Original: WRPN (Mishra et al., 2018) is training algorithm that compensates for the reduced precision by increasing the number of ﬁlter maps in a layer (doubling or tripling).
Revised: WRPN (Mishra et al., 2018) is training algorithm that compensates for the reduced precision by increasing the number of ﬁlter maps in a layer (doubling or tripling).

Original: TTQ (Zhu et al., 2017) quantizes the weights to ternary values by using per layer scaling coefﬁcients learnt during training.
Revised: TTQ (Zhu et al., 2017) quantizes the weights to ternary values by using per layer scaling coefﬁcients learnt during training.

Original: These scaling coefﬁcients are used to scale the weights during inference.
Revised: These scaling coefﬁcients are used to scale the weights during inference.

Original: PACT (Choi et al., 2018a) proposes a technique for quantizing activations by introducing an activation clipping parameter α .
Revised: PACT (Choi et al., 2018a) proposes a technique for quantizing activations by introducing an activation clipping parameter α .

Original: This parameter ( α ) is used to represent the clipping level in the activation function and is learned via back-propagation during training.
Revised: This parameter ( α ) is used to represent the clipping level in the activation function and is learned via back-propagation during training.

Original: More recently, VNQ (Achterhold et al., 2018) uses a variational Bayesian approach for quantizing neural network weights during training.
Revised: More recently, VNQ (Achterhold et al., 2018) uses a variational Bayesian approach for quantizing neural network weights during training.

Original: Loss-aware weight quantization.
Revised: Loss-aware weight quantization.

Original: Recent works pursued loss-aware minimization approaches for quantization.
Revised: Recent works pursued loss-aware minimization approaches for quantization.

Original: (Hou et al., 2017) and (Hou & Kwok, 2018) developed approximate solutions using proximal Newton algorithm to minimize the loss function directly under the constraints of low bitwidth weights.
Revised: (Hou et al., 2017) and (Hou & Kwok, 2018) developed approximate solutions using proximal Newton algorithm to minimize the loss function directly under the constraints of low bitwidth weights.

Original: One effort (Choi et al., 2018b) proposed to learn the quantization of DNNs through a regularization term of the mean-squared-quantization error.
Revised: One effort (Choi et al., 2018b) proposed to learn the quantization of DNNs through a regularization term of the mean-squared-quantization error.

Original: LQ-Net (Zhang et al., 2018) proposes to jointly train the network and its quantizers.
Revised: LQ-Net (Zhang et al., 2018) proposes to jointly train the network and its quantizers.

Original: DSQ (Gong et al., 2019) employs a series of tanh functions to gradually approximate the staircase function for low-bit quantization (e.g., sign for 1bit case), and meanwhile keeps the smoothness for easy gradient calculation.
Revised: DSQ (Gong et al., 2019) employs a series of tanh functions to gradually approximate the staircase function for low-bit quantization (e.g., sign for 1bit case), and meanwhile keeps the smoothness for easy gradient calculation.

Original: Although some of these techniques use regularization to guide the process of quantized training, none explores the use of adaptive sinusoidal regularizers for quantization.
Revised: Although some of these techniques use regularization to guide the process of quantized training, none explores the use of adaptive sinusoidal regularizers for quantization.

Revised: Most recently, (Nguyen et al., 2020) suggests using | cos | function as a regularizer.

Original: Moreover, unlike WaveQ , these techniques do not ﬁnd the bitwidth for quantizing the layers.
Revised: Moreover, unlike WaveQ , these techniques do not ﬁnd the bitwidth for quantizing the layers.

Original: Techniques for discovering quantization bitwidths.
Revised: Techniques for discovering quantization bitwidths.

Original: A recent line of research focused on methods which can also ﬁnd the optimal quantization parameters, e.g., the bitwidth, the stepsize, in parallel to the network weights.
Revised: A recent line of research focused on methods which can also ﬁnd the optimal quantization parameters, e.g., the bitwidth, the stepsize, in parallel to the network weights.

Original: Recent work (Ye et al., 2018) based on ADMM (adm) runs a binary search to minimize the total square quantization error in order to decide the quantization levels for the layers.
Revised: Recent work (Ye et al., 2018) based on ADMM (adm) runs a binary search to minimize the total square quantization error in order to decide the quantization levels for the layers.

Original: Most recently, (Uhlich et al., 2019) proposed to indirectly learn quantizer’s parameters via Straight
Revised: Most recently, (Uhlich et al., 2019) proposed to indirectly learn quantizer’s parameters via Straight

Original: Through Estimator (STE) (Bengio et al., 2013) based approach.
Revised: Through Estimator (STE) (Bengio et al., 2013) based approach.

Original: In a similar vein, (Esser et al., 2019) has proposed to learn the quantization mapping for each layer in a deep network by approximating the gradient to the quantizer step size that is sensitive to quantized state transitions.
Revised: In a similar vein, (Esser et al., 2019) has proposed to learn the quantization mapping for each layer in a deep network by approximating the gradient to the quantizer step size that is sensitive to quantized state transitions.

Original: On another side, recent works (Elthakeb et al., 2018; Wang et al., November 21, 2018) proposed a reinforcement learning based approach to ﬁnd an optimal bitwidth assignment policy.
Revised: On another side, recent works (Elthakeb et al., 2018; Wang et al., November 21, 2018) proposed a reinforcement learning based approach to ﬁnd an optimal bitwidth assignment policy.

Original: Quantizing Transformers.
Revised: Quantizing Transformers.

Original: FullyQT
Revised: FullyQT

Original: (Prato et al., 2019) uses a bucketing based uniform quantization proposed by QSGD (
Revised: (Prato et al., 2019) uses a bucketing based uniform quantization proposed by QSGD (

Original: Alistarh et al., 2016)
Revised: Alistarh et al., 2016)

Original: and extends it to Tranformers.
Revised: and extends it to Tranformers.

Original: Q8BERT (Zafrir et al., 2019) quantizes all the GEMM (General Matrix Multiply) operations to 8 bit by adding an additional term for quantization loss during training, which is calculated based on the rounding effect of ﬂoating point values (Shaw et al., 2018).
Revised: Q8BERT (Zafrir et al., 2019) quantizes all the GEMM (General Matrix Multiply) operations to 8 bit by adding an additional term for quantization loss during training, which is calculated based on the rounding effect of ﬂoating point values (Shaw et al., 2018).

Original: WaveQ , however, uses a sinusoidal regularizer to automatically push the weights towards the quantization levels.
Revised: WaveQ , however, uses a sinusoidal regularizer to automatically push the weights towards the quantization levels.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper devised WaveQ that casts the two problems of ﬁnding layer bitwidth and quantized weights as a gradient-based optimization through parametric sinusoidal regularization.
Revised: This paper devised WaveQ that casts the two problems of ﬁnding layer bitwidth and quantized weights as a gradient-based optimization through parametric sinusoidal regularization.

Original: WaveQ provides signiﬁcant improvements over the state-of-the-art and is even applicable to the Transformers.
Revised: WaveQ provides signiﬁcant improvements over the state-of-the-art and is even applicable to the Transformers.

Original: A D ETAILED T HEORETICAL A NALYSIS A. M OTIVATION
Revised: A D ETAILED T HEORETICAL A NALYSIS A. M OTIVATION

Original: The results of this section are motivated by the following question.
Revised: The results of this section are motivated by the following question.

Original: Question A.1.
Revised: Question A.1.

Original: Suppose that a function F : R n → [0 , ∞ ) has many global minima and that Q ⊂ R n is closed.
Revised: Suppose that a function F : R n → [0 , ∞ ) has many global minima and that Q ⊂ R n is closed.

Original: How do we isolate the global minima of F that are closest to Q without actually computing the full set of global minima of F ?
Revised: How do we isolate the global minima of F that are closest to Q without actually computing the full set of global minima of F ?

Original: Intuitively, we would like to show that if (cid:15) > 0 is very small, then the global minima of the function
Revised: Intuitively, we would like to show that if (cid:15) > 0 is very small, then the global minima of the function

Original: [Equation]
Revised: [Equation]

Original: ) are very close to the global minima of F closest to Q .
Revised: ) are very close to the global minima of F closest to Q .

Original: To achieve this we will have to introduce ﬁrst the concept of convergence of sets and then we will show that our intuition is correct by proving that the set of global minima to the above relaxed function converges to a subset of global minima of F closest to Q .
Revised: To achieve this we will have to introduce ﬁrst the concept of convergence of sets and then we will show that our intuition is correct by proving that the set of global minima to the above relaxed function converges to a subset of global minima of F closest to Q .

Original: A. R ELEVANT D EFINITIONS
Revised: A. R ELEVANT D EFINITIONS

Original: Deﬁnition A.2.
Revised: Deﬁnition A.2.

Original: If F : R n → [0 , ∞ ) satisﬁes lim | x |→∞ F ( x ) =
Revised: If F : R n → [0 , ∞ ) satisﬁes lim | x |→∞ F ( x ) =

Original: + ∞ , we will say that F is coercive.
Revised: + ∞ , we will say that F is coercive.

Original: Deﬁnition A.3.
Revised: Deﬁnition A.3.

Original: For a coercive function F : R n → [0 , ∞ ) we let S F
Revised: For a coercive function F : R n → [0 , ∞ ) we let S F

Original: = { x ∈ R n : F ( x ) = min y ∈ R n F ( y ) } be coercive.
Revised: = { x ∈ R n : F ( x ) = min y ∈ R n F ( y ) } be coercive.

Original: Lemma A.4.
Revised: Lemma A.4.

Original: Assume that F : R n → [0 , ∞ ) is continuous and coercive.
Revised: Assume that F : R n → [0 , ∞ ) is continuous and coercive.

Original: Then F has at least one global minimum.
Revised: Then F has at least one global minimum.

Original: That is, S F is non-empty.
Revised: That is, S F is non-empty.

Original: Furthermore, S F is a compact set.
Revised: Furthermore, S F is a compact set.

Original: Deﬁnition A.5.
Revised: Deﬁnition A.5.

Original: Let F, G : R n → [0 , ∞ ) are continuous and assume that F is coercive.
Revised: Let F, G : R n → [0 , ∞ ) are continuous and assume that F is coercive.

Original: Deﬁne
Revised: Deﬁne

Original: S F,G = { x
Revised: S F,G = { x

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: y ) } , the minima of F which minimize G among the minima of F .
Revised: y ) } , the minima of F which minimize G among the minima of F .

Original: Deﬁnition A.6.
Revised: Deﬁnition A.6.

Original: Let Q ⊂ R n be a closed set and assume that x ∈ R n .
Revised: Let Q ⊂ R n be a closed set and assume that x ∈ R n .

Original: Deﬁne the distance from x to the set Q to be
Revised: Deﬁne the distance from x to the set Q to be

Original: [Equation]
Revised: [Equation]

Original: Observe that since Q is a closed set we have that x ∈ Q if and only if d ( x, Q ) = 

Revised: Observe that since Q is a closed set we have that x ∈ Q if and only if d ( x, Q ) =

Original: and otherwise d ( x, Q ) > 0 .
Revised: and otherwise d ( x, Q ) > 0 .

Original: Deﬁnition A.7.
Revised: Deﬁnition A.7.

Original: Let A, B ⊂ R n be compact sets.
Revised: Let A, B ⊂ R n be compact sets.

Original: We deﬁne the Hausdorff distance between A and
Revised: We deﬁne the Hausdorff distance between A and

Original: B by
Revised: B by

Original: [Equation]
Revised: [Equation]

Original: Observe that d H ( A, B ) = 

Revised: Observe that d H ( A, B ) =

Original: if and only if A = B .
Revised: if and only if A = B .

Original: Deﬁnition A.8.
Revised: Deﬁnition A.8.

Original: Let { S δ } δ> 0 be a family of compact subsets of R n .
Revised: Let { S δ } δ> 0 be a family of compact subsets of R n .

Original: We say that lim δ
Revised: We say that lim δ

Original: → 0 S δ = S ∗ if
Revised: → 0 S δ = S ∗ if

Original: [Equation]
Revised: [Equation]

Original: Lemma A.9.
Revised: Lemma A.9.

Original: Let S δ be a family of compact subsets of R n , then lim
Revised: Let S δ be a family of compact subsets of R n , then lim

Original: δ
Revised: δ

Original: → 0 S δ = S ∗ if and only if the following two conditions hold.
Revised: → 0 S δ = S ∗ if and only if the following two conditions hold.

Original: If x δ ∈ S δ converges to x , then x ∈ S ∗ 2.
Revised: If x δ ∈ S δ converges to x , then x ∈ S ∗ 2.

Original: For every x ∈ S ∗ , there exists a family x δ ∈ S δ with x δ → x .
Revised: For every x ∈ S ∗ , there exists a family x δ ∈ S δ with x δ → x .

Original: The lemma is just an exercise in the deﬁnition.
Revised: The lemma is just an exercise in the deﬁnition.

Original: A. S TATEMENT OF THE T HEOREM
Revised: A. S TATEMENT OF THE T HEOREM

Original: Theorem 2.
Revised: Theorem 2.

Original: Let F, G : R n → [0 , ∞ ) are continuous and assume that F is coercive.
Revised: Let F, G : R n → [0 , ∞ ) are continuous and assume that F is coercive.

Original: Consider the sets S F + δG , the set of points at which F + δG is globally minimum.
Revised: Consider the sets S F + δG , the set of points at which F + δG is globally minimum.

Original: The following are true:
Revised: The following are true:

Original: If δ n → 0 and S F + δ n G → S ∗ , then S ∗ ⊂ S F,G 2.
Revised: If δ n → 0 and S F + δ n G → S ∗ , then S ∗ ⊂ S F,G 2.

Original: If δ n → 0 then there is a subsequence δ n k → 0 and a non-empty set S ∗ ⊂ S F,G so that S F + δ nk G
Revised: If δ n → 0 then there is a subsequence δ n k → 0 and a non-empty set S ∗ ⊂ S F,G so that S F + δ nk G

Original: → S ∗ .
Revised: → S ∗ .

Original: Proof.
Revised: Proof.

Original: The second statement follows from the standard theory of Hausdorff distance on compact metric spaces and the ﬁrst statement.
Revised: The second statement follows from the standard theory of Hausdorff distance on compact metric spaces and the ﬁrst statement.

Original: For the ﬁrst statement, assume that S F + δ n G → S ∗ .
Revised: For the ﬁrst statement, assume that S F + δ n G → S ∗ .

Original: We wish to show that S ∗ ⊂ S F,G .
Revised: We wish to show that S ∗ ⊂ S F,G .

Original: Assume that x n is a sequence of global minima of F + δ n G converging to x ∗ .
Revised: Assume that x n is a sequence of global minima of F + δ n G converging to x ∗ .

Original: It sufﬁces to show that x ∗ ∈ S F,G .
Revised: It sufﬁces to show that x ∗ ∈ S F,G .

Original: First let us observe that x ∗ ∈ S F .
Revised: First let us observe that x ∗ ∈ S F .

Original: Indeed, let
Revised: Indeed, let

Original: [Equation]
Revised: [Equation]

Original: and assume that x ∈ S F .
Revised: and assume that x ∈ S F .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Thus, since F is continuous and x n → x ∗ we have that F ( x ∗ ) =
Revised: Thus, since F is continuous and x n → x ∗ we have that F ( x ∗ ) =

Original: λ which implies x ∗ ∈ S F .
Revised: λ which implies x ∗ ∈ S F .

Original: Next, deﬁne
Revised: Next, deﬁne

Original: [Equation]
Revised: [Equation]

Original: Let ˆ x ∈ S F,G so that G (ˆ x ) = µ .
Revised: Let ˆ x ∈ S F,G so that G (ˆ x ) = µ .

Original: Now observe that, by the minimality of x n we have that
Revised: Now observe that, by the minimality of x n we have that

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: for all n .
Revised: for all n .

Original: Since G is continuous and x n → x ∗ we have that G ( x ∗ ) ≤ µ which implies that G ( x ∗ ) =
Revised: Since G is continuous and x n → x ∗ we have that G ( x ∗ ) ≤ µ which implies that G ( x ∗ ) =

Original: µ since x ∗ ∈ S F .
Revised: µ since x ∗ ∈ S F .

Original: Thus, x ∗ ∈ S F,G .
Revised: Thus, x ∗ ∈ S F,G .

Original: B Q UANTIZER
Revised: B Q UANTIZER

Original: Here, we give an overview about the used quantization method.
Revised: Here, we give an overview about the used quantization method.

Original: Consider a ﬂoating-point variable w f to be mapped into a quantized domain using ( b + 1) bits.
Revised: Consider a ﬂoating-point variable w f to be mapped into a quantized domain using ( b + 1) bits.

Original: Let Q be a set of (2 k + 1) quantized values, where k = 2 b − 1 .
Revised: Let Q be a set of (2 k + 1) quantized values, where k = 2 b − 1 .

Original: Considering linear quantization, Q can be represented as (cid:8) − 1 , − k − 1 k , ..., − 1 k , 0 , 1 k , ..., k − 1 k , 1 (cid:9) , where 1 k is the size of the quantization bin.
Revised: Considering linear quantization, Q can be represented as (cid:8) − 1 , − k − 1 k , ..., − 1 k , 0 , 1 k , ..., k − 1 k , 1 (cid:9) , where 1 k is the size of the quantization bin.

Original: Now, w f can be mapped to the b -bit quantization (Zhou et al., 2016) space as follows:
Revised: Now, w f can be mapped to the b -bit quantization (Zhou et al., 2016) space as follows:

Original: [Equation]
Revised: [Equation]

Original: where quantize b ( x ) = 1 2 b − 1 round((2 b − 1) x ) , w f is a scalar, W f is a vector, and w qo is a scalar and tanh is used to limit its range to [ − 1 , 1] .
Revised: where quantize b ( x ) = 1 2 b − 1 round((2 b − 1) x ) , w f is a scalar, W f is a vector, and w qo is a scalar and tanh is used to limit its range to [ − 1 , 1] .

Original: Then, a scaling factor c is determined per layer to map the ﬁnal quantized weight w q into the range [ − c, + c ] .
Revised: Then, a scaling factor c is determined per layer to map the ﬁnal quantized weight w q into the range [ − c, + c ] .

Original: As such, w q takes the form cw qo , where c > 0 , and w qo ∈ Q .
Revised: As such, w q takes the form cw qo , where c > 0 , and w qo ∈ Q .

Original: These learned parameters ( b, α ), as explained in Section 2.2, can be mapped to the quantizer parameters explained in Equation equation B.1.
Revised: These learned parameters ( b, α ), as explained in Section 2.2, can be mapped to the quantizer parameters explained in Equation equation B.1.

Original: For ( b + 1) bits quantization (the extra bit is the sign bit):
Revised: For ( b + 1) bits quantization (the extra bit is the sign bit):

Original: [Equation]
Revised: [Equation]

Original: C C ONVERGENCE A NALYSIS
Revised: C C ONVERGENCE A NALYSIS

Original: Figure 8 (a), (b) show the convergence behavior of WaveQ by visualizing both accuracy and regularization loss over ﬁnetuning epochs for two networks: CIFAR10 and SVHN.
Revised: Figure 8 (a), (b) show the convergence behavior of WaveQ by visualizing both accuracy and regularization loss over ﬁnetuning epochs for two networks: CIFAR10 and SVHN.

Original: As can be seen, the regularization loss ( WaveQ Loss) is minimized across the ﬁnetuning epochs while the accuracy is maximized.
Revised: As can be seen, the regularization loss ( WaveQ Loss) is minimized across the ﬁnetuning epochs while the accuracy is maximized.

Original: This demonstrates a validity for the proposed regularization being able to optimize the two objectives simultaneously.
Revised: This demonstrates a validity for the proposed regularization being able to optimize the two objectives simultaneously.

Original: Figure 8 (c), (d) contrasts the convergence behavior with and without
Revised: Figure 8 (c), (d) contrasts the convergence behavior with and without

Original: WaveQ for the case of training from scratch for VGG-11.
Revised: WaveQ for the case of training from scratch for VGG-11.

Original: As can be seen, at the onset of training, the accuracy in the presence of WaveQ is behind that without WaveQ .
Revised: As can be seen, at the onset of training, the accuracy in the presence of WaveQ is behind that without WaveQ .

Original: This can be explained as a result of optimizing for an extra objective in case of with WaveQ as compared to without.
Revised: This can be explained as a result of optimizing for an extra objective in case of with WaveQ as compared to without.

Original: Shortly thereafter, the regularization effect kicks in and eventually achieves ∼ 6% accuracy improvement.
Revised: Shortly thereafter, the regularization effect kicks in and eventually achieves ∼ 6% accuracy improvement.

Original: The convergence behavior, however, is primarily controlled by the regularization strengths ( λ w , λ β ) .
Revised: The convergence behavior, however, is primarily controlled by the regularization strengths ( λ w , λ β ) .

Original: As brieﬂy mentioned in Section 2.2, ( λ w , λ β ) ∈ [0 , ∞ ) is a hyperparameter that weights the relative contribution of the proposed regularization objective to the standard accuracy objective.
Revised: As brieﬂy mentioned in Section 2.2, ( λ w , λ β ) ∈ [0 , ∞ ) is a hyperparameter that weights the relative contribution of the proposed regularization objective to the standard accuracy objective.

Original: We reckon that careful setting of λ w , λ β across the layers and during the training epochs is essential for optimum results (Choi et al., 2018b).
Revised: We reckon that careful setting of λ w , λ β across the layers and during the training epochs is essential for optimum results (Choi et al., 2018b).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D W AVE Q PERFORMANCE ON BERT
Revised: D W AVE Q PERFORMANCE ON BERT

Original: Additionally, Table 5 provides layer-wise quantization with a heterogeneous mix of 4 and 5 bits for the BERT model.
Revised: Additionally, Table 5 provides layer-wise quantization with a heterogeneous mix of 4 and 5 bits for the BERT model.

Original: In all cases, WaveQ improves UPOS and LAS metrics for two French treebanks (SPOKEN, PARTUT).
Revised: In all cases, WaveQ improves UPOS and LAS metrics for two French treebanks (SPOKEN, PARTUT).

Original: E T RAINING FROM S CRATCH
Revised: E T RAINING FROM S CRATCH

Original: Table 6 shows a comparison between training from scratch with WaveQ vs without.
Revised: Table 6 shows a comparison between training from scratch with WaveQ vs without.

Original: It can be seen that incorporating WaveQ into the training process achieves strictly better accuracy than the baseline training without WaveQ across all cases.
Revised: It can be seen that incorporating WaveQ into the training process achieves strictly better accuracy than the baseline training without WaveQ across all cases.

Original: Moreover, higher improvements are obtained at lower bitwidths reaching to 35%
Revised: Moreover, higher improvements are obtained at lower bitwidths reaching to 35%

Original: F R EGULARIZATION S TRENGTHS
Revised: F R EGULARIZATION S TRENGTHS

Original: Having a regularization strength is a normal setting associated with any regularization method.
Revised: Having a regularization strength is a normal setting associated with any regularization method.

Original: The criterion for choosing λ w and λ β is to balance the magnitude of regularization loss to be smaller than the magnitude of accuracy loss.
Revised: The criterion for choosing λ w and λ β is to balance the magnitude of regularization loss to be smaller than the magnitude of accuracy loss.

Original: We then perform a grid search over a few points and chose the ones with the best convergence.
Revised: We then perform a grid search over a few points and chose the ones with the best convergence.

Original: From the theoretical perspective, while the theorem is stated in terms of a limit as the regularization parameter vanishes, the proof in fact gives a corresponding stability result.
Revised: From the theoretical perspective, while the theorem is stated in terms of a limit as the regularization parameter vanishes, the proof in fact gives a corresponding stability result.

Original: Namely, if the regularization parameter is sufﬁciently small relative to the main loss then the minimizers will be “almost” quantized.
Revised: Namely, if the regularization parameter is sufﬁciently small relative to the main loss then the minimizers will be “almost” quantized.

Original: G H ETEROGENEOUS C OMPARISON
Revised: G H ETEROGENEOUS C OMPARISON

Original: For Mobilenet-V2, WaveQ quantizes the network to an average bitwidth of 3.95(20.66 GBOPS) compared to 5.90(29.16 GBOPs) reported by Uhlich et al.
Revised: For Mobilenet-V2, WaveQ quantizes the network to an average bitwidth of 3.95(20.66 GBOPS) compared to 5.90(29.16 GBOPs) reported by Uhlich et al.

Original: Similarly, for Resnet-18, WaveQ achieves an average bitwidth of 3.57(62.56 GBOPs) compared to 5.47(65.90 GBOPs) by Uhlich et al.
Revised: Similarly, for Resnet-18, WaveQ achieves an average bitwidth of 3.57(62.56 GBOPs) compared to 5.47(65.90 GBOPs) by Uhlich et al.

Original: Table 7 shows this comparison.
Revised: Table 7 shows this comparison.

Original: However this is not a fair comparison since WaveQ is a regularization method and not a full-blown quantization technique.
Revised: However this is not a fair comparison since WaveQ is a regularization method and not a full-blown quantization technique.

Original: [Table]
Revised: [Table]