Original: P OST -T RAINING W EIGHTED Q UANTIZATION OF N EURAL N ETWORKS FOR L ANGUAGE M ODELS
Revised: P OST -T RAINING W EIGHTED Q UANTIZATION OF N EURAL N ETWORKS FOR L ANGUAGE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As a practical model compression technique, parameter quantization is effective especially for language models associated with a large memory footprint.
Revised: As a practical model compression technique, parameter quantization is effective especially for language models associated with a large memory footprint.

Original: Neural network quantization is usually performed to reduce quantization loss assuming that quantization error of each parameter equally contributes to the overall training loss.
Revised: Neural network quantization is usually performed to reduce quantization loss assuming that quantization error of each parameter equally contributes to the overall training loss.

Original: The importance of each parameter, however, may highly differ such that for the same number of quantization bits, certain parameters lead to higher training loss than the others after quantization.
Revised: The importance of each parameter, however, may highly differ such that for the same number of quantization bits, certain parameters lead to higher training loss than the others after quantization.

Original: In this paper, we consider a non-uniform quantization scheme, speciﬁcally binary-coding-based quantization, for high compression ratio and efﬁcient computations while avoiding large accuracy degradation by uniform quantization (e.g., INT8).
Revised: In this paper, we consider a non-uniform quantization scheme, speciﬁcally binary-coding-based quantization, for high compression ratio and efﬁcient computations while avoiding large accuracy degradation by uniform quantization (e.g., INT8).

Original: Then, we derive quantization optimization methods to take into account the importance of each parameter.
Revised: Then, we derive quantization optimization methods to take into account the importance of each parameter.

Original: We demonstrate that for post-training quantization, weight magnitude can represent importance and improve model accuracy signiﬁcantly compared to the previous schemes lacking importance considerations.
Revised: We demonstrate that for post-training quantization, weight magnitude can represent importance and improve model accuracy signiﬁcantly compared to the previous schemes lacking importance considerations.

Original: For various language models including BERT, DistilBERT, AWD-LSTM, and Transformer, we achieve 2-4 bits per weight by our proposed post-training quantization with reasonable accuracy degradation.
Revised: For various language models including BERT, DistilBERT, AWD-LSTM, and Transformer, we achieve 2-4 bits per weight by our proposed post-training quantization with reasonable accuracy degradation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training techniques for deep neural networks (DNNs) have been developed in ways to incur a lot of parameter redundancy to expedite seeking local minima (Denil et al., 2013; Jonathan Frankle, 2019).
Revised: Training techniques for deep neural networks (DNNs) have been developed in ways to incur a lot of parameter redundancy to expedite seeking local minima (Denil et al., 2013; Jonathan Frankle, 2019).

Original: As a result, various model compression techniques including parameter pruning (Han et al., 2015;
Revised: As a result, various model compression techniques including parameter pruning (Han et al., 2015;

Original: He et al., 2017), quantization (Courbariaux et al., 2015; Rastegari et al., 2016), low-rank approximation (N. Sainath et al., 2013; Prabhavalkar et al., 2016), and knowledge distillation (Hinton et al., 2015;
Revised: He et al., 2017), quantization (Courbariaux et al., 2015; Rastegari et al., 2016), low-rank approximation (N. Sainath et al., 2013; Prabhavalkar et al., 2016), and knowledge distillation (Hinton et al., 2015;

Original: Polino et al., 2018) are proposed to lower storage requirements and improve inference performance.
Revised: Polino et al., 2018) are proposed to lower storage requirements and improve inference performance.

Original: Several compression techniques can be combined in a synergistic way to enhance compression ratio (Han et al., 2016; Zhu et al., 2017).
Revised: Several compression techniques can be combined in a synergistic way to enhance compression ratio (Han et al., 2016; Zhu et al., 2017).

Original: In this work, we consider parameter quantization that maintains structured model formats and presents high compression ratio.
Revised: In this work, we consider parameter quantization that maintains structured model formats and presents high compression ratio.

Original: Note that due to limited hardware resources, quantization is an essential method for any inference systems.
Revised: Note that due to limited hardware resources, quantization is an essential method for any inference systems.

Original: In general, quantization is classiﬁed into uniform quantization based on ﬁxed-point parameter representations (Jacob et al., 2018; Han et al., 2016) and non-uniform quantization associated with the binary codes (Zhou et al., 2017; Rastegari et al., 2016) or codebooks (Choi et al., 2017; Stock et al., 2020).
Revised: In general, quantization is classiﬁed into uniform quantization based on ﬁxed-point parameter representations (Jacob et al., 2018; Han et al., 2016) and non-uniform quantization associated with the binary codes (Zhou et al., 2017; Rastegari et al., 2016) or codebooks (Choi et al., 2017; Stock et al., 2020).

Original: Most DNN quantization methods are performed based on the principle of minimizing the mean squared error (MSE) of quantized parameters (Rastegari et al., 2016; Xu et al., 2018; Zhou et al., 2017).
Revised: Most DNN quantization methods are performed based on the principle of minimizing the mean squared error (MSE) of quantized parameters (Rastegari et al., 2016; Xu et al., 2018; Zhou et al., 2017).

Original: Optimizing the MSE is also an underlying principle of low-rank approximation techniques such as the singular value decomposition (SVD) (Prabhavalkar et al., 2016; N. Sainath et al., 2013).
Revised: Optimizing the MSE is also an underlying principle of low-rank approximation techniques such as the singular value decomposition (SVD) (Prabhavalkar et al., 2016; N. Sainath et al., 2013).

Original: Note that, however, minimizing the MSE implies that each parameter is equally important (i.e., squared errors from parameters are accumulated without considering importance of each weight).
Revised: Note that, however, minimizing the MSE implies that each parameter is equally important (i.e., squared errors from parameters are accumulated without considering importance of each weight).

Original: In practice, the impact of each parameter perturbation from quantization on training loss can be vastly different and such impact needs to be analyzed through a sensitivity study of each parameter toward a change in training loss value.
Revised: In practice, the impact of each parameter perturbation from quantization on training loss can be vastly different and such impact needs to be analyzed through a sensitivity study of each parameter toward a change in training loss value.

Original: In other words, minimizing the MSE (or the Euclidean distance between original parameters and quantized parameters) may not correspond to minimizing training loss function after quantization.
Revised: In other words, minimizing the MSE (or the Euclidean distance between original parameters and quantized parameters) may not correspond to minimizing training loss function after quantization.

Original: Robustness to quantization error of each parameter can be expressed as sensitivity.
Revised: Robustness to quantization error of each parameter can be expressed as sensitivity.

Original: Sensitivity of i -th parameter w i is the amount of change in the loss function when w i is perturbed.
Revised: Sensitivity of i -th parameter w i is the amount of change in the loss function when w i is perturbed.

Original: A parameter associated with high sensitivity would require relatively smaller quantization error if quantization is performed in a group manner.
Revised: A parameter associated with high sensitivity would require relatively smaller quantization error if quantization is performed in a group manner.

Original: Several previous works acknowledge distinct sensitivity of each parameter to improve quantization quality.
Revised: Several previous works acknowledge distinct sensitivity of each parameter to improve quantization quality.

Original: Note that because exact sensitivity estimation of each parameter toward loss function is highly complicated, various heuristic techniques have been introduced.
Revised: Note that because exact sensitivity estimation of each parameter toward loss function is highly complicated, various heuristic techniques have been introduced.

Original: For example, Hessian-weighted k-means clustering is used for codebook-based implementations (Choi et al., 2017) or Taylor series expansion to bound loss function difference is conducted to decide the optimal quantization bits of each weight (Khoram & Li, 2018).
Revised: For example, Hessian-weighted k-means clustering is used for codebook-based implementations (Choi et al., 2017) or Taylor series expansion to bound loss function difference is conducted to decide the optimal quantization bits of each weight (Khoram & Li, 2018).

Original: The Hessian matrix can be used to assign different numbers of quantization bits for each layer (Dong et al., 2019;
Revised: The Hessian matrix can be used to assign different numbers of quantization bits for each layer (Dong et al., 2019;

Original: Shen et al., 2019).
Revised: Shen et al., 2019).

Original: Minimizing the reconstruction error on the output activations after each layer quantization is performed in (Stock et al., 2020).
Revised: Minimizing the reconstruction error on the output activations after each layer quantization is performed in (Stock et al., 2020).

Original: In this paper, we propose a weighted quantization framework where quantized parameters follow the structure of the binary codes so as to achieve high compression ratio and high computational efﬁciency (Rastegari et al., 2016; Jeon et al., 2020).
Revised: In this paper, we propose a weighted quantization framework where quantized parameters follow the structure of the binary codes so as to achieve high compression ratio and high computational efﬁciency (Rastegari et al., 2016; Jeon et al., 2020).

Original: Speciﬁcally, given that an importance of each parameter is represented as a real number between 0 and 1, we extract an optimal quantization solution modiﬁed from the previous binary-coding-based quantization methods that employ equal parameter importance.
Revised: Speciﬁcally, given that an importance of each parameter is represented as a real number between 0 and 1, we extract an optimal quantization solution modiﬁed from the previous binary-coding-based quantization methods that employ equal parameter importance.

Original: Similar to previous attempts, we also ﬁnd that calculating accurate importance of each parameter is challenging.
Revised: Similar to previous attempts, we also ﬁnd that calculating accurate importance of each parameter is challenging.

Original: As a successful approximation of importance, we suggest that magnitude-based importance estimation is especially effective for post-training non-uniform quantization.
Revised: As a successful approximation of importance, we suggest that magnitude-based importance estimation is especially effective for post-training non-uniform quantization.

Original:  P OST -T RAINING P ARAMETER Q UANTIZATION FOR L ANGUAGE M ODELS
Revised:  P OST -T RAINING P ARAMETER Q UANTIZATION FOR L ANGUAGE M ODELS

Original: The number of parameters for language models is dramatically increasing (e.g., GPT-3 (Brown et al., 2020) requires 175 billion parameters).
Revised: The number of parameters for language models is dramatically increasing (e.g., GPT-3 (Brown et al., 2020) requires 175 billion parameters).

Original: Correspondingly, model compression for language models is becoming a mandatory process to reduce response time and inference energy.
Revised: Correspondingly, model compression for language models is becoming a mandatory process to reduce response time and inference energy.

Original: We devise a compression method considering the followings:
Revised: We devise a compression method considering the followings:

Original: • Recent language models are usually memory-bound because of small batch size and lacking layers of high reuse (e.g., conv layers).
Revised: • Recent language models are usually memory-bound because of small batch size and lacking layers of high reuse (e.g., conv layers).

Original: Thus, reducing memory footprint is critical .
Revised: Thus, reducing memory footprint is critical .

Original: • Compression algorithms should be supported by dedicated kernels , designed speciﬁcally for language models if possible.
Revised: • Compression algorithms should be supported by dedicated kernels , designed speciﬁcally for language models if possible.

Original: • Compression-aware training is challenging and expensive if hyper-parameters are added to already huge language models (hence, we choose a post-training method. )
Revised: • Compression-aware training is challenging and expensive if hyper-parameters are added to already huge language models (hence, we choose a post-training method. )

Original: Fixed-point inference using uniform quantization is not desirable for language models because of noticeable accuracy degradation (Shen et al., 2019; Jeon et al., 2020) while the advantage of small computational units (e.g., INT8 MAC) is insigniﬁcant for memory-bound applications.
Revised: Fixed-point inference using uniform quantization is not desirable for language models because of noticeable accuracy degradation (Shen et al., 2019; Jeon et al., 2020) while the advantage of small computational units (e.g., INT8 MAC) is insigniﬁcant for memory-bound applications.

Original: Thus, we adopt ﬂoat-based parameter quantization (i.e., expected values of quantized parameters remains to be of full precision) that induce a lot smaller number of quantization bits compared to ﬁxed-point quantization (Xu et al., 2018; Stock et al., 2020).
Revised: Thus, we adopt ﬂoat-based parameter quantization (i.e., expected values of quantized parameters remains to be of full precision) that induce a lot smaller number of quantization bits compared to ﬁxed-point quantization (Xu et al., 2018; Stock et al., 2020).

Original: Recently, a kernel library, called BiQGEMM (Jeon et al., 2020), was introduced to support binarycoding-based quantization techniques to accelerate quantized neural networks.
Revised: Recently, a kernel library, called BiQGEMM (Jeon et al., 2020), was introduced to support binarycoding-based quantization techniques to accelerate quantized neural networks.

Original: Using lookup tables,
Revised: Using lookup tables,

Original: BiQGEMM enables byte-level memory accesses and achieves 8.3 × run-time memory footprints and 3.5 × speed up with a mobile CPU for Transformer (Chung et al., 2020).
Revised: BiQGEMM enables byte-level memory accesses and achieves 8.3 × run-time memory footprints and 3.5 × speed up with a mobile CPU for Transformer (Chung et al., 2020).

Original: As a result, binary-codingbased quantization has become a practical approach to quantizing language models.
Revised: As a result, binary-codingbased quantization has become a practical approach to quantizing language models.

Original: As such, we restrict our interests to binary-coding-based quantization technique in this paper .
Revised: As such, we restrict our interests to binary-coding-based quantization technique in this paper .

Original: Quantization-aware training is an active research area to improve model accuracy (Courbariaux et al., 2015; Lee et al., 2018).
Revised: Quantization-aware training is an active research area to improve model accuracy (Courbariaux et al., 2015; Lee et al., 2018).

Original: We note that in the case of language models, however, there are numerous occasions when retraining for quantization is not available.
Revised: We note that in the case of language models, however, there are numerous occasions when retraining for quantization is not available.

Original: For example, quantizationaware training requires in-depth knowledge on model compression while model designers may not have such expertise.
Revised: For example, quantizationaware training requires in-depth knowledge on model compression while model designers may not have such expertise.

Original: On the other hand, the original training code or the entire training data may not be shared with model compression engineers.
Revised: On the other hand, the original training code or the entire training data may not be shared with model compression engineers.

Original: Also, modifying the original DNN models to be aware of quantization would increase model design efforts and training time signiﬁcantly.
Revised: Also, modifying the original DNN models to be aware of quantization would increase model design efforts and training time signiﬁcantly.

Original: Since language models already demand signiﬁcant training time and cost, adding additional training complexity by quantization-aware training would not be a practical option.
Revised: Since language models already demand signiﬁcant training time and cost, adding additional training complexity by quantization-aware training would not be a practical option.

Original: As such, post-training quantization without retraining is gaining increasing attention (Zhao et al., 2019; Nagel et al., 2019).
Revised: As such, post-training quantization without retraining is gaining increasing attention (Zhao et al., 2019; Nagel et al., 2019).

Original:  W EIGHTED Q UANTIZATION B ASED ON THE B INARY C ODES
Revised:  W EIGHTED Q UANTIZATION B ASED ON THE B INARY C ODES

Original: As discussed, we choose post-training binary-coding-based quantization as our strategy to compress language DNN models efﬁciently.
Revised: As discussed, we choose post-training binary-coding-based quantization as our strategy to compress language DNN models efﬁciently.

Original: Following the Binary-Weight-Networks (Rastegari et al., 2016) introducing the binary codes as a quantization format, a weight vector w is approximated to be α b by using a scaling factor α ∈ R and a binary vector b (= {− 1 , +1 } n ) , where n is the vector size.
Revised: Following the Binary-Weight-Networks (Rastegari et al., 2016) introducing the binary codes as a quantization format, a weight vector w is approximated to be α b by using a scaling factor α ∈ R and a binary vector b (= {− 1 , +1 } n ) , where n is the vector size.

Original: A real number scaling factor is shared by multiple weights such that binary vector b occupies most of the weight storage requirements.
Revised: A real number scaling factor is shared by multiple weights such that binary vector b occupies most of the weight storage requirements.

Original: Binary codes eliminate the need for dequantization for inference, leading to reduced on-chip memory size for weights.
Revised: Binary codes eliminate the need for dequantization for inference, leading to reduced on-chip memory size for weights.

Original: In this section, we study general weighted quantization methodologies when quantization follows the format of the binary codes and and quantization error recognizes sensitivity information.
Revised: In this section, we study general weighted quantization methodologies when quantization follows the format of the binary codes and and quantization error recognizes sensitivity information.

Original: 3. G REEDY M ETHOD AND A LTERNATING M ETHOD WITHOUT I MPORTANCE C ONSIDERATIONS
Revised: 3. G REEDY M ETHOD AND A LTERNATING M ETHOD WITHOUT I MPORTANCE C ONSIDERATIONS

Original: In general, non-uniform weight quantization methods (in the form of the binary codes) strive to minimize (cid:107) w − α b (cid:107) 2 .
Revised: In general, non-uniform weight quantization methods (in the form of the binary codes) strive to minimize (cid:107) w − α b (cid:107) 2 .

Original: In the case of 1-bit quantization, we obtain the following analytical solution:
Revised: In the case of 1-bit quantization, we obtain the following analytical solution:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, in the case of multi-bit quantization, there is no analytical solution (Rastegari et al., 2016; Xu et al., 2018).
Revised: On the other hand, in the case of multi-bit quantization, there is no analytical solution (Rastegari et al., 2016; Xu et al., 2018).

Original: As a result, various approximated methods exist for multi-bit quantization.
Revised: As a result, various approximated methods exist for multi-bit quantization.

Original: Greedy Method As a computationally simple method, 1-bit quantization shown in Eq.
Revised: Greedy Method As a computationally simple method, 1-bit quantization shown in Eq.

Original: (1) can be extended to multi-bit ( q -bit) quantization (Guo et al., 2017).
Revised: (1) can be extended to multi-bit ( q -bit) quantization (Guo et al., 2017).

Original: Speciﬁcally, i th -bit ( i > 1 ) quantization is performed by minimizing the residue of ( i − 1) th -bit quantization as following:
Revised: Speciﬁcally, i th -bit ( i > 1 ) quantization is performed by minimizing the residue of ( i − 1) th -bit quantization as following:

Original: [Equation]
Revised: [Equation]

Original: i b i (cid:107) 2 , where r i − 1 = w − i
Revised: i b i (cid:107) 2 , where r i − 1 = w − i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The optimal solution of Eq.
Revised: The optimal solution of Eq.

Original: (2) is then given as
Revised: (2) is then given as

Original: [Equation]
Revised: [Equation]

Original: Alternating Method Greedy method described above is non-iterative.
Revised: Alternating Method Greedy method described above is non-iterative.

Original: In order to reduce (cid:107) w −
Revised: In order to reduce (cid:107) w −

Original: [Equation]
Revised: [Equation]

Original: qi =
Revised: qi =

Original: α i b i (cid:107) 2 further than Greedy method, iterative methods would be necessary while increasing the number of iterations tends to lower quantization error.
Revised: α i b i (cid:107) 2 further than Greedy method, iterative methods would be necessary while increasing the number of iterations tends to lower quantization error.

Original: Once initial α and b values are calculated by
Revised: Once initial α and b values are calculated by

Original: Greedy method, one can notice that { α i } qi =1 can be reﬁned (Guo et al., 2017) as
Revised: Greedy method, one can notice that { α i } qi =1 can be reﬁned (Guo et al., 2017) as

Original: [Equation]
Revised: [Equation]

Original: when B q = [ b
Revised: when B q = [ b

Original: [Equation]
Revised: [Equation]

Original: Then, B q can be reﬁned as well by binary search given a new reﬁned { α i } qi =1 .
Revised: Then, B q can be reﬁned as well by binary search given a new reﬁned { α i } qi =1 .

Original: As a result, { α i } qi = and B q are reﬁned alternatively.
Revised: As a result, { α i } qi = and B q are reﬁned alternatively.

Original: Alternating reﬁnements of { α i } qi =1 and B k are repeated until there is no noticeable improvement in quantization error.
Revised: Alternating reﬁnements of { α i } qi =1 and B k are repeated until there is no noticeable improvement in quantization error.

Original: Such iterative quantization procedure is introduced as Alternating multi-bit method (Xu et al., 2018).
Revised: Such iterative quantization procedure is introduced as Alternating multi-bit method (Xu et al., 2018).

Original: 3. I MPORTANCE - AWARE W EIGHTED Q UANTIZATION
Revised: 3. I MPORTANCE - AWARE W EIGHTED Q UANTIZATION

Original: Let us assume that the importance of the i -th parameter is normalized and given as m i ( 0 ≤ m i ≤ 1 ).
Revised: Let us assume that the importance of the i -th parameter is normalized and given as m i ( 0 ≤ m i ≤ 1 ).

Original: Then, we minimize the weighted quantization loss (cid:80) n i =1 ( m i ( w i − ˆ w i ) 2 ) where w i is quantized to be ˆ w i
Revised: Then, we minimize the weighted quantization loss (cid:80) n i =1 ( m i ( w i − ˆ w i ) 2 ) where w i is quantized to be ˆ w i

Original: = (cid:80) qj
Revised: = (cid:80) qj

Original: =1 ( α j b j ) .
Revised: =1 ( α j b j ) .

Original: Before studying how to estimate importance values, we are interested in ﬁnding modiﬁed versions of Greedy method and Alternating method when importance values are given.
Revised: Before studying how to estimate importance values, we are interested in ﬁnding modiﬁed versions of Greedy method and Alternating method when importance values are given.

Original: For 1-bit quantization, weighted quantization also has the following analytical solution:
Revised: For 1-bit quantization, weighted quantization also has the following analytical solution:

Original: [Equation]
Revised: [Equation]

Original: Note that if all importance values are equal (e.g., m i = 1 for all i ), then Eq.
Revised: Note that if all importance values are equal (e.g., m i = 1 for all i ), then Eq.

Original: (5) becomes the same as Eq.
Revised: (5) becomes the same as Eq.

Original: Correspondingly, Eq.
Revised: Correspondingly, Eq.

Original: (1) can be regarded as a special case of Eq.
Revised: (1) can be regarded as a special case of Eq.

Original: Compared to the conventional Greedy method, our proposed importance-aware Greedy method demands modiﬁcations to α i calculations as α ∗ i = (cid:80) ni
Revised: Compared to the conventional Greedy method, our proposed importance-aware Greedy method demands modiﬁcations to α i calculations as α ∗ i = (cid:80) ni

Original: =1 ( m i | r
Revised: =1 ( m i | r

Original: i − 1 | ) / (cid:80) ni
Revised: i − 1 | ) / (cid:80) ni

Original: =

Revised: =

Original: m i .
Revised: m i .

Original: For the importance-aware Alternating method, we ﬁrst conduct the importance-aware Greedy method.
Revised: For the importance-aware Alternating method, we ﬁrst conduct the importance-aware Greedy method.

Original: Then, Eq.
Revised: Then, Eq.

Original: (4) is to be transformed to employ importance.
Revised: (4) is to be transformed to employ importance.

Original: Let us deﬁne an n -by- n diagonal matrix
Revised: Let us deﬁne an n -by- n diagonal matrix

Original: M = diag( m 1 , ..., m n ) , where each diagonal element is an importance value m i .
Revised: M = diag( m 1 , ..., m n ) , where each diagonal element is an importance value m i .

Original: By solving linear least squares, α values are reﬁned as
Revised: By solving linear least squares, α values are reﬁned as

Original: [Equation]
Revised: [Equation]

Original: while reﬁning B q is still performed by binary search using reﬁned scaling factors.
Revised: while reﬁning B q is still performed by binary search using reﬁned scaling factors.

Original: Accordingly,
Revised: Accordingly,

Original: Eq.
Revised: Eq.

Original: (4) is a particular case of Eq.
Revised: (4) is a particular case of Eq.

Original: (6) when M is an identity matrix.
Revised: (6) when M is an identity matrix.

Original: Overall, our proposed importance-aware quantization scheme is comprehensive to include previous methods as a subset.
Revised: Overall, our proposed importance-aware quantization scheme is comprehensive to include previous methods as a subset.

Original: In the rest of this paper, we investigate simple and efﬁcient schemes to estimate importance metrics applicable to post-training non-uniform binary-coding-based quantization.
Revised: In the rest of this paper, we investigate simple and efﬁcient schemes to estimate importance metrics applicable to post-training non-uniform binary-coding-based quantization.

Original:  I MPORTANCE E STIMATION U SING W EIGHT M AGNITUDE
Revised:  I MPORTANCE E STIMATION U SING W EIGHT M AGNITUDE

Original: Sensitivity or importance of each parameter can be estimated by evaluating the loss function change induced by parameter perturbation.
Revised: Sensitivity or importance of each parameter can be estimated by evaluating the loss function change induced by parameter perturbation.

Original: Such estimation, however, is computationally demanding since each parameter’s perturbation requires computations of entire feedforward paths while the number of parameters is ever increasing in recent DNN designs.
Revised: Such estimation, however, is computationally demanding since each parameter’s perturbation requires computations of entire feedforward paths while the number of parameters is ever increasing in recent DNN designs.

Original: Moreover, it is difﬁcult to decide an appropriate amount of perturbation of parameters.
Revised: Moreover, it is difﬁcult to decide an appropriate amount of perturbation of parameters.

Original: In this section, we propose an efﬁcient importance estimation method based on weight magnitude for fast and high-accurate post-training weight quantization.
Revised: In this section, we propose an efﬁcient importance estimation method based on weight magnitude for fast and high-accurate post-training weight quantization.

Original: 4. W EIGHT M AGNITUDE AS I MPORTANCE
Revised: 4. W EIGHT M AGNITUDE AS I MPORTANCE

Original: Weight magnitude is a popular criterion for weight pruning to decide whether each weight is unimportant so as to be pruned from the network.

Original: In order to examine whether weight magnitude is also a useful one for quantization, we introduce Optimal Brain Damage (OBD) (LeCun et al., 1990).
Revised: As an effort to study a factor affecting importance, we introduce Optimal Brain Damage (OBD) (LeCun et al., 1990).

Original: OBD is proposed to ﬁnd unimportant weights to be removed to reduce the size of neural networks.

Original: As an effort to study a factor affecting importance, OBD approximates the loss function by the Taylor series and a perturbation of the loss function δL by weight perturbations is presented as
Revised: OBD approximates the loss function by the Taylor series and a perturbation of the loss function δL by weight perturbations is presented as

Original: [Equation]
Revised: [Equation]

Original: where v is the number of weights and h i,j is an element of the Hessian matrix.
Revised: where v is the number of weights and h i,j is an element of the Hessian matrix.

Original: Note that at a local minimum, the ﬁrst term is eliminated while all the h i,i are non-negative.
Revised: Note that at a local minimum, the ﬁrst term is eliminated while all the h i,i are non-negative.

Original: Using “diagonal” approximation and “quadratic” approximation (LeCun et al., 1990), Eq.[Equation]
Revised: Using “diagonal” approximation and “quadratic” approximation (LeCun et al., 1990), Eq.

Original: 7) is simpliﬁed as
Revised: (7) is simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: Note that training models converging to smooth minima produce small diagonal elements of the Hessian matrix (Goodfellow et al., 2016; Choromanska et al., 2015).
Revised: As described in Eq. 8, diagonal elements of the Hessian matrix can be used as effective importance metrics for loss-aware training algorithms for quantization (Hou et al., 2017; Hou & Kwok, 2018). Note that for post-training quantization, unfortunately, the ﬁrst or the second partial derivatives (i.e., the gradient or the Hessian) are not available.

Original: Then, weight magnitude can be  adominating factor for the loss function perturbation.
Revised: Thus, for post-training quantization, importance is required to be given as a function of weight magnitude.

Revised: In other words, we are interested whether Hessian-based importance can be replaced with magnitude-based importance.

Revised: The goal of this work is then to empirically show that magnitude-based importance is indeed practical for post-training binary-coding-based quantization.

Original: To verify our basic assumption that large weights would present higher importance for binary-codingbased quantization, we controlled scaling factors (i.e., α values of the minimum MSE are multiplied by ‘Scaling Factor Multiplier’) of a layer in an LSTM model using PTB dataset (see details of models and dataset in Appendix) as shown in Figure 1.
Revised: To verify our basic assumption that large weights would present higher importance for binary-codingbased quantization, we controlled scaling factors (i.e., α values of the minimum MSE are multiplied by ‘Scaling Factor Multiplier’) of a layer in an LSTM model using PTB dataset (see details of models and dataset in Appendix) as shown in Figure 1.

Original: Indeed, when scaling factors become larger than the ones obtained by minimizing MSE present, test perplexity is enhanced for both layers. 
Revised: Indeed, when scaling factors become larger than the ones obtained by minimizing MSE present, test perplexity is enhanced for both layers. [Figure]

Original: 4. H YPER - PARAMETERS FOR I MPORTANCE M ETRICS
Revised: 4. H YPER - PARAMETERS FOR I MPORTANCE M ETRICS

Original: The underlying principle of our importance estimation methodology is that relatively smaller quantization errors are supposed to be applied to weights of larger magnitude.
Revised: The underlying principle of our importance estimation methodology is that relatively smaller quantization errors are supposed to be applied to weights of larger magnitude.

Original: To ﬁne-tune our proposed 
Revised: To ﬁne-tune our proposed importance estimation scheme, we propose the following three hyper-parameters: 1) an exponent to control a correlation between importance and magnitude of a weight

Original: [Figure]
Revised: [Figure]

Original: importance estimation scheme, we propose the following three hyper-parameters: 1) an exponent to control a correlation between importance and magnitude of a weight, 2) a parameter to clip importance in order to handle outliers in magnitude distributions, and 3) a pruning parameter to exclude weights of low magnitude during quantization optimization.
Revised: 2) a parameter to clip importance in order to handle outliers in magnitude distributions [Figure] and 3) a pruning parameter to exclude weights of low magnitude during quantization optimization [Figure]

Original: E (Exponent) A basic form to calculate normalized importance of each weight is presented as
Revised: E (Exponent) A basic form to calculate normalized importance of each weight is presented as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where w max is the maximum weight magnitude in a given layer (i.e., m i is computed in a layer-wise manner) and the weight of the largest magnitude is considered to be the most important.
Revised: (9) where w max is the maximum weight magnitude in a given layer (i.e., m i is computed in a layer-wise manner) and the weight of the largest magnitude is considered to be the most important.

Original: A value between 0.0 and 1.0 for E is primarily adopted in our experiments to yield sub-linear importance increase as weight magnitude increases.
Revised: A value between 0.0 and 1.0 for E is primarily adopted in our experiments to yield sub-linear importance increase as weight magnitude increases.

Original: E = 0 results in the conventional quantization method with m i = 1 for all w i .
Revised: E = 0 results in the conventional quantization method with m i = 1 for all w i .

Original: C (Clipping Importance)
Revised: C (Clipping Importance)

Original: For a given distribution of weight magnitude, a few large outliers may distort the entire distribution of m i obtained by Eq.
Revised: For a given distribution of weight magnitude, a few large outliers may distort the entire distribution of m i obtained by Eq.

Original: In other words, because of a few exceptionally large weights, most weights may exhibit small importance values.
Revised: In other words, because of a few exceptionally large weights, most weights may exhibit small importance values.

Original: A conventional quantization technique to prevent outliers in a distribution is to clip weights/activations and/or gradients (Choi et al., 2018;
Revised: A conventional quantization technique to prevent outliers in a distribution is to clip weights/activations and/or gradients (Choi et al., 2018;

Original: Zhao et al., 2019; Goodfellow et al., 2016).
Revised: Zhao et al., 2019; Goodfellow et al., 2016).

Original: For Eq.
Revised: For Eq.

Original: (9), w max is decided to be the weight magnitude at the ( C × 100) -th percentile when 0 < C ≤ 1 .
Revised: (9), w max is decided to be the weight magnitude at the ( C × 100) -th percentile when 0 < C ≤ 1 .

Original: If m i exceeds 1.0, then m i is forced to be 1.0 (note that w i is not clipped).
Revised: If m i exceeds 1.0, then m i is forced to be 1.0 (note that w i is not clipped).

Original: P (Pruning for Quantization)
Revised: P (Pruning for Quantization)

Original: Due to regularization effects, a lot of weights employ small magnitudes and a weight distribution usually follows a Gaussian distribution (Goodfellow et al., 2016).
Revised: Due to regularization effects, a lot of weights employ small magnitudes and a weight distribution usually follows a Gaussian distribution (Goodfellow et al., 2016).

Original: As a result, a large number of small weights (with less importance as in Eq.
Revised: As a result, a large number of small weights (with less importance as in Eq.

Original: (8)) may take a large portion of total quantization error unless m i values of those weights are extremely small.
Revised: (8)) may take a large portion of total quantization error unless m i values of those weights are extremely small.

Original: Note that even though pruning prior to quantization is an effective method to improve quantization quality (Li & Liu, 2016; Zhu et al., 2017), pruning would yield one additional bit per weight for masking information or sparse matrix formats with low parallelism for DNN inference.
Revised: Note that even though pruning prior to quantization is an effective method to improve quantization quality (Li & Liu, 2016; Zhu et al., 2017), pruning would yield one additional bit per weight for masking information or sparse matrix formats with low parallelism for DNN inference.

Original: In our work, 1) we exclude weights of magnitude smaller than the ( P × 100) -th percentile from the quantization optimization, 2)
Revised: In our work, 1) we exclude weights of magnitude smaller than the ( P × 100) -th percentile from the quantization optimization, 2)

Original: ﬁnd the scaling factors and the binary codes using weights larger than the ( P × 100) -th percentile, and 3) all of the excluded small weights are assigned to a binary code with the smallest magnitude available from combining scaling factors (while each sign information is maintained).
Revised: ﬁnd the scaling factors and the binary codes using weights larger than the ( P × 100) -th percentile, and 3) all of the excluded small weights are assigned to a binary code with the smallest magnitude available from combining scaling factors (while each sign information is maintained).

Original: Accordingly, while we adopt a parameter pruning idea for quantization, additional pruning mask data is not necessary.
Revised: Accordingly, while we adopt a parameter pruning idea for quantization, additional pruning mask data is not necessary.

Original: In short, small weights are not considered while obtaining the binary codes, and then, replaced with the smallest weight in the binary codes.
Revised: In short, small weights are not considered while obtaining the binary codes, and then, replaced with the smallest weight in the binary codes.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4. E MPIRICAL O BSERVATIONS
Revised: 4. E MPIRICAL O BSERVATIONS

Original: To verify basic operations of our proposed method, we perform post-training weighted quantization using pre-trained models of BERT-base (Devlin et al., 2018) with a GLUE benchmark (Wang et al., 2018) on MNLI and MRPC dataset.
Revised: To verify basic operations of our proposed method, we perform post-training weighted quantization using ﬁne-tuned models of BERT-base (Devlin et al., 2018) on MNLI and MRPC dataset within a GLUE benchmark (Wang et al., 2018).

Original: In the case of  BERT models, we quantize all weights except those of a segment embedding layer and a classiﬁcation layer which show a tiny storage footprint.
Revised: In the case of ﬁne-tuned BERT models, we quantize all weights except those of a segment embedding layer and a classiﬁcation layer which show a tiny storage footprint.

Original: For conventional or weighted Alternating quantization methods, we conduct iterative reﬁnements of α and B values 20 times over which no further noticeable quantization error improvement is recognized.
Revised: For conventional or weighted Alternating quantization methods, we conduct iterative reﬁnements of α and B values 20 times over which no further noticeable quantization error improvement is recognized.

Original: Given a weight matrix or tensor, α and B are computed for each row, independently.
Revised: Given a weight matrix or tensor, α and B are computed for each row, independently (hence, we study row-wise quantization in this work).

Original: Due to the space limit, see Appendix for additional experimental results with various models not included in this section.
Revised: Due to the space limit, see Appendix for additional experimental results with various models not included in this section.

Original: We ﬁrst analyze how a simple weighted quantization scheme (with E =1.0, C =1.0, and P =0.0) adds distinguished features to the conventional quantization methods.
Revised: We ﬁrst analyze how a simple weighted quantization scheme (with E =1.0, C =1.0, and P =0.0) adds distinguished features to the conventional quantization methods.

Original: Table 1 presents comparisons on quantization error (MSE), average scaling factor values, training loss, and training accuracy.
Revised: Table 1 presents comparisons on quantization error (MSE), average scaling factor values, training loss, and training accuracy.

Original: Note that for Alternating weighted quantization, despite larger quantization MSE (i.e., (cid:80) ni
Revised: Note that for Alternating weighted quantization, despite larger quantization MSE (i.e., (cid:80) ni

Original: =1 ( w i − ˆ w i ) 2 ), training loss and training model accuracy are improved such that it is conﬁrmed that minimizing (cid:80) ni[Equation]
Revised: =1 ( w i − ˆ w i ) 2 ), training loss and training model accuracy are improved such that it is conﬁrmed that minimizing (cid:80) ni

Original: [Equation] ( w i − ˆ w i ) 2 ) is preferred to minimizing MSE for post-training quantization.
Revised: =1 ( m i ( w i − ˆ w i ) 2 ) is preferred to minimizing MSE for post-training quantization.

Original: It is interesting to see that for the Greedy method, quantization MSE is reduced by weighted quantization.
Revised: It is interesting to see that for the Greedy method, quantization MSE is reduced by weighted quantization.

Original: We conjecture that for weight distributions in DNNs, Eq.
Revised: We conjecture that for weight distributions in DNNs, Eq.

Original: (5) is probably a better approximation compared to Eq.
Revised: (5) is probably a better approximation compared to Eq.

Original: (1) even to minimize quantization MSE.
Revised: (1) even to minimize quantization MSE.

Original: For both Greedy and Alternating methods, scaling factors increase by weighted quantization due to the context of magnitude-based importance design.
Revised: For both Greedy and Alternating methods, scaling factors increase by weighted quantization due to the context of magnitude-based importance design.

Original: Let us study the impact of our proposed weighted quantization on model accuracy when E , C , and P can vary for ﬁne-tuning.
Revised: Let us study the impact of our proposed weighted quantization on model accuracy when E , C , and P can vary for ﬁne-tuning.

Original: Figure 2 describes test accuracy and quantization error of a ﬁnetuned BERT-base model on MRPC when we sweep only one of E , C , or P across all layers.
Revised: Figure 2 describes test accuracy and quantization error of a ﬁnetuned BERT-base model on MRPC when we sweep only one of E , C , or P across all layers.

Original: It is clear that all hyper-parameters for importance metrics enable new search space for model accuracy that is somewhat uncorrelated to quantization error.
Revised: It is clear that all hyper-parameters for importance metrics enable new search space for model accuracy that is somewhat uncorrelated to quantization error.

Original: Using various combinations of E , C , and P ,
Revised: Using various combinations of E , C , and P ,

Original: Table 2 describes test model accuracy of ﬁne-tuned BERT and DistilBERT models using Greedy and
Revised: Table 2 describes test model accuracy of ﬁne-tuned BERT and DistilBERT models using Greedy and

Original: [Table]
Revised: [Table]

Original: Alternating quantization methods.
Revised: Alternating quantization methods.

Original: Even though numerous hyper-parameter combinations outperformoriginal Greedy and Alternating methods, the best set of hyper-parameters varies for each model, and hence, an automated hyper-parameter search process is desirable.
Revised: Even though numerous hyper-parameter combinations outperform Greedy and Alternating methods without importance, the best set of hyper-parameters varies for each model, and hence, an automated hyper-parameter search process is desirable.

Original: Note that in order to enable such an automated process, we need to investigate whether a set of hyper-parameters for importance searched by using training dataset is also valid for test dataset.
Revised: Note that in order to enable such an automated process, we need to investigate whether a set of hyper-parameters for importance searched by using training dataset is also valid for test dataset.

Original: We extensively explored hyper-parameter combinations (using E , C , P ) with different number of reﬁnement iterations using BERT on MRPC and MNLI and conﬁrmed that training model accuracy achieved by our weighted quantization is highly correlated with test model accuracy such that our proposed hyper-parameters maintain a generalization capability (see Fig. 3 in Appendix).
Revised: We extensively explored hyper-parameter combinations (using E , C , P ) with different number of reﬁnement iterations using BERT on MRPC and MNLI and conﬁrmed that training model accuracy achieved by our weighted quantization is highly correlated with test model accuracy such that our proposed hyper-parameters maintain a generalization capability (see Fig. 4 in Appendix).

Original: Increasing the number of scaling factors (by decreasing the number of parameters sharing a scaling factor) enhances model accuracy despite increased memory footprint and computation overhead during inference.
Revised: Increasing the number of scaling factors (by decreasing the number of parameters sharing a scaling factor) enhances model accuracy despite increased memory footprint and computation overhead during inference.

Original: Our proposed weighted quantization obtains larger accuracy improvements when less number of scaling factors is utilized (see Table 7 in Appendix that compares model accuracy with different number of scaling factors when Alternating method is applied to BERT on MRPC.)
Revised: Our proposed weighted quantization obtains larger accuracy improvements when less number of scaling factors is utilized (see Table 7 in Appendix that compares model accuracy with different number of scaling factors when Alternating method is applied to BERT on MRPC.)

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We observed that an optimal set of hyper-parameters needs to be achieved empirically.
Revised: We observed that an optimal set of hyper-parameters needs to be achieved empirically.

Original: Unfortunately, optimizing E , C , and P for post-training quantization to obtain the best model accuracy is challenging because 1) trained models present a variety of weight distributions and 2) hyper-parameters are correlated.
Revised: Unfortunately, optimizing E , C , and P for post-training quantization to obtain the best model accuracy is challenging because 1) trained models present a variety of weight distributions and 2) hyper-parameters are correlated.

Original: As an effort to automate the hyper-parameter search process, we adopt Bayesian optimization (BO) that is implemented by a publicly available code introduced in (Nogueira, 2014–).
Revised: As an effort to automate the hyper-parameter search process, we adopt Bayesian optimization (BO) that is implemented by a publicly available code introduced in (Nogueira, 2014–).

Revised: Once we perform a rough and fast grid search for hyper-parameters (as shown in Table 2), then BO conducts ﬁne-tuning of hyper-parameters.

Revised: As a result, we achieve quick post-training quantization even when optimal hyper-parameters vary for each layer.

Original: For  experiments, training dataset D t is used during hyper-parameter search, and then test dataset D v validates the optimization procedure (refer to Table 4).
Revised: For BO experiments, training dataset D t is used during hyper-parameter search, and then test dataset D v validates the optimization procedure (refer to Table 4).

Original: In other words, given a hyper-parameter vector denoted by x = { E, C, P } ∈ R 3 , BO tries to ﬁnd the optimal x ∗ to be arg max x f ( x ; D t ) , where f measures accuracy of the model.
Revised: In other words, given a hyper-parameter vector denoted by x = { E, C, P } ∈ R 3 , BO tries to ﬁnd the optimal x ∗ to be arg max x f ( x ; D t ) , where f measures accuracy of the model.

Original: Then, test model accuracy is measured as f ( x ∗ ; D v ) .
Revised: Then, test model accuracy is measured as f ( x ∗ ; D v ) .

Original: To the best of our knowledge, our work is the ﬁrst post-training binary-coding-based quantization considering weight importance.
Revised: To the best of our knowledge, our work is the ﬁrst post-training binary-coding-based quantization considering weight importance.

Original: As a result, we compare our results with the conventional Greedy algorithm and Alternating algorithm.
Revised: As a result, we compare our results with the conventional Greedy algorithm and Alternating algorithm.

Original: We consider three different search methods for our proposed scheme: 1) (manual search) we investigate prearranged 16 sets of hyper-parameters for importance metric (described in Table 5), 2) (model-wise BO) for all layers, the same values of E , C , and
Revised: We consider three different search methods for our proposed scheme: 1) (manual search) we investigate prearranged 16 sets of hyper-parameters for importance [Figure] [Figure] SQUAD 0.

Original:  Pare explored and applied for quantization, and 3) (layer-wise BO) hyper-parameters are locally searched for each layer, and the ﬁxed before proceeding to the next layer (hence, BO for quantization
Revised: [Figure] E C P

Original: [Table]
Revised: [Table]

Original:  is performed in layer-by-layer manner). 
Revised: metric (described in Table 5), 2) (model-wise BO) for all layers, the same values of E , C , and P are explored and applied for quantization, and 3) (layer-wise BO) hyper-parameters are locally searched for each layer, and the ﬁxed before proceeding to the next layer (hence, BO for quantization is performed in layer-by-layer manner). For all model-wise or layer-wise BO for various models, the same 16 sets of hyper-parameters (given in Table 5) are explored ﬁrst as initial samples.

Original: BO outperforms our manual search while layer-wise 
Revised: BO outperforms our manual search while layer-wise BO improves test score further as presented in

Original: BO improves test score further as presented in Table 8 (BERT-base), 9 (DistilBERT-base),(Longformer), 13 (AWD-LSTM)
Revised: Table 8 (BERT-base), 9 (DistilBERT-base), 11 (Longformer), 13 (AWD-LSTM), and 14 (Transformer

Original: [Table] and 14 (Transformer NMT).
Revised: NMT).

Original: Among three methods considered, layer-wise BO is the best because the optimal set of hyper-parameters turns out to be vastly different for each layer as shown in Figure 4, 5, 6, [Table] and 9.
Revised: Among three methods considered, layer-wise BO is the best because the optimal set of hyper-parameters turns out to be vastly different for each layer as shown in Figure 3, 5, 6, 8, and 9.

Original: Table 3 presents the overall comparison on test scores of various language models that are quantized by conventional Alternating quantization and our proposed weighted quantization.
Revised: Table 3 presents the overall comparison on test scores of various language models that are quantized by conventional Alternating quantization and our proposed weighted quantization.

Original: Compared to conventional Alternating quantization (that is our baseline for post-training binary-coding-based quantization) of equal importance for each parameter, ours improves test scores for all language models in Table 3.
Revised: Compared to conventional Alternating quantization (that is our baseline for post-training binary-coding-based quantization) of equal importance for each parameter, ours improves test scores for all language models in Table 3.

Original: We note that weighted quantization yields relatively different amounts of improvements on test scores depending on a given model.
Revised: We note that weighted quantization yields relatively different amounts of improvements on test scores depending on a given model.

Original: Even though thorough analysis of such different accuracy enhancement would entail in-depth sensitivity analysis of parameters toward test scores, pruning weights in the context of magnitude (without retraining) provides approximated correlation between importance and magnitude for a given weight distribution (see Figure 7 in
Revised: Even though thorough analysis of such different accuracy enhancement would entail in-depth sensitivity analysis of parameters toward test scores, pruning weights in the context of magnitude (without retraining) provides approximated correlation between importance and magnitude for a given weight distribution (see Figure 7 in

Original: Appendix).
Revised: Appendix).

Original: Indeed, the pruning rate of the Longformer is a lot lower compared to the other models (as shown in Figure 7) that can partly explain challenges to enhance Longformer quantization.
Revised: Indeed, the pruning rate of the Longformer is a lot lower compared to the other models (as shown in Figure 7) that can partly explain challenges to enhance Longformer quantization.

Original: We also note that our proposed method highly depends on the target objective function.
Revised: We also note that our proposed method highly depends on the target objective function.

Original: As such, Transformers show different quantization results depending on the target selection of perplexity (PPL) or BLEU score due to somewhat low correlation between PPL and BLEU score (Appendix C.3).
Revised: As such, Transformers show different quantization results depending on the target selection of perplexity (PPL) or BLEU score due to somewhat low correlation between PPL and BLEU score (Appendix C.3).

Original: We applied our weighted quantization scheme to ResNet models on CIFAR-10 and ImageNet (refer to Appendix D) for which model accuracy is also signiﬁcantly enhanced similar to language models.
Revised: We applied our weighted quantization scheme to ResNet models on CIFAR-10 and ImageNet (refer to Appendix D) for which model accuracy is also signiﬁcantly enhanced similar to language models.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this paper, we propose a weighted quantization framework employing importance metrics that are useful when each parameter shows a different sensitivity toward a loss function change.
Revised: In this paper, we propose a weighted quantization framework employing importance metrics that are useful when each parameter shows a different sensitivity toward a loss function change.

Original: For the binary-coding-based quantization that is our choice for language models because of existing efﬁcient kernel designs (e.g., BiQGEMM ) and high compression ratio, we extract modiﬁed Greedy method and
Revised: For the binary-coding-based quantization that is our choice for language models because of existing efﬁcient kernel designs (e.g., BiQGEMM ) and high compression ratio, we extract modiﬁed Greedy method and

Original: Alternating method assuming that each importance value is represented as a real number betweenand 1.
Revised: Alternating method assuming that each importance value is represented as a real number betweenand 1.

Original: Using various DNN models, we demonstrate that a magnitude-based importance metric is effective for post-training quantization in the form of the binary codes.
Revised: Using various DNN models, we demonstrate that a magnitude-based importance metric is effective for post-training quantization in the form of the binary codes.

Original: To ﬁne-tune model accuracy, we also propose three hyper-parameters that need to be empirically investigated since an optimal set of hyper-parameters varies depending on each model design.
Revised: To ﬁne-tune model accuracy, we also propose three hyper-parameters that need to be empirically investigated since an optimal set of hyper-parameters varies depending on each model design.

Original: As such, we suggest Bayesian optimization as an effective technique to automate hyper-parameter search process.
Revised: As such, we suggest Bayesian optimization as an effective technique to automate hyper-parameter search process.

Original: Our proposed hyper-parameters can be independently optimized to each layer to further improve compression ratio and/or model accuracy.
Revised: Our proposed hyper-parameters can be independently optimized to each layer to further improve compression ratio and/or model accuracy.

Original: It would be interesting to study additional hyper-parameters effective for post-training quantization.
Revised: It would be interesting to study additional hyper-parameters effective for post-training quantization.

Original: Since our weighted quantization framework is general (rather than depending on a particular approximation such as the Hessian), if proper importance metrics are found, our proposed quantization techniques can be extended to a quantization-aware training method.
Revised: Since our weighted quantization framework is general (rather than depending on a particular approximation such as the Hessian), if proper importance metrics are found, our proposed quantization techniques can be extended to a quantization-aware training method.

Original: A M ODELS AND D ATASETS A. LSTM M ODELS
Revised: A M ODELS AND D ATASETS A. LSTM M ODELS

Original: 1-Layer LSTM model (in Fig.
Revised: 1-Layer LSTM model (in Fig.

Original: 1) (Zaremba et al., 2014) :
Revised: 1) (Zaremba et al., 2014) :

Original: One layer LSTM model with 300 hidden states 2 is used to predict PTB dataset.
Revised: One layer LSTM model with 300 hidden states 2 is used to predict PTB dataset.

Original: We compress the a LSTM layer and an embedding layer of the pre-trained language model to draw the Figure 1.
Revised: We compress the a LSTM layer and an embedding layer of the pre-trained language model to draw the Figure 1.

Original: A scaling factor is extracted for each raw, e.g., for the (10000, 300) embedding layer, there exist 10,000 scaling factors as a result of quantization.
Revised: A scaling factor is extracted for each raw, e.g., for the (10000, 300) embedding layer, there exist 10,000 scaling factors as a result of quantization.

Original: AWD-LSTM (Merity et al., 2017) :
Revised: AWD-LSTM (Merity et al., 2017) :

Original: We use a 3-layer AWD-LSTM model.
Revised: We use a 3-layer AWD-LSTM model.

Original: The embedding size is 400 and the hidden vector size within LSTM layers is 1550.
Revised: The embedding size is 400 and the hidden vector size within LSTM layers is 1550.

Original: We train the original model during 500 epochs and then ﬁne-tune during additional 300 epochs 3 .
Revised: We train the original model during 500 epochs and then ﬁne-tune during additional 300 epochs 3 .

Original: We compress both models including embedding and softmax layers by our post-training quantization method.
Revised: We compress both models including embedding and softmax layers by our post-training quantization method.

Original: Dataset : To evaluate our quantization method, we use the test dataset of Penn Treebank (PTB) dataset (Marcus et al., 1993) for the LSTM language models.
Revised: Dataset : To evaluate our quantization method, we use the test dataset of Penn Treebank (PTB) dataset (Marcus et al., 1993) for the LSTM language models.

Original: To ﬁnd the quantization parameters with BO, we use valid dataset.
Revised: To ﬁnd the quantization parameters with BO, we use valid dataset.

Original: A. H UGGINGFACE L ANGUAGE M ODELS
Revised: A. H UGGINGFACE L ANGUAGE M ODELS

Original: To evaluate recently developed language models, we utilize the transformers library (PyTorch version) developed by huggingface (Wolf et al., 2019).
Revised: To evaluate recently developed language models, we utilize the transformers library (PyTorch version) developed by huggingface (Wolf et al., 2019).

Original: For all bert-based models, the last classiﬁcation layer and the sequence embedding layer are not quantized because the sizes of weights are relatively smaller than other weights.
Revised: For all bert-based models, the last classiﬁcation layer and the sequence embedding layer are not quantized because the sizes of weights are relatively smaller than other weights.

Original: BERT (Devlin et al., 2018) / DistilBERT (Sanh et al., 2019) :
Revised: BERT (Devlin et al., 2018) / DistilBERT (Sanh et al., 2019) :

Original: We ﬁne-tune the pre-trained BERTbase and DistilBERT-base models to evaluate our method.
Revised: We ﬁne-tune the pre-trained BERTbase and DistilBERT-base models to evaluate our method.

Original: BERT-base model consists of 12 encoder blocks with 768 hidden size and DistilBERT-base model consists of 6 encoder blocks with 768 hidden size.
Revised: BERT-base model consists of 12 encoder blocks with 768 hidden size and DistilBERT-base model consists of 6 encoder blocks with 768 hidden size.

Original: We follow ﬁne-tuning recipes found in the transformers repository 4 .
Revised: We follow ﬁne-tuning recipes found in the transformers repository 4 .

Original: For the MRPC and
Revised: For the MRPC and

Original: MNLI tasks, the initial learning rate is 2e-5 and the training epoch is 3.
Revised: MNLI tasks, the initial learning rate is 2e-5 and the training epoch is 3.

Original: For the SQUAD task, the initial learning rate is 3e-5 and the training epoch is 2.
Revised: For the SQUAD task, the initial learning rate is 3e-5 and the training epoch is 2.

Original: Longformer (Beltagy et al., 2020) :
Revised: Longformer (Beltagy et al., 2020) :

Original: To evaluate the Longformer model, we choose the pre-trained longformer-base model that has 12 blocks of Longformer encoder with 768 hidden size.
Revised: To evaluate the Longformer model, we choose the pre-trained longformer-base model that has 12 blocks of Longformer encoder with 768 hidden size.

Original: We ﬁne-tune the pre-trained model 5 for SQUAD v1.1 dataset with the same recipe.
Revised: We ﬁne-tune the pre-trained model 5 for SQUAD v1.1 dataset with the same recipe.

Original: Dataset : We use three language tasks: MRPC, MNLI and SQUAD(v1.1).
Revised: Dataset : We use three language tasks: MRPC, MNLI and SQUAD(v1.1).

Original: To search quantization parameters ( E , C and P ), we use a randomly sampled fraction of train dataset when the evaluation time is too time-consuming.
Revised: To search quantization parameters ( E , C and P ), we use a randomly sampled fraction of train dataset when the evaluation time is too time-consuming.

Original: For the MRPC task, we use the whole train dataset because the train dataset is small enough to use.
Revised: For the MRPC task, we use the whole train dataset because the train dataset is small enough to use.

Original: For the MNLI task, we use only 10% of train dataset.
Revised: For the MNLI task, we use only 10% of train dataset.

Original: For the SQUAD task, we use only 6.7% of train dataset for bayesian optimization while dev dataset is used for testing because there is no published test dataset.
Revised: For the SQUAD task, we use only 6.7% of train dataset for bayesian optimization while dev dataset is used for testing because there is no published test dataset.

Original: A. O PEN NMT T RANSFORMER
Revised: A. O PEN NMT T RANSFORMER

Original: We use the pre-trained transformer model for Neural Machine Translation task (Klein et al., 2018) 6 .
Revised: We use the pre-trained transformer model for Neural Machine Translation task (Klein et al., 2018) 6 .

Original: The transformer model consists of 6 encoder blocks, 6 decoder blocks and embedding layers.
Revised: The transformer model consists of 6 encoder blocks, 6 decoder blocks and embedding layers.

Original: Note that the embedding layers are not shared, e.g. embedding weights are not tied.
Revised: Note that the embedding layers are not shared, e.g. embedding weights are not tied.

Original: The vocabulary size is 50k and the size of the hidden vector is 512.
Revised: The vocabulary size is 50k and the size of the hidden vector is 512.

Original: Dataset : We evaluate the pre-trained model in a translation direction: English to German (en2de).
Revised: Dataset : We evaluate the pre-trained model in a translation direction: English to German (en2de).

Original: We use valid dataset in newstest2017 for BO processes and test dataset for the test evaluation.
Revised: We use valid dataset in newstest2017 for BO processes and test dataset for the test evaluation.

Original: All datasets are pre-processed by SentencePiece (Kudo & Richardson, 2018).
Revised: All datasets are pre-processed by SentencePiece (Kudo & Richardson, 2018).

Original: All the translation scores are BLEU scores by sacrebleu script (Post, 2018) as the beam size is 1.
Revised: All the translation scores are BLEU scores by sacrebleu script (Post, 2018) as the beam size is 1.

Original: detailed parameters
Revised: detailed parameters

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A. R ES N ET FOR I MAGE C LASSIFICATION
Revised: A. R ES N ET FOR I MAGE C LASSIFICATION

Original: We conduct experiments using ResNet32 on CIFAR

Revised: We conduct experiments using ResNet32 on CIFAR

Original: (Krizhevsky, 2009) and ResNet18 (He et al., 2016) on ImageNet (Russakovsky et al., 2015).
Revised: (Krizhevsky, 2009) and ResNet18 (He et al., 2016) on ImageNet (Russakovsky et al., 2015).

Original: For convolution tensors, α and B are computed for each channel.
Revised: For convolution tensors, α and B are computed for each channel.

Original: We maintain full-precision on the ﬁrst and last layers of ResNet models because those layers are very small while a lot of quantization bits are required (McDonnell, 2018).
Revised: We maintain full-precision on the ﬁrst and last layers of ResNet models because those layers are very small while a lot of quantization bits are required (McDonnell, 2018).

Original: For ResNet-18, we use ImageNet1K 7 training dataset, which is a small subset of ImageNet dataset, for fast hyper-parameter search, while test accuracy is still measured by the entire ImageNet test dataset.
Revised: For ResNet-18, we use ImageNet1K 7 training dataset, which is a small subset of ImageNet dataset, for fast hyper-parameter search, while test accuracy is still measured by the entire ImageNet test dataset.

Original: To obtain the same accuracy for the same set of hyper-parameters, the training dataset is not randomly manipulated (e.g., by cropping and ﬂipping).
Revised: To obtain the same accuracy for the same set of hyper-parameters, the training dataset is not randomly manipulated (e.g., by cropping and ﬂipping).

Original: B B AYESIAN O PTIMIZATION FOR W EIGHTED Q UANTIZATION
Revised: B B AYESIAN O PTIMIZATION FOR W EIGHTED Q UANTIZATION

Original: BO is one of automated machine learning (AutoML) techniques to search optimal hyper-parameters for target networks.
Revised: BO is one of automated machine learning (AutoML) techniques to search optimal hyper-parameters for target networks.

Original: Given a black box function f , BO aims to ﬁnd the optimal x ∗ to be arg max x f ( x ) .
Revised: Given a black box function f , BO aims to ﬁnd the optimal x ∗ to be arg max x f ( x ) .

Original: Suppose observations are described as y =[
Revised: Suppose observations are described as y =[

Original: f ( x 1 ) , f ( x 2 ) , . . .
Revised: f ( x 1 ) , f ( x 2 ) , . . .

Original: , f ( x n )]
Revised: , f ( x n )]

Original: T and y ∗ is an output of any unobserved x ∗ , then under the assumption that f ( x ) is drawn from Gaussian Process , the distribution of y ∗ | y follows N ( K ∗ K − 1 y , K ∗∗ − K ∗ K − 1 K T ∗ ) , where
Revised: T and y ∗ is an output of any unobserved x ∗ , then under the assumption that f ( x ) is drawn from Gaussian Process , the distribution of y ∗ | y follows N ( K ∗ K − 1 y , K ∗∗ − K ∗ K − 1 K T ∗ ) , where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The kernel function k
Revised: The kernel function k

Original: ( x , x (cid:48) ) is one of the hyper-parameter for BO and measures a similarity between x and x (cid:48)
Revised: ( x , x (cid:48) ) is one of the hyper-parameter for BO and measures a similarity between x and x (cid:48)

Original: (i.e., the output is high when they are close).
Revised: (i.e., the output is high when they are close).

Original: There are various kernel functions (Rasmussen & Williams, 2006), and we use squared-exponential kernel that is one of the popular choices for regression (Ebden, 2015).
Revised: There are various kernel functions (Rasmussen & Williams, 2006), and we use squared-exponential kernel that is one of the popular choices for regression (Ebden, 2015).

Original: To identify which of the unobserved data to be taken as x n +1 , the acquisition function needs to be speciﬁed.
Revised: To identify which of the unobserved data to be taken as x n +1 , the acquisition function needs to be speciﬁed.

Original: In general, the expected improvement acquisition function a EI (Lizotte, 2008) (see Eq.
Revised: In general, the expected improvement acquisition function a EI (Lizotte, 2008) (see Eq.

Original: (11)) is most commonly used and is selected for our experiments.
Revised: (11)) is most commonly used and is selected for our experiments.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The parameter ξ is the trade-off factor between exploitation and exploration .
Revised: The parameter ξ is the trade-off factor between exploitation and exploration .

Original: In our experiments, we set ξ to 0.2, which implies that exploitation has more inﬂuence on determining x n +1 .
Revised: In our experiments, we set ξ to 0.2, which implies that exploitation has more inﬂuence on determining x n +1 .

Original: After computing a EI for unobserved random sampled data x ∗ s, x n +1 is computed as arg max x ∗ a EI ( x ∗ | y ) .
Revised: After computing a EI for unobserved random sampled data x ∗ s, x n +1 is computed as arg max x ∗ a EI ( x ∗ | y ) .

Original: Further details of BO can be found in (Brochu et al., 2010; Nogueira, 2014–; Snoek et al., 2012).
Revised: Further details of BO can be found in (Brochu et al., 2010; Nogueira, 2014–; Snoek et al., 2012).

Original: [Figure]
Revised: [Figure]

Original: T e s t
Revised: T e s t

Original: A CC .
Revised: A CC .

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: Model

Original: [Table]
Revised: [Table]

Original: wise
Revised: wise

Original: Layer-wise
Revised: Layer-wise

Original: [Table]
Revised: [Table]

Original: Treebank valid test
Revised: Treebank valid test

Original: [Table]
Revised: [Table]

Original: train test
Revised: train test

Original: MNLI 10% of train test
Revised: MNLI 10% of train test

Original: SQUAD v1.
Revised: SQUAD v1.

Original: [Table]
Revised: [Table]

Original: 7% of
Revised: 7% of

Original: [Table]
Revised: [Table]

Original: dev  newstest valid test
Revised: dev  newstest valid test

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: CIFAR train test
Revised: CIFAR train test

Original: ImageNet train of ImageNet1K test
Revised: ImageNet train of ImageNet1K test

Original: C R ESULTS ON L ANGUAGE MODELS
Revised: C R ESULTS ON L ANGUAGE MODELS

Original: =
Revised: =

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Set Set
Revised: Set Set

Original: Set Set
Revised: Set Set

Original: Set Set
Revised: Set Set

Original: Set Set
Revised: Set Set

Original: E 1.
Revised: E 1.

Original: C 0.
Revised: C 0.

Original: C =1.0 (default)
Revised: C =1.0 (default)

Original: Set Set
Revised: Set Set

Original: Set Set
Revised: Set Set

Original: Set Set
Revised: Set Set

Original: Set Set
Revised: Set Set

Original: E 0.
Revised: E 0.

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: Full-Precision (ours) [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] by Table 5)
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Fig.

Original: [Table]
Revised: [Table]

Original: Manual (by Table 5)

Original: [Table]
Revised: [Table]

Original: Fig.

Original: [Table]
Revised: [Table]

Original: Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Fig. 60 MRPC 0. 60 MNLI 0. 60 SQUAD 0.
Revised: [Table]

Original: E C P [Table]
Revised: [Table]

Original: Manual (by Table 5)

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] Fig.
Revised: [Table]

Original: MNLI

Original: Manual (by Table 5)
Revised: Manual (by Table 5)

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: SQUAD
Revised: [Table]

Original: Manual (by Table 5)
Revised: Manual (by Table 5)

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 30 MRPC

Original: [Figure]
Revised: [Table]

Original: [Figure] [Figure] [Figure] 30 MNLI 0.
Revised: (by Table 5)

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: (by Table 5)

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: [Figure]

Original: C

Original: P [Figure] [Figure] R e l a t i v e
Revised: [Table]

Original: D e g r a d a t i o n [Figure] For the same rate, the Longformer presents sharper score degradation that partly explains the difﬁculty of improving test scores by our proposed weighted quantization method compared to the conventional Alternating quantization. C. AWD-LSTM ON PTB DATASET
Revised: (by Table 5)

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] val
Revised: MNLI

Original: [Table]
Revised: [Table]

Original: test ACC. Parameters( E [Table] C
Revised: (by Table 5)

Original: [Table]
Revised: [Table]

Original: P [Table] (by Table 5)- 64. [Table]
Revised: Manual (by Table 5)

Original: [Table]
Revised: [Table]

Original: Fig. [Table] (by Table 5) - 59. Model
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: BO 61.

Original: Layer

Original: [Table]
Revised: [Table]

Original: BO 60.

Original: Fig. [Table]
Revised: [Figure]

Original: Manual (by Table 5) - 57.

Original: [Table]
Revised: [Figure]

Original: BO 59.
Revised: For the same pruning rate, the Longformer presents sharper score degradation that partly explains the difﬁculty of improving test scores by our proposed weighted quantization method compared to the conventional Alternating quantization. C. AWD-LSTM ON PTB DATASET [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: Layer
Revised: [Table] [Table] [Table] [Table]

Revised: Fig.

Original: [Table]
Revised: [Table]

Revised: Table

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: Fig
Revised: Fig.

Original: [Table]
Revised: [Table]

Revised: Table 5)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: C. T RANSFORMER
Revised: C. T RANSFORMER

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: E
Revised: E

Original: C
Revised: C

Original: P
Revised: P

Original: [Figure]
Revised: [Figure]

Original: D R ESULTS ON R ES N ET MODELS
Revised: D R ESULTS ON R ES N ET MODELS

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: is the number of quantization bits.

Original: [Table]
Revised: [Table]

Original: Method
Revised: Method

Original: E
Revised: E

Original: C
Revised: C

Original: P q = q = q = q =
Revised: P q = q = q = q =

Original: Greedy
Revised: Greedy

Original: Original 12.
Revised: No Importance 12.

Original: Alter.
Revised: Alter.

Original: Original 84.
Revised: No Importance 84.

Original: Full-Precision 92.
Revised: Full-Precision 92.

Original: [Table]
Revised: [Table]

Original: (ImageNet)
Revised: (ImageNet)

Original: [Table]
Revised: [Table]

Original: ResNet- (CIFAR-10)
Revised: ResNet- (CIFAR-10)

Original: [Table] [Figure]
Revised: [Table] [Figure]