Original: T RAINING WITH Q UANTIZATION N OISE FOR E XTREME M ODEL C OMPRESSION
Revised: T RAINING WITH Q UANTIZATION N OISE FOR E XTREME M ODEL C OMPRESSION

Revised: Pierre Stock ∗†

Original: Anonymous authors
Revised: Facebook AI Research, Inria Angela Fan ∗ Facebook AI Research, LORIA Benjamin Graham

Revised: Facebook AI Research

Original: Paper under double-blind review
Revised: Edouard Grave Facebook AI Research Rémi Gribonval † Inria

Revised: Hervé Jégou

Revised: Facebook AI Research

Revised: Armand Joulin

Revised: Facebook AI Research

Original: A BSTRACT
Revised: A BSTRACT

Original: We tackle the problem of producing compact models, maximizing their accuracy for a given model size.
Revised: We tackle the problem of producing compact models, maximizing their accuracy for a given model size.

Original: A standard solution is to train networks with Quantization
Revised: A standard solution is to train networks with Quantization

Original: Aware Training (Jacob et al., 2018), where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator (Bengio et al., 2013).
Revised: Aware Training (Jacob et al., 2018), where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator (Bengio et al., 2013).

Original: In this paper, we extend this approach to work beyond int8 ﬁxedpoint quantization with extreme compression methods where the approximations introduced by STE are severe, such as Product Quantization.
Revised: In this paper, we extend this approach to work beyond int8 ﬁxedpoint quantization with extreme compression methods where the approximations introduced by STE are severe, such as Product Quantization.

Original: Our proposal is to only quantize a different random subset of weights during each forward, allowing for unbiased gradients to ﬂow through the other weights.
Revised: Our proposal is to only quantize a different random subset of weights during each forward, allowing for unbiased gradients to ﬂow through the other weights.

Original: Controlling the amount of noise and its form allows for extreme compression rates while maintaining the performance of the original model.
Revised: Controlling the amount of noise and its form allows for extreme compression rates while maintaining the performance of the original model.

Original: As a result we establish new state-of-the-art compromises between accuracy and model size both in natural language processing and image classiﬁcation.
Revised: As a result we establish new state-of-the-art compromises between accuracy and model size both in natural language processing and image classiﬁcation.

Original: For example, applying our method to state-of-the-art
Revised: For example, applying our method to state-of-the-art

Original: Transformer and ConvNet architectures, we can achieve 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB and 80.0% top-1 accuracy on ImageNet by compressing an EfﬁcientNet-B3 to 3.3 MB.
Revised: Transformer and ConvNet architectures, we can achieve 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB and 80.0% top-1 accuracy on ImageNet by compressing an EfﬁcientNet-B3 to 3.3 MB.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many of the best performing neural network architectures in real-world applications have a large number of parameters.
Revised: Many of the best performing neural network architectures in real-world applications have a large number of parameters.

Original: For example, the current standard machine translation architecture, Transformer (Vaswani et al., 2017), has layers that contain millions of parameters.
Revised: For example, the current standard machine translation architecture, Transformer (Vaswani et al., 2017), has layers that contain millions of parameters.

Original: Even models that are designed to jointly optimize the performance and the parameter efﬁciency, such as EfﬁcientNets (Tan & Le, 2019), still require dozens to hundreds of megabytes, which limits their applications to domains like robotics or virtual assistants.
Revised: Even models that are designed to jointly optimize the performance and the parameter efﬁciency, such as EfﬁcientNets (Tan & Le, 2019), still require dozens to hundreds of megabytes, which limits their applications to domains like robotics or virtual assistants.

Original: Model compression schemes reduce the memory footprint of overparametrized models.
Revised: Model compression schemes reduce the memory footprint of overparametrized models.

Original: Pruning (LeCun et al., 1990) and distillation (Hinton et al., 2015) remove parameters by reducing the number of network weights.
Revised: Pruning (LeCun et al., 1990) and distillation (Hinton et al., 2015) remove parameters by reducing the number of network weights.

Original: In contrast, quantization focuses on reducing the bits per weight.
Revised: In contrast, quantization focuses on reducing the bits per weight.

Original: This makes quantization particularly interesting when compressing models that have already been carefully optimized in terms of network architecture.
Revised: This makes quantization particularly interesting when compressing models that have already been carefully optimized in terms of network architecture.

Original: Whereas deleting weights or whole hidden units will inevitably lead to a drop in performance, we demonstrate that quantizing the weights can be performed with little to no loss in accuracy.
Revised: Whereas deleting weights or whole hidden units will inevitably lead to a drop in performance, we demonstrate that quantizing the weights can be performed with little to no loss in accuracy.

Original: Popular postprocessing quantization methods, like scalar quantization, replace the ﬂoating-point weights of a trained network by a lower-precision representation, like ﬁxed-width integers (Vanhoucke et al., 2011).
Revised: Popular postprocessing quantization methods, like scalar quantization, replace the ﬂoating-point weights of a trained network by a lower-precision representation, like ﬁxed-width integers (Vanhoucke et al., 2011).

Original: These approaches achieve a good compression rate with the additional beneﬁt of accelerating inference on supporting hardware.
Revised: These approaches achieve a good compression rate with the additional beneﬁt of accelerating inference on supporting hardware.

Original: However, the errors made by these approximations accumulate in the computations operated during the forward pass, inducing a signiﬁcant drop in performance (Stock et al., 2019).
Revised: However, the errors made by these approximations accumulate in the computations operated during the forward pass, inducing a signiﬁcant drop in performance (Stock et al., 2019).

Original: A solution to address this drifting effect is to directly quantize the network during training.
Revised: A solution to address this drifting effect is to directly quantize the network during training.

Original: This raises two challenges.
Revised: This raises two challenges.

Original: First, the discretization operators have a null gradient — the derivative with respect to the input is zero almost everywhere.
Revised: First, the discretization operators have a null gradient — the derivative with respect to the input is zero almost everywhere.

Original: This requires special workarounds to train a network with these operators.
Revised: This requires special workarounds to train a network with these operators.

Original: The second challenge that often comes with these workarounds is the discrepancy that appears between the train and test functions implemented by the network.
Revised: The second challenge that often comes with these workarounds is the discrepancy that appears between the train and test functions implemented by the network.

Original: Quantization Aware
Revised: Quantization Aware

Original: Training (QAT) (Jacob et al., 2018) resolves these issues by quantizing all the weights during the forward and using a straight through estimator (STE) (Bengio et al., 2013) to compute the gradient.
Revised: Training (QAT) (Jacob et al., 2018) resolves these issues by quantizing all the weights during the forward and using a straight through estimator (STE) (Bengio et al., 2013) to compute the gradient.

Original: This works when the error introduced by STE is small, like with int8 quantization, but does not sufﬁce in compression regimes where the approximation made by the compression is more severe.
Revised: This works when the error introduced by STE is small, like with int8 quantization, but does not sufﬁce in compression regimes where the approximation made by the compression is more severe.

Original: In this work, we show that quantizing only a subset of weights instead of the entire network during training is more stable for high compression schemes.
Revised: In this work, we show that quantizing only a subset of weights instead of the entire network during training is more stable for high compression schemes.

Original: Indeed, by quantizing only a random fraction of the network at each forward, most the weights are updated with unbiased gradients.
Revised: Indeed, by quantizing only a random fraction of the network at each forward, most the weights are updated with unbiased gradients.

Original: Interestingly, we show that our method can employ a simpler quantization scheme during the training.
Revised: Interestingly, we show that our method can employ a simpler quantization scheme during the training.

Original: This is particularly useful for quantizers with trainable parameters, such as Product Quantizer (PQ), for which our quantization proxy is not parametrized.
Revised: This is particularly useful for quantizers with trainable parameters, such as Product Quantizer (PQ), for which our quantization proxy is not parametrized.

Original: Our approach simply applies a quantization noise, called
Revised: Our approach simply applies a quantization noise, called

Original: Quant-Noise, to a random subset of the weights, see Figure 1.
Revised: Quant-Noise, to a random subset of the weights, see Figure 1.

Original: We observe that this makes a network resilient to various types of discretization methods: it signiﬁcantly improves the accuracy associated with (a) low precision representation of weights like int8 ; and (b) state-of-the-art PQ.
Revised: We observe that this makes a network resilient to various types of discretization methods: it signiﬁcantly improves the accuracy associated with (a) low precision representation of weights like int8 ; and (b) state-of-the-art PQ.

Original: Further, we demonstrate that Quant-Noise can be applied to existing trained networks as a post-processing step, to improve the performance network after quantization.
Revised: Further, we demonstrate that Quant-Noise can be applied to existing trained networks as a post-processing step, to improve the performance network after quantization.

Original: In summary, this paper makes the following contributions:
Revised: In summary, this paper makes the following contributions:

Original: • We introduce the Quant-Noise technique to learn networks that are more resilient to a variety of quantization methods such as int4 , int8 , and PQ; • Adding Quant-Noise to PQ leads to new state-of-the-art trade-offs between accuracy and model size.
Revised: • We introduce the Quant-Noise technique to learn networks that are more resilient to a variety of quantization methods such as int4 , int8 , and PQ; • Adding Quant-Noise to PQ leads to new state-of-the-art trade-offs between accuracy and model size.

Original: For instance, for natural language processing (NLP), we reach 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB.
Revised: For instance, for natural language processing (NLP), we reach 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB.

Original: Similarly for computer vision, we report 80.0% top-1 accuracy on ImageNet by compressing an EfﬁcientNet-B3 to 3.3 MB; • By combining PQ and int8 to quantize weights and activations for networks trained with Quant-Noise, we obtain extreme compression with ﬁxed-precision computation and achieve 79.8% top-1 accuracy on ImageNet and 21 .
Revised: Similarly for computer vision, we report 80.0% top-1 accuracy on ImageNet by compressing an EfﬁcientNet-B3 to 3.3 MB; • By combining PQ and int8 to quantize weights and activations for networks trained with Quant-Noise, we obtain extreme compression with ﬁxed-precision computation and achieve 79.8% top-1 accuracy on ImageNet and 21 .

Original: 1 perplexity on WikiText-103.
Revised: 1 perplexity on WikiText-103.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Model compression.
Revised: Model compression.

Original: Many compression methods focus on efﬁcient parameterization, via weight pruning (LeCun et al., 1990;
Revised: Many compression methods focus on efﬁcient parameterization, via weight pruning (LeCun et al., 1990;

Original: Li et al., 2016; Huang et al., 2018; Mittal et al., 2018), weight sharing (Dehghani et al., 2018; Turc et al., 2019; Lan et al., 2019) or with dedicated architectures (Tan & Le, 2019; Zhang et al., 2017; Howard et al., 2019).
Revised: Li et al., 2016; Huang et al., 2018; Mittal et al., 2018), weight sharing (Dehghani et al., 2018; Turc et al., 2019; Lan et al., 2019) or with dedicated architectures (Tan & Le, 2019; Zhang et al., 2017; Howard et al., 2019).

Original: Weight pruning is implemented during training (Louizos et al., 2017) or as a ﬁne-tuning post-processing step (Han et al., 2015; 2016).
Revised: Weight pruning is implemented during training (Louizos et al., 2017) or as a ﬁne-tuning post-processing step (Han et al., 2015; 2016).

Original: Many pruning methods are unstructured, i.e., remove individual weights (LeCun et al., 1990; Molchanov et al., 2017).
Revised: Many pruning methods are unstructured, i.e., remove individual weights (LeCun et al., 1990; Molchanov et al., 2017).

Original: On the other hand, structured pruning methods follow the structure of the weights to reduce both the memory footprint and the inference time of a model (Li et al., 2016;
Revised: On the other hand, structured pruning methods follow the structure of the weights to reduce both the memory footprint and the inference time of a model (Li et al., 2016;

Original: Luo et al., 2017;
Revised: Luo et al., 2017;

Original: Fan et al., 2019).
Revised: Fan et al., 2019).

Original: We refer the reader to Liu et al.
Revised: We refer the reader to Liu et al.

Original: (2018) for a review of different pruning strategies.
Revised: (2018) for a review of different pruning strategies.

Original: Other authors have worked on lightweight architectures, by modifying existing models (Zhang et al., 2018; Wu et al., 2019; Sukhbaatar et al., 2019a) or developing new networks, such as MobileNet (Howard et al., 2019), ShufﬂeNet (Zhang et al., 2017), and EfﬁcientNet (Tan & Le, 2019) in vision.
Revised: Other authors have worked on lightweight architectures, by modifying existing models (Zhang et al., 2018; Wu et al., 2019; Sukhbaatar et al., 2019a) or developing new networks, such as MobileNet (Howard et al., 2019), ShufﬂeNet (Zhang et al., 2017), and EfﬁcientNet (Tan & Le, 2019) in vision.

Original: Finally, knowledge distillation (Hinton et al., 2015) has been applied to sentence representation (Turc et al., 2019; Sanh et al., 2019a;
Revised: Finally, knowledge distillation (Hinton et al., 2015) has been applied to sentence representation (Turc et al., 2019; Sanh et al., 2019a;

Original: Sun et al., 2019;
Revised: Sun et al., 2019;

Original: Zhao et al., 2019; Jiao et al., 2019), to reduce the size of a BERT model (Devlin et al.
Revised: Zhao et al., 2019; Jiao et al., 2019), to reduce the size of a BERT model (Devlin et al., 2018).

Original: Quantization.
Revised: Quantization.

Original: There are extensive studies of scalar quantization to train networks with lowprecision weightsand activations (Courbariaux et al., 2015; Courbariaux & Bengio, 2016; Rastegari et al., 2016; McDonnell, 2018).
Revised: There are extensive studies of scalar quantization to train networks with lowprecision weights and activations (Courbariaux et al., 2015; Courbariaux & Bengio, 2016; Rastegari et al., 2016; McDonnell, 2018).

Original: These methods beneﬁt from specialized hardware to also improve the runtime during inference (Vanhoucke et al., 2011).
Revised: These methods beneﬁt from specialized hardware to also improve the runtime during inference (Vanhoucke et al., 2011).

Original: Other quantization methods such as Vector
Revised: Other quantization methods such as Vector

Original: Quantization (VQ) and PQ (Jegou et al., 2011) quantize blocks of weights simultaneously to achieve higher compression rate (Stock et al., 2019;
Revised: Quantization (VQ) and PQ (Jegou et al., 2011) quantize blocks of weights simultaneously to achieve higher compression rate (Stock et al., 2019;

Original: Gong et al., 2014; Joulin et al., 2016; Carreira-Perpiñán & Idelbayev, 2017).
Revised: Gong et al., 2014; Joulin et al., 2016; Carreira-Perpiñán & Idelbayev, 2017).

Original: Closer to our work, several works have focused at simulatenously training and quantizing a network (Jacob et al., 2018; Krishnamoorthi, 2018; Gupta et al., 2015; Dong et al., 2019).
Revised: Closer to our work, several works have focused at simultaneously training and quantizing a network (Jacob et al., 2018; Krishnamoorthi, 2018; Gupta et al., 2015; Dong et al., 2019).

Original: Gupta et al.
Revised: Gupta et al.

Original: (2015) assigns weights to a quantized bin stochastically which is speciﬁc to scalar quantization, but allows training with ﬁxed point arithmetic.
Revised: (2015) assigns weights to a quantized bin stochastically which is speciﬁc to scalar quantization, but allows training with ﬁxed point arithmetic.

Original: Finally, our method can be interpreted as a form of Bayesian compression (Louizos et al., 2017), using the Bayesian interpretation of
Revised: Finally, our method can be interpreted as a form of Bayesian compression (Louizos et al., 2017), using the Bayesian interpretation of

Original: Dropout (Gal & Ghahramani, 2016).
Revised: Dropout (Gal & Ghahramani, 2016).

Original: As opposed to their work, we select our noise to match the weight transformation of a target quantization methods without restricting it to a scale mixture prior.
Revised: As opposed to their work, we select our noise to match the weight transformation of a target quantization methods without restricting it to a scale mixture prior.

Original:  Q UANTIZING N EURAL N ETWORKS
Revised:  Q UANTIZING N EURAL N ETWORKS

Original: In this section, we present the principles of quantization, several standard quantization methods, and describe how to combine scalar and product quantization.
Revised: In this section, we present the principles of quantization, several standard quantization methods, and describe how to combine scalar and product quantization.

Original: For clarity, we focus on the case of a ﬁxed real matrix W ∈ R n × p .
Revised: For clarity, we focus on the case of a ﬁxed real matrix W ∈ R n × p .

Original: We suppose that this matrix is split into m × q blocks b kl :
Revised: We suppose that this matrix is split into m × q blocks b kl :

Original: [Equation]
Revised: [Equation]

Original: where the nature of these blocks is determined by the quantization method.
Revised: where the nature of these blocks is determined by the quantization method.

Original: A codebook is a set of K vectors, i.e., C = { c [1] , . . .
Revised: A codebook is a set of K vectors, i.e., C = { c [1] , . . .

Original: , c [ K ] } .
Revised: , c [ K ] } .

Original: Quantization methods compress the matrix W by assigning to each block b kl an index that points to a codeword c in a codebook C , and storing the codebook C and the resulting indices (as the entries I kl of an index matrix I ) instead of the real weights.
Revised: Quantization methods compress the matrix W by assigning to each block b kl an index that points to a codeword c in a codebook C , and storing the codebook C and the resulting indices (as the entries I kl of an index matrix I ) instead of the real weights.

Original: During the inference, they reconstruct an approximation (cid:99) W of the original matrix W such that (cid:98) b kl =
Revised: During the inference, they reconstruct an approximation (cid:99) W of the original matrix W such that (cid:98) b kl =

Original: c
Revised: c

Original: [ I kl ] .
Revised: [ I kl ] .

Original: We distinguish scalar quantization, such as int8 , where each block b kl consists of a single weight, from vector quantization, where several weights are quantized jointly.
Revised: We distinguish scalar quantization, such as int8 , where each block b kl consists of a single weight, from vector quantization, where several weights are quantized jointly.

Original: 3. F IXED - POINT S CALAR Q UANTIZATION
Revised: 3. F IXED - POINT S CALAR Q UANTIZATION

Original: Fixed-point scalar quantization methods replace ﬂoating-point number representations by lowprecision ﬁxed-point representations.
Revised: Fixed-point scalar quantization methods replace ﬂoating-point number representations by lowprecision ﬁxed-point representations.

Original: They simultaneously reduce a model’s memory footprint and accelerate inference by using ﬁxed-point arithmetic on supporting hardware.
Revised: They simultaneously reduce a model’s memory footprint and accelerate inference by using ﬁxed-point arithmetic on supporting hardware.

Original: Fixed-point scalar quantization operates on blocks that represent a single weight, i.e., b kl = W kl .
Revised: Fixed-point scalar quantization operates on blocks that represent a single weight, i.e., b kl = W kl .

Original: Floating-point weights are replaced by N bit ﬁxed-point numbers (Gupta et al., 2015), with the extreme case of binarization where N = 1 (Courbariaux et al., 2015).
Revised: Floating-point weights are replaced by N bit ﬁxed-point numbers (Gupta et al., 2015), with the extreme case of binarization where N = 1 (Courbariaux et al., 2015).

Original: More precisely, the weights are rounded to one of 2 N possible codewords.
Revised: More precisely, the weights are rounded to one of 2 N possible codewords.

Original: These codewords correspond to bins evenly spaced by a scale factor s and shifted by a bias z .
Revised: These codewords correspond to bins evenly spaced by a scale factor s and shifted by a bias z .

Original: Each weight W kl is mapped to its nearest codeword c by successively quantizing with z (cid:55)→ round ( W kl /s + z ) and dequantizing with the inverse operation:
Revised: Each weight W kl is mapped to its nearest codeword c by successively quantizing with z (cid:55)→ round ( W kl /s + z ) and dequantizing with the inverse operation:

Original: [Equation]
Revised: [Equation]

Original: where we compute the scale and bias as:
Revised: where we compute the scale and bias as:

Original: [Equation]
Revised: [Equation]

Original: We focus on this uniform rounding scheme instead of other non-uniform schemes (Choi et al., 2018;
Revised: We focus on this uniform rounding scheme instead of other non-uniform schemes (Choi et al., 2018;

Original: Li et al., 2019), because it allows for ﬁxed-point arithmetic with implementations in PyTorch and
Revised: Li et al., 2019), because it allows for ﬁxed-point arithmetic with implementations in PyTorch and

Original: Tensorﬂow (see Appendix).
Revised: Tensorﬂow (see Appendix).

Original: The compression rate is × 32 /N .
Revised: The compression rate is × 32 /N .

Original: The activations are also rounded to
Revised: The activations are also rounded to

Original: N -bit ﬁxed-point numbers.
Revised: N -bit ﬁxed-point numbers.

Original: With int8 for instance, this leads to × 2 to × 4 faster inference on dedicated hardware.
Revised: With int8 for instance, this leads to × 2 to × 4 faster inference on dedicated hardware.

Original: In this work, we consider both int4 and int8 quantization.
Revised: In this work, we consider both int4 and int8 quantization.

Original: 3. P RODUCT Q UANTIZATION
Revised: 3. P RODUCT Q UANTIZATION

Original: Several quantization methods work on groups of weights, such as vectors, to beneﬁt from the correlation induced by the structure of the network.
Revised: Several quantization methods work on groups of weights, such as vectors, to beneﬁt from the correlation induced by the structure of the network.

Original: In this work, we focus on Product Quantization for its good performance at extreme compression ratio (Stock et al., 2019).
Revised: In this work, we focus on Product Quantization for its good performance at extreme compression ratio (Stock et al., 2019).

Original: Traditional PQ.
Revised: Traditional PQ.

Original: In vector quantization methods, the blocks are predeﬁned groups of weights instead of single weights.
Revised: In vector quantization methods, the blocks are predeﬁned groups of weights instead of single weights.

Original: The codewords are groups of values, and the index matrix I maps groups of weights from the matrix W to these codewords.
Revised: The codewords are groups of values, and the index matrix I maps groups of weights from the matrix W to these codewords.

Original: In this section, we present the Product Quantization framework as it generalizes both scalar and vector quantization.
Revised: In this section, we present the Product Quantization framework as it generalizes both scalar and vector quantization.

Original: We consider the case where we apply
Revised: We consider the case where we apply

Original: PQ to the columns of W and thus assume that q = p .
Revised: PQ to the columns of W and thus assume that q = p .

Original: Traditional vector quantization techniques split the matrix W into its p columns and learn a codebook on the resulting p vectors.
Revised: Traditional vector quantization techniques split the matrix W into its p columns and learn a codebook on the resulting p vectors.

Original: Instead, Product Quantization splits each column into m subvectors and learns the same codebook for each of the resulting m × p subvectors.
Revised: Instead, Product Quantization splits each column into m subvectors and learns the same codebook for each of the resulting m × p subvectors.

Original: Each quantized vector is subsequently obtained by assigning its subvectors to the nearest codeword in the codebook.
Revised: Each quantized vector is subsequently obtained by assigning its subvectors to the nearest codeword in the codebook.

Original: Learning the codebook is traditionally done using k -means with a ﬁxed number K of centroids, typically
Revised: Learning the codebook is traditionally done using k -means with a ﬁxed number K of centroids, typically

Original: K = 256 to store the index matrix I using int8 .
Revised: K = 256 to store the index matrix I using int8 .

Original: Thus, the objective function is written as:
Revised: Thus, the objective function is written as:

Original: [Equation]
Revised: [Equation]

Original: PQ shares representations between subvectors, which allows for higher compression rates than intN .
Revised: PQ shares representations between subvectors, which allows for higher compression rates than intN .

Original: Iterative PQ.
Revised: Iterative PQ.

Original: When quantizing a full network rather than a single matrix, extreme compression with PQ induces a quantization drift as reconstruction error accumulates (Stock et al., 2019).
Revised: When quantizing a full network rather than a single matrix, extreme compression with PQ induces a quantization drift as reconstruction error accumulates (Stock et al., 2019).

Original: Indeed, subsequent layers take as input the output of preceding layers, which are modiﬁed by the quantization of the preceding layers.
Revised: Indeed, subsequent layers take as input the output of preceding layers, which are modiﬁed by the quantization of the preceding layers.

Original: This creates a drift in the network activations, resulting in large losses of performance.
Revised: This creates a drift in the network activations, resulting in large losses of performance.

Original: A solution proposed by Stock et al.
Revised: A solution proposed by Stock et al.

Original: (2019), which we call iterative PQ (iPQ), is to quantize layers sequentially from the lowest to the highest, and ﬁnetune the upper layers as the lower layers are quantized, under the supervision of the uncompressed (teacher) model.
Revised: (2019), which we call iterative PQ (iPQ), is to quantize layers sequentially from the lowest to the highest, and ﬁnetune the upper layers as the lower layers are quantized, under the supervision of the uncompressed (teacher) model.

Original: Codewords of each layer are ﬁnetuned by averaging the gradients of their assigned elements with gradient steps:
Revised: Codewords of each layer are ﬁnetuned by averaging the gradients of their assigned elements with gradient steps:

Original: [Equation]
Revised: [Equation]

Original: where J c = { ( k, l ) | c
Revised: where J c = { ( k, l ) | c

Original: [ I kl ]
Revised: [ I kl ]

Original: = c } , L is the loss function and η > 0 is a learning rate.
Revised: = c } , L is the loss function and η > 0 is a learning rate.

Original: This adapts the upper layers to the drift appearing in their inputs, reducing the impact of the quantization approximation on the overall performance.
Revised: This adapts the upper layers to the drift appearing in their inputs, reducing the impact of the quantization approximation on the overall performance.

Original: 3. C OMBINING F IXED -P OINT WITH P RODUCT Q UANTIZATION
Revised: 3. C OMBINING F IXED -P OINT WITH P RODUCT Q UANTIZATION

Original: Fixed-point quantization and Product Quantization are often regarded as competing choices, but can be advantageously combined.
Revised: Fixed-point quantization and Product Quantization are often regarded as competing choices, but can be advantageously combined.

Original: Indeed, PQ/iPQ compresses the network by replacing vectors of weights by their assigned centroids, but these centroids are in ﬂoating-point precision.
Revised: Indeed, PQ/iPQ compresses the network by replacing vectors of weights by their assigned centroids, but these centroids are in ﬂoating-point precision.

Original: Fixed-point quantization compresses both activations and weights to ﬁxed-point representations.
Revised: Fixed-point quantization compresses both activations and weights to ﬁxed-point representations.

Original: Combining both approaches means that the vectors of weights are mapped to centroids that are compressed to ﬁxed-point representations, along with the activations.
Revised: Combining both approaches means that the vectors of weights are mapped to centroids that are compressed to ﬁxed-point representations, along with the activations.

Original: This beneﬁts from the extreme compression ratio of iPQ and the ﬁnite-precision arithmetics of intN quantization.
Revised: This beneﬁts from the extreme compression ratio of iPQ and the ﬁnite-precision arithmetics of intN quantization.

Original: More precisely, for a given matrix, we store the int8 representation of the K centroids of dimension d along with the log 2 K representations of the centroid assignments of the m × p subvectors.
Revised: More precisely, for a given matrix, we store the int8 representation of the K centroids of dimension d along with the log 2 K representations of the centroid assignments of the m × p subvectors.

Original: The int8 representation of the centroids is obtained with Eq.
Revised: The int8 representation of the centroids is obtained with Eq.

Original: The overall storage of the matrix and activations during a forward pass with batch size 1 (recalling that the input dimension is n) writes
Revised: The overall storage of the matrix and activations during a forward pass with batch size 1 (recalling that the input dimension is n) writes

Original: [Equation]
Revised: [Equation]

Original: In particular, when K = 256 , the centroid assignments are also stored in int8 , which means that every value required for a forward pass is stored in an int8 format.
Revised: In particular, when K = 256 , the centroid assignments are also stored in int8 , which means that every value required for a forward pass is stored in an int8 format.

Original: We divide by 4 the float overhead of storing the centroids, although the storage requirement associated with the centroids is small compared to the cost of indexing the subvectors for standard networks.
Revised: We divide by 4 the float overhead of storing the centroids, although the storage requirement associated with the centroids is small compared to the cost of indexing the subvectors for standard networks.

Original: In contrast to iPQ alone where we only quantize the weights, we also quantize the activations using int8 .
Revised: In contrast to iPQ alone where we only quantize the weights, we also quantize the activations using int8 .

Original: We evaluate this approach on both natural language processing and computer vision tasks in Section 5.
Revised: We evaluate this approach on both natural language processing and computer vision tasks in Section 5.

Original:  M ETHOD
Revised:  M ETHOD

Original: Deep networks are not exposed to the noise caused by the quantization drift during training, leading to suboptimal performance.
Revised: Deep networks are not exposed to the noise caused by the quantization drift during training, leading to suboptimal performance.

Original: A solution to make the network robust to quantization is to introduce it during training.
Revised: A solution to make the network robust to quantization is to introduce it during training.

Original: Quantization Aware Training (QAT) (Jacob et al., 2018) exposes the network during training by quantizing weights during the forward pass.
Revised: Quantization Aware Training (QAT) (Jacob et al., 2018) exposes the network during training by quantizing weights during the forward pass.

Original: This transformation is not differentiable and gradients are approximated with a straight through estimator (STE) (Bengio et al., 2013; Courbariaux & Bengio, 2016).
Revised: This transformation is not differentiable and gradients are approximated with a straight through estimator (STE) (Bengio et al., 2013; Courbariaux & Bengio, 2016).

Original: STE introduces a bias in the gradients that depends on level of quantization of the weights, and thus, the compression ratio.
Revised: STE introduces a bias in the gradients that depends on level of quantization of the weights, and thus, the compression ratio.

Original: In this section, we propose a simple modiﬁcation to control this induced bias with a stochastic amelioration of QAT, called Quant-Noise.
Revised: In this section, we propose a simple modiﬁcation to control this induced bias with a stochastic amelioration of QAT, called Quant-Noise.

Original: The idea is to quantize a randomly selected fraction of the weights instead of the full network as in QAT, leaving some unbiased gradients ﬂow through unquantized weights.
Revised: The idea is to quantize a randomly selected fraction of the weights instead of the full network as in QAT, leaving some unbiased gradients ﬂow through unquantized weights.

Original: Our general formulation can simulate the effect of both quantization and of pruning during training.
Revised: Our general formulation can simulate the effect of both quantization and of pruning during training.

Original: 4. T RAINING N ETWORKS WITH Q UANTIZATION N OISE
Revised: 4. T RAINING N ETWORKS WITH Q UANTIZATION N OISE

Original: We consider the case of a real matrix W as in Section 3.
Revised: We consider the case of a real matrix W as in Section 3.

Original: During the training of a network, our proposed Quant-Noise method works as follows:
Revised: During the training of a network, our proposed Quant-Noise method works as follows:

Original: ﬁrst, we compute blocks b kl related to a target quantization method.
Revised: ﬁrst, we compute blocks b kl related to a target quantization method.

Original: Then, during each forward pass, we randomly select a subset of these blocks and apply some distortion to them.
Revised: Then, during each forward pass, we randomly select a subset of these blocks and apply some distortion to them.

Original: More formally, given a set of tuples of indices J ⊂ { ( k, l ) } for 1 ≤ k ≤ m , 1 ≤ l ≤ q and a distortion or noise function ϕ
Revised: More formally, given a set of tuples of indices J ⊂ { ( k, l ) } for 1 ≤ k ≤ m , 1 ≤ l ≤ q and a distortion or noise function ϕ

Original: acting on a block, we deﬁne an operator ψ ( · | J ) such that, for each block b kl , we apply the following transformation:
Revised: acting on a block, we deﬁne an operator ψ ( · | J ) such that, for each block b kl , we apply the following transformation:

Original: [Equation]
Revised: [Equation]

Original: The noise function ϕ simulates the change in the weights produced by the target quantization method (see Section 4.2 for details).
Revised: The noise function ϕ simulates the change in the weights produced by the target quantization method (see Section 4.2 for details).

Original: We replace the matrix W by the resulting noisy matrix W noise during the forward pass to compute a noisy output y noise , i.e.,
Revised: We replace the matrix W by the resulting noisy matrix W noise during the forward pass to compute a noisy output y noise , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where x is an input vector.
Revised: where x is an input vector.

Original: During the backward pass, we apply STE, which amounts to replacing the distorted weights W noise by their non-distorted counterparts.
Revised: During the backward pass, we apply STE, which amounts to replacing the distorted weights W noise by their non-distorted counterparts.

Original: Note that our approach is equivalent to
Revised: Note that our approach is equivalent to

Original: QAT when J containts all the tuples of indices.
Revised: QAT when J containts all the tuples of indices.

Original: However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to ﬂow via blocks unaffected by the noise.
Revised: However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to ﬂow via blocks unaffected by the noise.

Original: As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function ϕ .
Revised: As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function ϕ .

Original: As a side effect, our quantization noise regularizes the network in a similar way as DropConnect (Wan et al., 2013) or LayerDrop (Fan et al., 2019).
Revised: As a side effect, our quantization noise regularizes the network in a similar way as DropConnect (Wan et al., 2013) or LayerDrop (Fan et al., 2019).

Original: Composing quantization noises.
Revised: Composing quantization noises.

Original: As noise operators are compositionally commutative, we can make a network robust to a combination of quantization methods by composing their noise operators:
Revised: As noise operators are compositionally commutative, we can make a network robust to a combination of quantization methods by composing their noise operators:

Original: [Equation]
Revised: [Equation]

Original: This property is particularly useful to combine quantization with pruning operators during training, as well as combining scalar quantization with product quantization.
Revised: This property is particularly useful to combine quantization with pruning operators during training, as well as combining scalar quantization with product quantization.

Original: 4. A DDING N OISE TO S PECIFIC Q UANTIZATION M ETHODS
Revised: 4. A DDING N OISE TO S PECIFIC Q UANTIZATION M ETHODS

Original: In this section, we propose several implementations of the noise function ϕ for the quantization methods described in Section 3.
Revised: In this section, we propose several implementations of the noise function ϕ for the quantization methods described in Section 3.

Original: We also show how to handle pruning with it.
Revised: We also show how to handle pruning with it.

Revised: [Table]

Original: Fixed-point scalar quantization.
Revised: Fixed-point scalar quantization.

Original: In intN quantization, the blocks are atomic and weights are rounded to their nearest neighbor in the codebook.
Revised: In intN quantization, the blocks are atomic and weights are rounded to their nearest neighbor in the codebook.

Original: The function ϕ replaces weight W kl with the output of the rounding function deﬁned in Eq.
Revised: The function ϕ replaces weight W kl with the output of the rounding function deﬁned in Eq.

Original: (2), i.e., 
Revised: (2), i.e., ϕ intN

Original: [Equation]
Revised: [Equation]

Original: where s and z are updated during training.
Revised: where s and z are updated during training.

Original: In particular, the application of Quant-Noise to int scalar quantization is [Table] [Table]
Revised: In particular, the application of Quant-Noise to int scalar quantization is a stochastic amelioration of QAT.

Original: Product quantization.
Revised: Product quantization.

Original: As opposed to intN , codebooks in PQ require a clustering step based on weight values.
Revised: As opposed to intN , codebooks in PQ require a clustering step based on weight values.

Original: During training, we learn codewords online and use the resulting centroids to implement the quantization noise.
Revised: During training, we learn codewords online and use the resulting centroids to implement the quantization noise.

Original: More precisely, the noise function ϕ
Revised: More precisely, the noise function ϕ

Original: PQ assigns a selected block b to its nearest codeword in the associated codebook C :
Revised: PQ assigns a selected block b to its nearest codeword in the associated codebook C :

Original: [Equation]
Revised: [Equation]

Original: Updating the codebooks online works well.
Revised: Updating the codebooks online works well.

Original: However, empirically, running k -means once per epoch is faster and does not noticeably modify the resulting accuracy.
Revised: However, empirically, running k -means once per epoch is faster and does not noticeably modify the resulting accuracy.

Original: Note that computing the exact noise function for PQ is computationally demanding.
Revised: Note that computing the exact noise function for PQ is computationally demanding.

Original: We propose a simpler and faster alternative approximation ϕ proxy to the operational transformation of PQ and iPQ.
Revised: We propose a simpler and faster alternative approximation ϕ proxy to the operational transformation of PQ and iPQ.

Original: The noise function simply zeroes out the subvectors of the selected blocks, i.e., ϕ proxy ( v ) = 0 .
Revised: The noise function simply zeroes out the subvectors of the selected blocks, i.e., ϕ proxy ( v ) = 0 .

Original: As a sidenote, we considered other alternatives, for instance one where the subvectors are mapped to the mean subvector.
Revised: As a sidenote, we considered other alternatives, for instance one where the subvectors are mapped to the mean subvector.

Original: In practice, we found that these approximations lead to similar performance, see
Revised: In practice, we found that these approximations lead to similar performance, see

Original: Section 7.2.
Revised: Section 7.2.

Original: This proxy noise function is a form of Structured Dropout and encourages correlations between the subvectors.
Revised: This proxy noise function is a form of Structured Dropout and encourages correlations between the subvectors.

Original: This correlation is beneﬁcial to the subsequent clustering involved in PQ/iPQ.
Revised: This correlation is beneﬁcial to the subsequent clustering involved in PQ/iPQ.

Original: Adding pruning to the quantization noise.
Revised: Adding pruning to the quantization noise.

Original: The speciﬁc form of quantization noise can be adjusted to incorporate additional noise speciﬁc to pruning.
Revised: The speciﬁc form of quantization noise can be adjusted to incorporate additional noise speciﬁc to pruning.

Original: We simply combine the noise operators of quantization and pruning by composing them following Eq.
Revised: We simply combine the noise operators of quantization and pruning by composing them following Eq.

Original: We consider the pruning noise function of Fan et al.
Revised: We consider the pruning noise function of Fan et al.

Original: (2019) where they randomly drop predeﬁned structures during training.
Revised: (2019) where they randomly drop predeﬁned structures during training.

Original: In particular, we focus on LayerDrop , where the structures are the residual blocks of highway-like layers (Srivastava et al., 2015), as most modern architectures, such as ResNet or Transformer, are composed of this structure.
Revised: In particular, we focus on LayerDrop , where the structures are the residual blocks of highway-like layers (Srivastava et al., 2015), as most modern architectures, such as ResNet or Transformer, are composed of this structure.

Original: More precisely, the corresponding noise operator over residual blocks v is ϕ LayerDrop ( v ) = 0 .
Revised: More precisely, the corresponding noise operator over residual blocks v is ϕ LayerDrop ( v ) = 0 .

Original: For pruning, we do not use STE to backpropagate the gradient of pruned weights, as dropping them entirely during training has the beneﬁt of speeding convergence (Huang et al., 2016).
Revised: For pruning, we do not use STE to backpropagate the gradient of pruned weights, as dropping them entirely during training has the beneﬁt of speeding convergence (Huang [Figure] [Figure] [Table]

Original:  Once a model is trained with LayerDrop, the number of layers kept at inference can be adapted to match computation budget or time constraint.
Revised: [Figure] [Table] model size et al., 2016). Once a model is trained with LayerDrop, the number of layers kept at inference can be adapted to match computation budget or time constraint.

Original:  R ESULTS
Revised:  R ESULTS

Original: We demonstrate the impact of Quant-Noise on the performance of several quantization schemes in a variety of settings (see Appendix - Sec. 7.5).[Figure] [Figure] [Table]
Revised: We demonstrate the impact of Quant-Noise on the performance of several quantization schemes in a variety of settings (see Appendix - Sec. 7.5).

Original: [Figure]

Original: [Table]

Original: 5. I MPROVING C OMPRESSION WITH Q UANT -N OISE
Revised: 5. I MPROVING C OMPRESSION WITH Q UANT -N OISE

Original: Quant-Noise is a regularization method that makes networks more robust to the target quantization scheme or combination of quantization schemes during training.
Revised: Quant-Noise is a regularization method that makes networks more robust to the target quantization scheme or combination of quantization schemes during training.

Original: We show the impact of Quant-Noise in Table 1 for a variety of quantization methods: int8/int4 and iPQ.
Revised: We show the impact of Quant-Noise in Table 1 for a variety of quantization methods: int8/int4 and iPQ.

Original: We experiment in 2 different settings: a Transformer network trained for language modeling on
Revised: We experiment in 2 different settings: a Transformer network trained for language modeling on

Original: WikiText-103 and a EfﬁcientNet-B3 convolutional network trained for image classiﬁcation on
Revised: WikiText-103 and a EfﬁcientNet-B3 convolutional network trained for image classiﬁcation on

Original: ImageNet-1k.
Revised: ImageNet-1k.

Original: Our quantization noise framework is general and ﬂexible — Quant-Noise improves the performance of quantized models for every quantization scheme in both experimental settings.
Revised: Our quantization noise framework is general and ﬂexible — Quant-Noise improves the performance of quantized models for every quantization scheme in both experimental settings.

Original: Importantly, Quant-Noise only changes model training by adding a regularization noise similar to dropout, with no impact on convergence and very limited impact on training speed (< 5% slower).
Revised: Importantly, Quant-Noise only changes model training by adding a regularization noise similar to dropout, with no impact on convergence and very limited impact on training speed (< 5% slower).

Original: This comparison of different quantization schemes shows that Quant-Noise works particularly well with high performance quantization methods, like iPQ, where QAT tends to degrade the performances, even compared to quantizing as a post-processing step.
Revised: This comparison of different quantization schemes shows that Quant-Noise works particularly well with high performance quantization methods, like iPQ, where QAT tends to degrade the performances, even compared to quantizing as a post-processing step.

Original: In subsequent experiments in this section, we focus on applications with iPQ because it offers the best trade-off between model performance and compression, and has little negative impact on FLOPS. 
Revised: In subsequent experiments in this section, we focus on applications with iPQ because it offers the best trade-off between model performance and compression, and has little negative impact on FLOPS. [Table]

Original: Fixed-Point Product Quantization.
Revised: Fixed-Point Product Quantization.

Original: Combining iPQ and int8 as described in Section 3.3 allows us to take advantage of the high compression rate of iPQ with a ﬁxed-point representation of both centroids and activations.
Revised: Combining iPQ and int8 as described in Section 3.3 allows us to take advantage of the high compression rate of iPQ with a ﬁxed-point representation of both centroids and activations.

Original: As shown in Table 1, this combination incurs little loss in accuracy with respect to iPQ + Quant-Noise.
Revised: As shown in Table 1, this combination incurs little loss in accuracy with respect to iPQ + Quant-Noise.

Original: Most of the memory footprint of iPQ comes from indexing and not storing centroids, so the compression ratios are comparable.
Revised: Most of the memory footprint of iPQ comes from indexing and not storing centroids, so the compression ratios are comparable.

Original: Complementarity with Weight Pruning and Sharing.
Revised: Complementarity with Weight Pruning and Sharing.

Original: We analyze how Quant-Noise is compatible and complementary with pruning (“+Prune”) and weight sharing (“+Share”), see Appendix for[Table] details on weight sharing.
Revised: We analyze how Quant-Noise is compatible and complementary with pruning (“+Prune”) and weight sharing (“+Share”), see Appendix for details on weight sharing.

Original: We report results for Language modeling on WikiText-103, pre-trained sentence representations on MNLI and object classiﬁcation on ImageNet-1k in Table 2.
Revised: We report results for Language modeling on WikiText-103, pre-trained sentence representations on MNLI and object classiﬁcation on ImageNet-1k in Table 2.

Original: The conclusions are remarkably consistent across tasks and benchmarks: Quant-Noise gives a large improvement over strong iPQ baselines.
Revised: The conclusions are remarkably consistent across tasks and benchmarks: Quant-Noise gives a large improvement over strong iPQ baselines.

Original: Combining it with sharing and pruning offers additional interesting operating points of performance vs size.
Revised: Combining it with sharing and pruning offers additional interesting operating points of performance vs size.

Original: 5. C OMPARISON WITH THE STATE OF THE ART
Revised: 5. C OMPARISON WITH THE STATE OF THE ART

Original: We now compare our approach on the same tasks against the state of the art.
Revised: We now compare our approach on the same tasks against the state of the art.

Original: We compare iPQ +
Revised: We compare iPQ +

Original: Quant-Noise with 6 methods of network compression for Language modeling, 8 state-of-the-art methods for Text classiﬁcation, and 8 recent methods evaluate image classiﬁcation on Imagenet with compressed models.
Revised: Quant-Noise with 6 methods of network compression for Language modeling, 8 state-of-the-art methods for Text classiﬁcation, and 8 recent methods evaluate image classiﬁcation on Imagenet with compressed models.

Original: These comparisons demonstrate that Quant-Noise leads to extreme compression rates at a reasonable cost in accuracy.
Revised: These comparisons demonstrate that Quant-Noise leads to extreme compression rates at a reasonable cost in accuracy.

Original: We apply our best quantization setup on competitive models and reduce their memory footprint by × 20 − 94 when combining with weight sharing and pruning, offering extreme compression for good performance.
Revised: We apply our best quantization setup on competitive models and reduce their memory footprint by × 20 − 94 when combining with weight sharing and pruning, offering extreme compression for good performance.

Original: Natural Language Processing.
Revised: Natural Language Processing.

Original: In Figure 2, we examine the trade-off between performance and model size.
Revised: In Figure 2, we examine the trade-off between performance and model size.

Original: Our quantized RoBERTa offers a competitive trade-off between size and performance with memory reduction methods dedicated to BERT, like TinyBERT, MobileBERT, or AdaBERT.
Revised: Our quantized RoBERTa offers a competitive trade-off between size and performance with memory reduction methods dedicated to BERT, like TinyBERT, MobileBERT, or AdaBERT.

Original: Image Classiﬁcation.
Revised: Image Classiﬁcation.

Original: We compress EfﬁcientNet-B3 from 46 .
Revised: We compress EfﬁcientNet-B3 from 46 .

Original: 7 Mb to 3 .
Revised: 7 Mb to 3 .

Original: 3 Mb ( × 14 compression) while maintaining high top-1 accuracy ( 78 . 5% versus 80% for the original model).
Revised: 3 Mb ( × 14 compression) while maintaining high top-1 accuracy ( 78 . 5% versus 80% for the original model).

Original: As shown in
Revised: As shown in

Original: Figure 2, our quantized EfﬁcientNet-B3 is smaller and more accurate than architectures dedicated to optimize on-device performance with limited size like MobileNet or ShufﬂeNet.
Revised: Figure 2, our quantized EfﬁcientNet-B3 is smaller and more accurate than architectures dedicated to optimize on-device performance with limited size like MobileNet or ShufﬂeNet.

Original: We further evaluate the beneﬁcial effect of Quant-Noise on ResNet-50 to compare directly with Stock et al.
Revised: We further evaluate the beneﬁcial effect of Quant-Noise on ResNet-50 to compare directly with Stock et al.

Original: Results shown in Table 4 indicate improvement with Quant-Noise compared to previous work.
Revised: Results shown in Table 4 indicate improvement with Quant-Noise compared to previous work.

Original: Incorporating pruning noise into quantization is also beneﬁcial.
Revised: Incorporating pruning noise into quantization is also beneﬁcial.

Original: For example, with pruning iPQ+Quant-Noise reduces size by × 25 with only a drop of 2 . 4 PPL in language modeling.
Revised: For example, with pruning iPQ+Quant-Noise reduces size by × 25 with only a drop of 2 . 4 PPL in language modeling.

Original: Further, pruning reduces FLOPS by the same ratio as its compression factor, in our case, × 2 .
Revised: Further, pruning reduces FLOPS by the same ratio as its compression factor, in our case, × 2 .

Original: By adding sharing with pruning, in language modeling, we achieve an extreme compression ratio of × 94 with a drop of 6 . 4 PPL with FLOPS reduction from pruning entire shared chunks of layers.
Revised: By adding sharing with pruning, in language modeling, we achieve an extreme compression ratio of × 94 with a drop of 6 . 4 PPL with FLOPS reduction from pruning entire shared chunks of layers.

Original: For comparison, our 10 MB model has the same performance as the 570 MB Transformer-XL base.
Revised: For comparison, our 10 MB model has the same performance as the 570 MB Transformer-XL base.

Original: 5. F INETUNING WITH Q UANT -N OISE FOR P OST -P ROCESSING Q UANTIZATION
Revised: 5. F INETUNING WITH Q UANT -N OISE FOR P OST -P ROCESSING Q UANTIZATION

Original: We explore taking existing models and post-processing with Quant-Noise instead of training from scratch.
Revised: We explore taking existing models and post-processing with Quant-Noise instead of training from scratch.

Original: For language modeling, we train for 10 additional epochs.
Revised: For language modeling, we train for 10 additional epochs.

Original: For RoBERTa, we train for 25k additional updates.
Revised: For RoBERTa, we train for 25k additional updates.

Original: Finetuning with Quant-Noise incorporates the beneﬁts and almost matches training from scratch (Table 3).
Revised: Finetuning with Quant-Noise incorporates the beneﬁts and almost matches training from scratch (Table 3).

Original: In language modeling, there is only a 0 . 2 PPL difference.
Revised: In language modeling, there is only a 0 . 2 PPL difference.

Original: We further examine how to incorporate Quant-Noise more ﬂexibly into pretraining RoBERTa.
Revised: We further examine how to incorporate Quant-Noise more ﬂexibly into pretraining RoBERTa.

Original: We take an already trained RoBERTa model and incorporate Quant-Noise during sentence classiﬁcation ﬁnetuning.
Revised: We take an already trained RoBERTa model and incorporate Quant-Noise during sentence classiﬁcation ﬁnetuning.

Original: This is effective at compressing while retaining accuracy after quantization.
Revised: This is effective at compressing while retaining accuracy after quantization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We show that quantizing a random subset of weights during training maintains performance in the high quantization regime.
Revised: We show that quantizing a random subset of weights during training maintains performance in the high quantization regime.

Original: We validate that Quant-Noise works with a variety of different quantization schemes on several applications in text and vision.
Revised: We validate that Quant-Noise works with a variety of different quantization schemes on several applications in text and vision.

Original: Our method can be applied to a combination of iPQ and int8 to beneﬁt from extreme compression ratio and ﬁxed-point arithmetic.
Revised: Our method can be applied to a combination of iPQ and int8 to beneﬁt from extreme compression ratio and ﬁxed-point arithmetic.

Original: Finally, we show that Quant-Noise can be used as a post-processing step to prepare already trained networks for subsequent quantization, to improve the performance of the compressed model.
Revised: Finally, we show that Quant-Noise can be used as a post-processing step to prepare already trained networks for subsequent quantization, to improve the performance of the compressed model.

Original:  A PPENDIX 7. Q UANTIZATION OF A DDITIONAL A RCHITECTURES
Revised:  A PPENDIX 7. Q UANTIZATION OF A DDITIONAL A RCHITECTURES

Original: ResNet-50.
Revised: ResNet-50.

Original: We explore the compression of ResNet-50, a standard architecture used Computer
Revised: We explore the compression of ResNet-50, a standard architecture used Computer

Original: Vision.
Revised: Vision.

Original: In Table 4, we compare Quant-Noise to iPQ Compression from Stock et al.
Revised: In Table 4, we compare Quant-Noise to iPQ Compression from Stock et al.

Original: and show that Quant-Noise provide consistent additional improvement.
Revised: and show that Quant-Noise provide consistent additional improvement.

Original: 7. A BLATIONS
Revised: 7. A BLATIONS

Original: In this section, we examine the impact of the level of noise during training as well as the impact of approximating iPQ during training.
Revised: In this section, we examine the impact of the level of noise during training as well as the impact of approximating iPQ during training.

Original: 7. I MPACT OF N OISE R ATE
Revised: 7. I MPACT OF N OISE R ATE

Original: We analyze the performance for various values of Quant-Noise in Figure 3 on a Transformer for language modeling.
Revised: We analyze the performance for various values of Quant-Noise in Figure 3 on a Transformer for language modeling.

Original: For iPQ, performance is impacted by high rates of quantization noise.
Revised: For iPQ, performance is impacted by high rates of quantization noise.

Original: For example, a Transformer with the noise function ϕ proxy degrades with rate higher than 0.5, i.e., when half of the weights are passed through the noise function ϕ proxy .
Revised: For example, a Transformer with the noise function ϕ proxy degrades with rate higher than 0.5, i.e., when half of the weights are passed through the noise function ϕ proxy .

Original: We hypothesize that for large quantities of noise, a larger effect of using proxy rather than the exact PQ noise is observed.
Revised: We hypothesize that for large quantities of noise, a larger effect of using proxy rather than the exact PQ noise is observed.

Original: For int8 quantization and its noise function, higher rates of noise are slightly worse but not as severe.
Revised: For int8 quantization and its noise function, higher rates of noise are slightly worse but not as severe.

Original: A rate of 1 for int8 quantization is equivalent to the Quantization Aware Training of (Krishnamoorthi, 2018), as the full matrix is quantized with STE, showing the potential beneﬁt of partial quantization during training.
Revised: A rate of 1 for int8 quantization is equivalent to the Quantization Aware Training of (Krishnamoorthi, 2018), as the full matrix is quantized with STE, showing the potential beneﬁt of partial quantization during training.

Original: 7. I MPACT OF A PPROXIMATING THE N OISE F UNCTION
Revised: 7. I MPACT OF A PPROXIMATING THE N OISE F UNCTION

Original: We study the impact of approximating quantization noise during training.
Revised: We study the impact of approximating quantization noise during training.

Original: We focus on the case of iPQ with the approximation described in Section 4.2.
Revised: We focus on the case of iPQ with the approximation described in Section 4.2.

Original: In Table 5, we compare the correct noise function for iPQ with its approximation ϕ proxy .
Revised: In Table 5, we compare the correct noise function for iPQ with its approximation ϕ proxy .

Original: This approximate noise function does not consider cluster assignments or centroid values and simply zeroes out the selected blocks.
Revised: This approximate noise function does not consider cluster assignments or centroid values and simply zeroes out the selected blocks.

Original: For completeness, we include an intermediate approximation where we consider cluster assignments to apply noise
Revised: For completeness, we include an intermediate approximation where we consider cluster assignments to apply noise

Original: [Table]
Revised: [Table]

Original: within each cluster, but still zero-out the vectors.
Revised: within each cluster, but still zero-out the vectors.

Original: These approximations do not affect the performance of the quantized models.
Revised: These approximations do not affect the performance of the quantized models.

Original: This suggests that increasing the correlation between subvectors that are jointly clustered is enough to maintain the performance of a model quantized with iPQ.
Revised: This suggests that increasing the correlation between subvectors that are jointly clustered is enough to maintain the performance of a model quantized with iPQ.

Original: Since PQ tends to work well on highly correlated vectors, such as activations in convolutional networks, this is not surprising.
Revised: Since PQ tends to work well on highly correlated vectors, such as activations in convolutional networks, this is not surprising.

Original: Using the approximation ϕ proxy presents the advantage of speed and practicality.
Revised: Using the approximation ϕ proxy presents the advantage of speed and practicality.

Original: Indeed, one does not need to compute cluster assignments and centroids for every layer in the network after each epoch.
Revised: Indeed, one does not need to compute cluster assignments and centroids for every layer in the network after each epoch.

Original: Moreover, the approach ϕ proxy is less involved in terms of code.
Revised: Moreover, the approach ϕ proxy is less involved in terms of code.

Original: 7. E XPERIMENTAL S ETTING
Revised: 7. E XPERIMENTAL S ETTING

Original: We assess the effectiveness of Quant-Noise on competitive language and vision benchmarks.
Revised: We assess the effectiveness of Quant-Noise on competitive language and vision benchmarks.

Original: We consider Transformers for language modeling, RoBERTa for pre-training sentence representations, and EfﬁcientNet for image classiﬁcation.
Revised: We consider Transformers for language modeling, RoBERTa for pre-training sentence representations, and EfﬁcientNet for image classiﬁcation.

Original: Our models are implemented in PyTorch (Paszke et al., 2017).
Revised: Our models are implemented in PyTorch (Paszke et al., 2017).

Original: We use fairseq (Ott et al., 2019) for language modeling and pre-training for sentence representation tasks and Classy Vision (Adcock et al., 2019) for EfﬁcientNet.
Revised: We use fairseq (Ott et al., 2019) for language modeling and pre-training for sentence representation tasks and Classy Vision (Adcock et al., 2019) for EfﬁcientNet.

Original: Language Modeling.
Revised: Language Modeling.

Original: We experiment on the Wikitext-103 benchmark (Merity et al., 2016) that contains 100 M tokens and a vocabulary of 260 k words.
Revised: We experiment on the Wikitext-103 benchmark (Merity et al., 2016) that contains 100 M tokens and a vocabulary of 260 k words.

Original: We train a 16 layer Transformer following Baevski & Auli (2018) with a LayerDrop rate of 0.2 (Fan et al., 2019).
Revised: We train a 16 layer Transformer following Baevski & Auli (2018) with a LayerDrop rate of 0.2 (Fan et al., 2019).

Original: We report perplexity (PPL) on the test set.
Revised: We report perplexity (PPL) on the test set.

Original: Pre-Training of Sentence Representations.
Revised: Pre-Training of Sentence Representations.

Original: We pre-train the base BERT model (Devlin et al., 2018) on the BooksCorpus + Wiki dataset with a LayerDrop rate of 0 . 2 .
Revised: We pre-train the base BERT model (Devlin et al., 2018) on the BooksCorpus + Wiki dataset with a LayerDrop rate of 0 . 2 .

Original: We ﬁnetune the pre-trained models on the MNLI task (Williams et al., 2018) from the GLUE Benchmark (Wang et al., 2019) and report accuracy.
Revised: We ﬁnetune the pre-trained models on the MNLI task (Williams et al., 2018) from the GLUE Benchmark (Wang et al., 2019) and report accuracy.

Original: We follow the parameters in Liu et al.
Revised: We follow the parameters in Liu et al.

Original: (2019) training and ﬁnetuning.
Revised: (2019) training and ﬁnetuning.

Original: Image Classiﬁcation.
Revised: Image Classiﬁcation.

Original: We train an EfﬁcientNet-B3 model (Tan & Le, 2019) on the ImageNet object classiﬁcation benchmark (Deng et al., 2009).
Revised: We train an EfﬁcientNet-B3 model (Tan & Le, 2019) on the ImageNet object classiﬁcation benchmark (Deng et al., 2009).

Original: The EfﬁcientNet-B3 of Classy Vision achieves a
Revised: The EfﬁcientNet-B3 of Classy Vision achieves a

Original: Top-1 accuracy of 81 . 5% , which is slightly below than the performance of 81 .
Revised: Top-1 accuracy of 81 . 5% , which is slightly below than the performance of 81 .

Original: 9% reported by Tan &
Revised: 9% reported by Tan &

Original: Le (2019).
Revised: Le (2019).

Original: 7. T RAINING D ETAILS
Revised: 7. T RAINING D ETAILS

Original: Language Modeling To handle the large vocabulary of Wikitext-103, we follow (Dauphin et al., 2017) and (Baevski & Auli, 2018) in using adaptive softmax (Grave et al., 2016) and adaptive input for computational efﬁciency.
Revised: Language Modeling To handle the large vocabulary of Wikitext-103, we follow (Dauphin et al., 2017) and (Baevski & Auli, 2018) in using adaptive softmax (Grave et al., 2016) and adaptive input for computational efﬁciency.

Original: For both input and output embeddings, we use dimension size 1024 and three adaptive bands: 20 K, 40 K, and
Revised: For both input and output embeddings, we use dimension size 1024 and three adaptive bands: 20 K, 40 K, and

Original: 200 K.
Revised: 200 K.

Original: We use a cosine learning rate schedule (Baevski & Auli, 2018; Loshchilov & Hutter, 2016) and train with Nesterov’s accelerated gradient (Sutskever et al., 2013).
Revised: We use a cosine learning rate schedule (Baevski & Auli, 2018; Loshchilov & Hutter, 2016) and train with Nesterov’s accelerated gradient (Sutskever et al., 2013).

Original: We set the momentum to 0.99 and renormalize gradients if the norm exceeds 0.1 (Pascanu et al., 2014).
Revised: We set the momentum to 0.99 and renormalize gradients if the norm exceeds 0.1 (Pascanu et al., 2014).

Original: During training, we partition the data into blocks of contiguous tokens that ignore document boundaries.
Revised: During training, we partition the data into blocks of contiguous tokens that ignore document boundaries.

Original: At test time, we respect sentence boundaries.
Revised: At test time, we respect sentence boundaries.

Original: We set LayerDrop to 0.2.
Revised: We set LayerDrop to 0.2.

Original: We set
Revised: We set

Original: Quant-Noise value to 0.05.
Revised: Quant-Noise value to 0.05.

Original: During training time, we searched over the parameters (0.05, 0.1, 0.2) to determine the optimal value of Quant-Noise.
Revised: During training time, we searched over the parameters (0.05, 0.1, 0.2) to determine the optimal value of Quant-Noise.

Original: During training time, the block size of Quant-Noise is 8.
Revised: During training time, the block size of Quant-Noise is 8.

Original: RoBERTa The base architecture is a 12 layer model with embedding size 768 and FFN size 3072 .
Revised: RoBERTa The base architecture is a 12 layer model with embedding size 768 and FFN size 3072 .

Original: We follow (Liu et al., 2019) in using the subword tokenization scheme from (Radford et al., 2019), which uses bytes as subword units.
Revised: We follow (Liu et al., 2019) in using the subword tokenization scheme from (Radford et al., 2019), which uses bytes as subword units.

Original: This eliminates unknown tokens.
Revised: This eliminates unknown tokens.

Original: We train with large batches of size 8192 and maintain this batch size using gradient accumulation.
Revised: We train with large batches of size 8192 and maintain this batch size using gradient accumulation.

Original: We do not use next sentence prediction (Lample & Conneau, 2019).
Revised: We do not use next sentence prediction (Lample & Conneau, 2019).

Original: We optimize with Adam with a polynomial decay learning rate schedule.
Revised: We optimize with Adam with a polynomial decay learning rate schedule.

Original: We set LayerDrop to 0.2.
Revised: We set LayerDrop to 0.2.

Original: We set Quant-Noise value to 0.1.
Revised: We set Quant-Noise value to 0.1.

Original: We did not hyperparameter
Revised: We did not hyperparameter

Original: [Table]
Revised: [Table]

Original: Lower [Table]
Revised: Lower perplexity is better.

Original: search to determine the optimal value of Quant-Noise as training RoBERTa is computationally intensive.
Revised: search to determine the optimal value of Quant-Noise as training RoBERTa is computationally intensive.

Original: During training time, the block size of Quant-Noise is 8.
Revised: During training time, the block size of Quant-Noise is 8.

Original: During ﬁnetuning, we hyperparameter search over three learning rate options (1e-5, 2e-5, 3e-5) and batchsize (16 or 32 sentences).
Revised: During ﬁnetuning, we hyperparameter search over three learning rate options (1e-5, 2e-5, 3e-5) and batchsize (16 or 32 sentences).

Original: The other parameters are set following (Liu et al., 2019).
Revised: The other parameters are set following (Liu et al., 2019).

Original: We do single task ﬁnetuning, meaning we only tune on the data provided for the given natural language understanding task.
Revised: We do single task ﬁnetuning, meaning we only tune on the data provided for the given natural language understanding task.

Original: We do not perform ensembling.
Revised: We do not perform ensembling.

Original: When ﬁnetuning models trained with LayerDrop, we apply LayerDrop and Quant-Noise during ﬁnetuning time as well.
Revised: When ﬁnetuning models trained with LayerDrop, we apply LayerDrop and Quant-Noise during ﬁnetuning time as well.

Original: EfﬁcientNet We use the architecture of EfﬁcientNet-B3 deﬁned in Classy Vision (Adcock et al., 2019) and follow the default hyperparameters for training.
Revised: EfﬁcientNet We use the architecture of EfﬁcientNet-B3 deﬁned in Classy Vision (Adcock et al., 2019) and follow the default hyperparameters for training.

Original: We set Quant-Noise value to 0.1.
Revised: We set Quant-Noise value to 0.1.

Original: During training time, we searched over the parameters (0.05, 0.1, 0.2) to determine the optimal value of Quant-Noise.
Revised: During training time, we searched over the parameters (0.05, 0.1, 0.2) to determine the optimal value of Quant-Noise.

Original: During training time, the block size of Quant-Noise is set to 4 for all 1 ×convolutions, 9 for depth-wise 3 × 3 convolutions, 5 for depth-wise 5 × 5 convolutions and 4 for the classiﬁer.
Revised: During training time, the block size of Quant-Noise is set to 4 for all 1 ×convolutions, 9 for depth-wise 3 × 3 convolutions, 5 for depth-wise 5 × 5 convolutions and 4 for the classiﬁer.

Original: For sharing, we shared weights between blocks 9-10, 11-12, 14-15, 16-17, 19-20-21, 22-23 and refer to blocks that share the same weights as a chunk .
Revised: For sharing, we shared weights between blocks 9-10, 11-12, 14-15, 16-17, 19-20-21, 22-23 and refer to blocks that share the same weights as a chunk .

Original: For LayerDrop, we drop the chunks of blocks deﬁned previously with probability 0.2 and evaluate only with chunks 9-10, 14-15 and 19-20-21.
Revised: For LayerDrop, we drop the chunks of blocks deﬁned previously with probability 0.2 and evaluate only with chunks 9-10, 14-15 and 19-20-21.

Original: 7. S CALAR Q UANTIZATION D ETAILS
Revised: 7. S CALAR Q UANTIZATION D ETAILS

Original: We closely follow the methodology of PyTorch 1.4.
Revised: We closely follow the methodology of PyTorch 1.4.

Original: We emulate scalar quantization by quantizing the weights and the activations.
Revised: We emulate scalar quantization by quantizing the weights and the activations.

Original: The scales and zero points of activations are determined by doing a few forward passes ahead of the evaluation and then ﬁxed.
Revised: The scales and zero points of activations are determined by doing a few forward passes ahead of the evaluation and then ﬁxed.

Original: We use the Histogram method to compute s and z , which aims at approximately minimizing the L 2 quantization error by adjusting s and z .
Revised: We use the Histogram method to compute s and z , which aims at approximately minimizing the L 2 quantization error by adjusting s and z .

Original: This scheme is a reﬁnement of the MinMax scheme.
Revised: This scheme is a reﬁnement of the MinMax scheme.

Original: Per channel quantization is also discussed in
Revised: Per channel quantization is also discussed in

Original: Table 10.
Revised: Table 10.

Original: 7. I PQ Q UANTIZATION D ETAILS
Revised: 7. I PQ Q UANTIZATION D ETAILS

Original: Language Modeling We quantize FFN with block size 8, embeddings with block size 8, and attention with block size 4.
Revised: Language Modeling We quantize FFN with block size 8, embeddings with block size 8, and attention with block size 4.

Original: We tuned the block size for attention between the values (4, 8) to ﬁnd the best performance.
Revised: We tuned the block size for attention between the values (4, 8) to ﬁnd the best performance.

Original: Note that during training with apply Quant-Noise to all the layers.
Revised: Note that during training with apply Quant-Noise to all the layers.

Original: RoBERTa We quantize FFN with block size 4, embeddings with block size 4, and attention with block size 4.
Revised: RoBERTa We quantize FFN with block size 4, embeddings with block size 4, and attention with block size 4.

Original: We tuned the block size between the values (4, 8) to ﬁnd the best performance.
Revised: We tuned the block size between the values (4, 8) to ﬁnd the best performance.

Original: Note that during training with apply Quant-Noise to all the layers.
Revised: Note that during training with apply Quant-Noise to all the layers.

Original: EfﬁcientNet We quantize blocks sequentially and end up with the classiﬁer.
Revised: EfﬁcientNet We quantize blocks sequentially and end up with the classiﬁer.

Original: The block sizes arefor all 1 × 1 convolutions, 9 for depth-wise 3 × 3 convolutions, 5 for depth-wise 5 × 5 convolutions and 4 for the classiﬁer.
Revised: The block sizes arefor all 1 × 1 convolutions, 9 for depth-wise 3 × 3 convolutions, 5 for depth-wise 5 × 5 convolutions and 4 for the classiﬁer.

Original: Note that during training with apply Quant-Noise to all the weights in
Revised: Note that during training with apply Quant-Noise to all the weights in

Original: InvertedResidual Blocks (except the Squeeze-Excitation subblocks), the head convolution and the 
Revised: InvertedResidual Blocks (except the Squeeze-Excitation subblocks), the head convolution and the classiﬁer.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: D ETAILS OF P RUNING AND L AYER S HARING
Revised: D ETAILS OF P RUNING AND L AYER S HARING

Original: We apply the Every Other Layer strategy from Fan et al.
Revised: We apply the Every Other Layer strategy from Fan et al.

Original: When combining layer sharing with pruning, we train models with shared layers and then prune chunks of shared layers.
Revised: When combining layer sharing with pruning, we train models with shared layers and then prune chunks of shared layers.

Original: When sharing layers, the weights of adjacent layers are shared in chunks of two.
Revised: When sharing layers, the weights of adjacent layers are shared in chunks of two.

Original: For a concrete example, imagine we have a model with layers A, B, C, D, E, F, G, H.
Revised: For a concrete example, imagine we have a model with layers A, B, C, D, E, F, G, H.

Original: We share layers A and B, C and D, E and F, G and H. To prune, every other chunk would be pruned away, for example we could prune A, B, E, F.
Revised: We share layers A and B, C and D, E and F, G and H. To prune, every other chunk would be pruned away, for example we could prune A, B, E, F.

Original: N UMERICAL R ESULTS FOR G RAPHICAL D IAGRAMS
Revised: N UMERICAL R ESULTS FOR G RAPHICAL D IAGRAMS

Original: We report the numerical values displayed in Figures 2 in Table 6 for language modeling, Table 7 for
Revised: We report the numerical values displayed in Figures 2 in Table 6 for language modeling, Table 7 for

Original: BERT, and Table 8 for ImageNet.
Revised: BERT, and Table 8 for ImageNet.

Original: 7. F URTHER A BLATIONS 7.11. I MPACT OF Q UANT -N OISE FOR THE V ISION SETUP
Revised: 7. F URTHER A BLATIONS 7.11. I MPACT OF Q UANT -N OISE FOR THE V ISION SETUP

Original: We provide another study showing the impact of the proportion of elements on which to apply
Revised: We provide another study showing the impact of the proportion of elements on which to apply

Original: Quant-Noise in Table 9.
Revised: Quant-Noise in Table 9.

Original: 7.11. I MPACT OF THE NUMBER OF CENTROIDS
Revised: 7.11. I MPACT OF THE NUMBER OF CENTROIDS

Original: We quantize with 256 centroids which represents a balance between size and representation capacity.
Revised: We quantize with 256 centroids which represents a balance between size and representation capacity.

Original: The effect of the number of centroids on performance and size is shown in Figure 4 (a).
Revised: The effect of the number of centroids on performance and size is shown in Figure 4 (a).

Original: Quantizingp
Revised: Quantizing

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Top80.
Revised: Top80.

Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 7.11. E FFECT OF I NITIAL M ODEL S IZE
Revised: 7.11. E FFECT OF I NITIAL M ODEL S IZE

Original: Large, overparameterized models are more easily compressed.
Revised: Large, overparameterized models are more easily compressed.

Original: In Figure 5, we explore quantizing both shallower and skinnier models.
Revised: In Figure 5, we explore quantizing both shallower and skinnier models.

Original: For shallow models, the gap between quantized and non-quantized perplexity does not increase as layers are removed (Figure 5, left).
Revised: For shallow models, the gap between quantized and non-quantized perplexity does not increase as layers are removed (Figure 5, left).

Original: In contrast, there is a larger gap in performance for models with smaller FFN (Figure 5, right).
Revised: In contrast, there is a larger gap in performance for models with smaller FFN (Figure 5, right).

Original: As the FFN size decreases, the weights are less redundant and more difﬁcult to quantize with iPQ.
Revised: As the FFN size decreases, the weights are less redundant and more difﬁcult to quantize with iPQ.

Original: 7.11. D IFFICULTY OF Q UANTIZING D IFFERENT M ODEL S TRUCTURES
Revised: 7.11. D IFFICULTY OF Q UANTIZING D IFFERENT M ODEL S TRUCTURES

Original: Quantization is applied to various portions of the Transformer architecture — the embedding, attention, feedforward, and classiﬁer output.
Revised: Quantization is applied to various portions of the Transformer architecture — the embedding, attention, feedforward, and classiﬁer output.

Original: We compare the quantizability of various portions of the network in this section.
Revised: We compare the quantizability of various portions of the network in this section.

Original: Is the order of structures important?
Revised: Is the order of structures important?

Original: We quantize speciﬁc network structures ﬁrst — this is important as quantizing weight matrices can accumulate reconstruction error.
Revised: We quantize speciﬁc network structures ﬁrst — this is important as quantizing weight matrices can accumulate reconstruction error.

Original: Some structures of the network should be quantized last so the ﬁnetuning process can better adjust the centroids.
Revised: Some structures of the network should be quantized last so the ﬁnetuning process can better adjust the centroids.

Original: We ﬁnd that there are small variations in performance based on quantization order (see Figure 6).
Revised: We ﬁnd that there are small variations in performance based on quantization order (see Figure 6).

Original: We choose to quantize FFN, then embeddings, and ﬁnally the attention matrices in Transformer networks.
Revised: We choose to quantize FFN, then embeddings, and ﬁnally the attention matrices in Transformer networks.

Original: Which structures can be compressed the most?
Revised: Which structures can be compressed the most?

Original: Finally, we analyze which network structures can be most compressed.
Revised: Finally, we analyze which network structures can be most compressed.

Original: During quantization, various matrix block sizes can be chosen as a parameter — the larger the block size, the more compression, but also the larger the potential reduction of performance.
Revised: During quantization, various matrix block sizes can be chosen as a parameter — the larger the block size, the more compression, but also the larger the potential reduction of performance.

Original: Thus, it is important to understand how much each network structure can be compressed to reduce the memory footprint of the ﬁnal model as much as possible.
Revised: Thus, it is important to understand how much each network structure can be compressed to reduce the memory footprint of the ﬁnal model as much as possible.

Original: In Figure 6, we quantize two model structures with a ﬁxed block size and vary the block size of the third between 4 and 32.
Revised: In Figure 6, we quantize two model structures with a ﬁxed block size and vary the block size of the third between 4 and 32.

Original: As shown, the FFN and embedding structures are more robust to aggressive compression, while the attention drastically loses performance as larger block sizes are used.
Revised: As shown, the FFN and embedding structures are more robust to aggressive compression, while the attention drastically loses performance as larger block sizes are used.

Original: 7.11. A PPROACH TO INT N S CALAR Q UANTIZATION
Revised: 7.11. A PPROACH TO INT N S CALAR Q UANTIZATION

Original: We compare quantizing per-channel to using a histogram quantizer in Table 10.
Revised: We compare quantizing per-channel to using a histogram quantizer in Table 10.

Original: The histogram quantizer maintains a running min/max and minimizes L2 distance between quantized and nonquantized values to ﬁnd the optimal min/max.
Revised: The histogram quantizer maintains a running min/max and minimizes L2 distance between quantized and nonquantized values to ﬁnd the optimal min/max.

Original: Quantizing per channel learns scales and offsets as vectors along the channel dimension, which provides more ﬂexibility since scales and offsets can be different.
Revised: Quantizing per channel learns scales and offsets as vectors along the channel dimension, which provides more ﬂexibility since scales and offsets can be different.

Original: [Figure]
Revised: [Figure]

Original: more skinny models
Revised: more skinny models

Original: [Figure]
Revised: [Figure]

Original: 7.11. L AYER D ROP WITH STE
Revised: 7.11. L AYER D ROP WITH STE

Original: For quantization noise
Revised: For quantization noise

Original: [Figure]
Revised: [Figure]

Original: we apply the straight through estimator (STE) to remaining weights in the backward pass.
Revised: we apply the straight through estimator (STE) to remaining weights in the backward pass.

Original: We experiment with applying STE to the backward pass of LayerDrop’s pruning noise.
Revised: We experiment with applying STE to the backward pass of LayerDrop’s pruning noise.

Original: Results are shown in Table 11 and ﬁnd slightly worse results
Revised: Results are shown in Table 11 and ﬁnd slightly worse results

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: pruning noise.
Revised: pruning noise.