QD ROP : R ANDOMLY D ROPPING Q UANTIZATION FOR E XTREMELY L OW - BIT P OST - TRAINING Q UANTIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently, post-training quantization (PTQ) has driven much attention to produce efficient neural networks without long-time retraining.
Despite the low cost, current PTQ works always fail under the extremely low-bit setting.
In this study, we pioneeringly confirm that properly incorporating activation quantization into the
PTQ reconstruction benefits the final accuracy.
To deeply understand the inherent reason, a theoretical framework is established, which inspires us that the flatness of the optimized low-bit model on calibration and test data is crucial.
Based on the conclusion, a simple yet effective approach dubbed as QD ROP is proposed, which randomly drops the quantization of activations during reconstruction.
Extensive experiments on various tasks including computer vision (image classification, object detection) and natural language processing (text classification and question answering) prove its superiority.
With QD ROP , the limit of PTQ is pushed to the 2-bit activation for the first time and the accuracy boost can be up to 51.49%.
Without bells and whistles, QD ROP establishes a new state of the art for PTQ.
I NTRODUCTION
In recent years, deep learning has been applied to all walks of life and offered substantial convenience for people’s production and activities.
While the performance of deep neural networks continues to increase, the memory and computation cost also scale up fastly and bring new challenges for edge devices.
Model compression techniques such as network pruning (Han et al., 2015), distillation (Hinton et al., 2015), network quantization (Jacob et al., 2018) and neural architecture search (Zoph & Le, 2016) etc., are dedicated to reducing calculation and storage overhead.
In this paper, we study quantization which adopts low-bit representation for weights and activations to enable fixed-point computation and less memory space.
Based on the cost of a quantization algorithm, researchers usually divide the quantization work into two categories: (1) Quantization-Aware Training (QAT) and (2) Post-Training Quantization (PTQ).
QAT finetunes a pre-trained model by leveraging the whole dataset and GPU effort.
On the contrary,
PTQ demands much less computation to obtain a quantized model since it does not require end-toend training.
Therefore, much attention has recently been paid to PTQ (Cai et al., 2020;
Wang et al., 2020; Hubara et al., 2021; Banner et al., 2019; Nahshan et al., 2019; Zhang et al., 2021; Li et al., 2021b)
due to its low cost and easy-to-use characteristics in practice.
Traditionally, PTQ pursues accuracy by performing the rounding-to-nearest operation, which focuses on minimizing the distance from the full-precision (FP) model in parameter space.
In recent progress, Nagel et al.
; Li et al.
(2021a) considered minimizing the distance in model space, i.e. the final loss objective.
They use Taylor Expansion to analyze the change of loss value and derive a method to reconstruct the pre-trained model’s feature by learning the rounding scheme.
Such methods are efficient and effective in 4-bit quantization and can even push the limit of weight quantization to 2-bit.
However, the extremely low-bit activation quantization, which faces more challenges, still fails to achieve satisfactory accuracy.
We argue that one key reason is that existing theoretical analyses only model the weight quantization as perturbation while ignoring activation’s.
This will lead to the same optimized model no matter which bit the activations use, which is obviously counter-intuitive and thus causes a sub-optimal solution.
In this work, the effect of activation quantization in PTQ is deeply investigated for the first time.
We empirically observe that perceiving the activation quantization benefits the extremely low-bit
PTQ reconstruction and surprisingly find that only partial activation quantization is more preferable.
An intuitive understanding is that incorporating activation will lead to a different optimized weight.
Inspired by this, we conduct theoretical studies on how activation quantization affects the weight tuning, and the conclusion is that involving activation quantization into the reconstruction helps the flatness of model on calibration data and dropping partial quantization contributes to the flatness on test data.
Motivated by both empirical and theoretical findings, we propose QD ROP that randomly drops quantization during the PTQ reconstruction to pursue the flatness from a general perspective.
With this simple and effective approach, we set up a new state of the art for PTQ on various tasks including image classification, object detection for computer vision, and text classification and question answering for natural language processing.
This paper makes the following contributions:
We confirm the benefits unprecedentedly from involving activation quantization in the PTQ reconstruction and surprisingly observe that partial involvement of activation quantization performs better than the whole.
A theoretical framework is established to deeply analyze the influence of incorporating activation quantization into weight tuning.
Using this framework, we conclude that the flatness of the optimized low-bit model on calibration data and test data is crucial for the final accuracy.
Based on the empirical and theoretical analyses, we propose a simple yet effective method QD ROP that achieves the flatness from a general perspective.
QD ROP is easy to implement and can consistently boost existing methods as a plug-and-play module for various neural networks including CNNs like ResNets and Transformers like BERT.
Extensive experiments on a large variety of tasks and models prove that our method set up a new state of the art for PTQ.
With QD ROP , the 2-bit post-training quantization becomes possible for the first time.
P RELIMINARIES
Basic Notations.
Throughout this paper, matrices (or tensors) are marked as X , whereas the vectors are denoted by x .
Sometimes we use w to represent the flattened version of the weight matrix W .
Operator · is marked as scalar multiplication, ⊙ is marked as element-wise multiplication for matrices or vectors.
For matrix multiplication, we denote W x as matrix-vector multiplication or W X as matrix-matrix multiplication. For a feedforward neural network with activation function, we denote it as G ( x , w ) and the loss function as L ( x , w ) , where x and w are the network inputs and weights, respectively.
Note that we assume x is sampled from training dataset D t , thus the final loss is denoted by E x
∼D t [ L ( ˆ w , x )] .
For the network forward function, we can write it as:
[Equation]
where W i,j denotes weight connecting the j th activation neuron and the i th output.
The bracket superscript ( ℓ ) is the layer index.
f ( · ) indicates the activation function.
Post-training Quantization.
Uniform quantizer maps continuous values x ∈ R into fixed-point integers.
For instance, the activation quantization function can be written as ˆ x = ⌊ xs ⌉ s , where ⌊·⌉ denotes the rounding-to-nearest operator, s is the step size between two subsequent quantization levels.
While rounding-to-nearest operation minimizes the mean squared error between ˆ x and x , the minimization of parameter space certainly cannot equal to the minimization in final task loss (Li et al., 2021a), i.e., E x ∼D t
[ L ( ˆ w , x )] .
However, in the post-training setting, we only have a tiny subset D c ∈ D t that only contains 1k images.
Thus, it is hard to minimize the final loss objective with limited data.
Recently, a series of works (Nagel et al., 2020; Li et al., 2021a) learn to either round up or down and view the new rounding mechanism as weight perturbation, i.e., ˆ w = w + ∆ w .
Take a pre-trained network G as an example, they leverage Taylor Expansion to analyze the target, which reveals the quantization interactions among weights:
[Equation]
where H w = E x
∼D t ∇ 2 w L ( w , x ) is the expected second-order derivative.
The above objective could be transformed into the change of output weighted by the output Hessian.
[Equation]
About the above minimization, they finetune only the weight by reconstructing each block/layer output (See Figure 1(a)).
But they did not explore the activation quantization during output reconstruction with only modeling weight quantization as noise.
The step size for activation quantization is determined after the reconstruction stage.
Intuitively, when quantizing the activations of a full-precision model to 2-bit and 3-bit, there should be different suitable weights.
However, the existing works result in the same optimized weight due to the neglect of activation quantization.
Therefore, we argue that when quantizing the neural network, the noise caused by activation quantization should be considered coherently with weights.
M ETHODOLOGY
In this section, to reveal the influence of introducing activation quantization before output reconstruction, we first conduct empirical experiments and present two observations.
Then a theoretical framework is built to investigate how the activation quantization affects the optimized weights.
Last, equipped with the analysis conclusions, a simple yet effective method dubbed QD ROP is proposed.
[Figure]
[Table]
3. E MPIRICAL O BSERVATIONS
To investigate the influence of activation quantization when reconstructing the layer/block output, we conduct preliminary experiments on the ImageNet (Russakovsky et al., 2015) dataset.
Our experiments are based on the open-sourced code Li et al.
(2021a)
except that we will introduce activation quantization from 1 to k − 1 blocks before the k th block’s reconstruction.
We give a simple visualization in Fig.
1 to show 3 cases for putting activation quantization in different stages.
Case 1 means that all activation are kept in 32-bit full-precision during the reconstruction of block output, which is also adopted in existing work Nagel et al.
; Li et al.
(2021a).
Case 2 and Case 3 are used for incorporating activation quantization into the reconstruction stage.
However, Case 3 will omit the current block’s quantization while Case 2 will not.
The detailed results of these three cases are listed in Table 1 (Comparisons on W2A2 except that MobileNetV2 and MnasNet take W2A3 for crashed results on 2-bit activation) and the algorithm is put in algorithm 2.
According to the comparison, we can obtain two observations:
For extremely low-bit quantization (e.g., W2A2), there will be huge accuracy improvement when considering activation quantization during weight tuning.
This is confirmed by comparing with Case 1 and Case 2.
We find Case 1 barely converges while Case 2 achieves good accuracy.
It reveals that a separate optimization of weights and activations cannot find an optimal solution.
After introducing the activation quantization, the weights will learn to diminish the influence of activation quantization.
Partially introducing block-wise activation quantization surpasses introducing the whole activation quantization.
Case 3 does not quantize the activations inside the current tuning block but achieves better results than Case 2.
This inspires us that how much quantization of activations we introduce will affect the final accuracy.
3. H OW DOES ACTIVATION QUANTIZATION AFFECT WEIGHT TUNING
The empirical observations have highlighted the importance of activation quantization during the
PTQ pipeline.
To further explore how activation quantization will affect the weight tuning, we build a theoretical framework that analyzes the final loss objective with both weights and activations being quantized, which presents clues of high accuracy for extremely low-bit post-training quantization.
Conventionally, the activation quantization could be modeled as injecting some form of noise imposed on the full-precision counterpart, defined as e =
(ˆ a − a ) .
To remove the influence of activation range on e , we translate the noise into a multiplicative form, i.e., ˆ a = a · (1 + u ) , where the range of u is affected by bit-width and rounding error. Detailed illustration of the new form noise can be found in Appendix A.
To present the activation noise, we add another argument in calculating the loss function and define our optimization objective in PTQ as:
[Equation]
And we adopt 1 + u ( x ) to denote such noise since it is related to specific input data point x .
We hereby use a transformation that can absorb the noise on activation.
Consider a simple matrixvector multiplication W a in forward pass, we have W ( a ⊙ ( 1 + u ( x )))
=
( W ⊙ ( 1 + V ( x )))
a , given by
[Equation]
By taking V i,j = u j , quantization noise on the activation vector ( 1 + u ( x ) ) can be transplanted into perturbations on weight, where we use 1 + v ( x ) to denote it. Note that for a specific input data point x , there are two distinct u ( x ) and v ( x ) . Proofs is available at Sec.
B.1. Also note that for a convolutional layer, we cannot apply such transformation since the input to convolution is a matrix and will cause different V .
Nonetheless, we can give a formal lemma that absorbs u and holds corresponding v (See the Appendix Sec. B.2 for rigorous proof for convolutional structure):
Lemma 1.
For a quantized (convolutional) neural network, the influence of activation quantization on the final loss objective in post-training quantization can be transformed into weight perturbation.
[Equation]
By interpolating L ( ˆ w , x ) into Lemma 1, we can obtain the final theorem:
Theorem 1.
For a neural network G with quantized weight ˆ w and activation perturbation 1 + u ( x ) , we have:
[Equation]
Here, Theorem 1 divides optimization objective into two terms.
Term (7-1) is the same as Eq.
(2) explored in (Nagel et al., 2020; Li et al., 2021a), which reveals how weight quantization interacts with loss function.
Term (7-2) is the loss change by introducing activation quantization on the weight quantized network G ( ˆ w , x ) .
In another way to interpret Eq.
(7), the term (7-2) stands for the loss change with jitters on the quantized weight.
This type of noise correlates with certain kinds of robustness.
[Figure] As stated in some works about generalization and flatness (Dinh et al., 2017; Hochreiter & Schmidhuber, 1997), intuitively, flat minimum means relatively small loss change under perturbations in the parameters, otherwise, the minimum is sharp. In this paper, we follow the notion of flatness defined in (Neyshabur et al., 2017), which considers loss change from the perspective of statistical expectation. And as (Neyshabur et al., 2017) and (Jiang et al., 2019) refer to, we consider the magnitude of the perturbation with respect to the magnitude of parameters and take the formulation as E v ∼D
[ L ( f w ⊙ ( 1 + v ) ) − L ( f w )] , where each element of v is a random variable sampled from a noise distribution D and L represents for optimization objective on the training set.
From this perspective, the term (7-2) can be interpreted as the flatness with perturbation related to input data, and thereby we could achieve the following corollary.
Corollary 1.
On calibration data x , with activation quantization noise u ( x ) , there exists the corresponding v ( x ) that the trained quantized model is flatter under the perturbation v ( x ) .
With Corollary 1, Case 2 and 3 discussed in Sec.
3.1 enjoy a flatter loss landscape benefited from perceiving the activation quantization.
This explains their superiority compared with Case 1.
The measurement of sharpness on calibration data (left part) in Fig.
2 further validates this point.
With similar perturbation magnitude, Case 2 and 3 suffer less loss degradation than Case 1.
3. QD ROP
As aforementioned, introducing activation quantization is theoretically proved to produce a flatter model than existing works and the directions of flatness depend on the data distribution.
Since the
PTQ is especially sensitive to calibration data (Yu et al., 2021), we need to transfer the investigations in Sec.
3.2 on calibration data into the test setting for a thorough understanding.
In specific, we consider Eq.
(7) on test set and inspect two terms separately in the following.
Based on the analyses, our method QD ROP will be derived to pursue an excellent performance on test data.
Term (7-1) on test set.
As suggested in Sec. 3.2, with both quantized activations and weights, we additionally optimize the term (7-2) representing the flatness on calibration data.
This term will encourage the quantized model to learn a flat minimum.
As a result, the traditional objective of AdaRound (term (7-1)) can naturally generalize better for test data (i.e., E x ∼D t ( L ( ˆ w , x , 1 ) − L ( w , x , 1 )) ).
Term (7-2) on test set.
Furthermore, we should also concern about the term (7-2) on test data, i.e. E x
∼D t [ L ( ˆ w ⊙ ( 1 + v ( x )) , x , 1 ) − L ( ˆ w , x , 1 )] .
As revealed in Sec. 3.2, the term (7-2) implies the flatness where its situation on calibration data has been exploited.
Here, we further investigate the flatness on test samples.
Note that v ( x ) is converted from u ( x ) and this activation quantization noise varies with input data.
Fig.
2 shows that there is a gap between the test data and calibration data for the flatness of the 3 cases.
According to Corollary 1, these 3 cases actually introduce different u mathematically and thus will result in different flatness directions, given by:
[Equation]
During training phase, substitute ˆ a i − 1 with corresponding a i − 1 at neuron-level with probability p and mark the replaced input as ˜ a
i − 1 ; for l = i to j do a l ← W l ˜ a l
; ˆ a l ← Quantize
During training phase, randomly drop some ˆ a l with a l as defined in Eq.
(9) and get ˜ a l ;
Compute ∆ a j = ˜ a j − a j ;
Tune the weight by gradient descent ; return Quantized block ;
For Case 1, there is no activation quantization during calibration without taking flatness into account.
Case 2 suggests the activation perturbation totally and therefore enjoys a good flatness on calibration data.
However, due to the mismatch on calibration data and test one, it is highly possible that Case 2 causes overfitting (See Table 9 for more details).
Case 3, in fact achieves the best performance by dropping some activation quantization along with a little different weight perturbation and might not be restricted to flatness on calibration data (More evidence can be found in Table 11).
This inspires us to pursue a flat minimum from a general perspective, that only optimizing the target on calibration set is suboptimal to test set.
QD ROP .
Inspired by this, we propose QD ROP to further increase the flatness on as many directions as possible.
In particular, we randomly disable and enable the quantization of the activation each forward pass:
[Equation]
We name it QD ROP because it randomly drops the quantization of activation.
Theoretically, by masking some u ( x ) randomly, QD ROP can have more diverse v ( x ) and cover more directions of flatness thus flatter on test samples, which contributes to the final high accuracy.
Fig.
3 support our analysis where QD ROP has smoother loss landscape than Case 3, the winner among 3 Cases on test data. Meanwhile, it is indeed a fine-grained version of Case 3 since Case 3 drops the quantization in a block-wise manner, whereas our QD ROP operates in an element-wise way.
Discussions.
QD ROP can be viewed as a generalized form of the existing schemes.
Case 1 andrespectively corresponds to the dropping probability of p = 1 and p = 0 .
Case 3 is equivalent to setting the block being optimized with dropping probability p = 1 and remains the quantization of other parts.
Note that the p obeys Bernoulli distribution and thus can be set as 0.5 for the maximal entropy (Qin et al., 2020), which is helpful for flatness across various directions.
QD ROP is easy to implement for various neural networks including CNNs and Transformers, and plug-and-play with little additional computational complexity.
With QD ROP , the complicated problem of choosing optimization order, i.e. different cases in Sec.
[-v0.2 0. Loss 
 [Figure] [Table]-]3.1, can be avoided.
E XPERIMENTS
In this section, we conduct two sets of experiments to verify the effectiveness of QD ROP .
In Sec. 4.1, we first conduct an ablation study for the impact with and without dropping quantization and analyze the option of distinct quantization dropping rates.
In Sec. 4.2, we compare our method with other existing approaches across vision and language tasks.
Implementation Details.
Our code is based on PyTorch Paszke et al.
(2019), and has been attached in our supplemental material.
We set the default dropping probability p as 0.5, except we explicitly mention it.
Our experiments including image classification on ImageNet, object detection on [Figure] [Figure] No
[Figure] 16.482 QD ROP 2/ [Table] [Table]
[Table]
[Table]
No Drop 2/
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
MS COCO, and natural language processing on GLUE benchmark and SQuAD.
The weight tuning method is the same with Nagel et al.
; Li et al.
(2021a).
Each block or layer output is reconstructed for 20k iterations.
For ImageNet dataset, we sample 1024 images as calibration set, while
COCO we use 256 images.
In NLP, we sample 1024 examples.
We also keep the first and the last layer in 8-bit except NLP tasks and adopt per-channel weight quantization.
We use W4A4 to represent 4-bit weight and activation quantization.
More model choices and other settings is described in Appendix E.
4. A BLATION STUDY
Effect of QD ROP .
We propose QD ROP and here we would like to test the effect of PTQ with or without QD ROP .
We use ImageNet classification benchmark and quantize the weight parameters to 2-bit and quantize activation to 2/3-bit.
As shown in Table 2, QD ROP improves the accuracy across all bit settings evaluated for 6 models on ImageNet.
Furthermore, the gains are more obvious when applying QD ROP to lightweight network architecture: 4.69% increment for MobileNetV2 under
W2A3 and 14.13% for RegNet-3.2GF with W2A2.
Effect of Dropping Probability.
We also explore the dropping probability in PTQ.
We choose p in [0,0.25,0.5,0.75,1] and test on ResNetand MobileNetV2.
The results are summarized in Fig.
We find 0.5 generally performs best among 5 candidates.
Although there could be a fine-grained best solution for each architecture, we shall avoid cumbersome hyper-parameter search and continue using 0.5.
4. L ITERATURE COMPARISON
ImageNet.
We choose ResNet-18 and -50 (He et al., 2016), MobileNetV2 (Sandler et al., 2018) in
ImageNet comparison.
Besides these common architectures, we also compare the searched MNasNet (Tan et al., 2019) and RegNet (Radosavovic et al., 2020).
Some baseline methods do not report the low bit cases, therefore we implement them based on the open-source codes with unified settings.
We summarize the results in Table 3.
First, the W4A4 quantization is investigated.
It can [Table] [Table] be observed that QD ROP provides 0 ∼ 3 % accuracy uplift when compared to strong baselines including AdaRound, B RECQ , AdaQuant.
With W2A4 quantization, QD ROP can improve the accuracy of ResNet-50 by 0.5%, and RegNet-3.2GF by 4.6%.
In addition, to fully exploit the limit of
QD ROP , we conduct more challenging cases which quantize the weights and activations to 2 orbits.
According to the last two rows of Table 3, our proposed QD ROP consistently achieves good results while existing methods suffer from non-negligible accuracy drop.
In W3A3 quantization, our method can quantize ResNet-50 to 72.4% accuracy while the best existing work only has 69% accuracy.
On MobileNetV2, the difference is even larger, where our method reaches 58% accuracy and
B only gets 23%.
In the W2A2 setting, the PTQ becomes much harder.
QD ROP outperforms the competing method by a large margin: 12.18% upswings for ResNet-18, 29.66% for ResNetand 51.49% for RegNet-3.2GF.
MS COCO.
In this part, we validate the performance of QD ROP on object detection task using MS
COCO dataset.
We use both two-stage Faster RNN (Ren et al., 2015) and one-stage RetinaNet (Lin et al., 2017) models.
Backbone are selected from ResNet-18, ResNet-50, and MobileNetV2.
Note that we set the first layer and the last layer to 8-bit and do not quantize the head of the model, however, the neck (FPN) is quantized.
Experiments show that W4A4 quantization using QD ROP nearly do not affect Faster RCNN’s mAP.
For RethinaNet, our method has 5 mAP improvement on
MobileNetV2 backbone.
In low bit setting W2A4, our method shows great improvement both on
Faster-RCNN and RetinaNet, up to 6.5 mAP.
GLUE benchmark and SQuAD.
Transformer has made significant progress in the language processing field.
And Bert, as a representative baseline in NLP, is getting larger and many compression [Table] works have been proposed.
We test QD ROP in NLP tasks including the GLUE benchmark and
SQuAD1.1.
They are all conducted on the typical NLP model, i [Table] e, BERT (Devlin et al., 2018)
[Table] Compared with those QAT methods (Zafrir et al., 2019; Bai et al., 2020), which usually adopt data augmentation trick to achieve dozens of times the original data, we only randomly extractexamples without any extra data processing and quantize all the parts of BERT while some works (Zafrir et al., 2019) ignore quantization of attention probability.
Besides AdaQuant and B RECQ , which suffer a huge accuracy degradation, our QD ROP surpasses No Drop all the tasks, specifically on QNLI(8.7%), QQP(4.6%) and RTE(7.2%).
As for SST-2, despite little enhancement by dropping quantization, it is indeed close to the FP32 value within 4.4%.
And for STS-B, we argue that the original fine-tuned model is trained with limited data, which might not be very representative to show the power of our method.
4. R OBUSTNESS OF QD ROP
In this part, we discuss the effectiveness of QD ROP under more challenging situations including even less data and cross-domain ones.
Concerning about size of calibration data, we consider another 4 options.
It can be observed that dropping some quantization behaves better under each setting and is even comparable with No Drop with half of the original calibration data.
Motivated by Yu et al.
(2021), we also reconstruct block output by 1024 examples from out-of-domain data, i.e, CIFAR100 (Krizhevsky et al., 2009), MS COCO, and test on ImageNet.
Results are available in Table 6, where our QD ROP still works steadily.
It is worth noting that on MS COCO W4A4, it actually reaches the accuracy from calibrating by in-domain data with and without dropping quantization, which implies the similar data distribution between MS COCO and ImageNet.
[Figure]
Calibration
Bits (W/A) No Drop QD ROP MS COCO 4/ 69.
MS COCO 3/ 65.
CIFAR 4/ 56.
[Table]
C ONCLUSION
In this paper, we have introduced QD ROP , a novel mechanism for post-training quantization.
QDrop aims to achieve good test accuracy given a tiny calibration set.
This is done by optimization towards a flat minima.
We dissect the PTQ objective theoretically into a flatness problem and improve the flatness from a general perspective.
QD ROP is easy to implement, computationally fast, and can be combined with existing methods to achieve further improvement.
We comprehensively verify the effectiveness of QD ROP on a large variety of tasks.
It can achieve a nearly lossless 4-bit quantized network and can significantly improve the 2-bit quantization results, establishing a new state of the art for post-training quantization.
A A CTUAL MEANING OF NOISE u
As mentioned in Sec. 2, we refer the quantizer to ˆ a = ⌊ as ⌉ · s , where s is the step size that can be learned or be determined by collecting activation distribution in PTQ.
By marking the rounding error as c , which subjects to U [ − 0 . 5 , 0 . 5] , the additive noise e = ˆ a − a can be denoted as c · s .
However, this traditional noise does not decouple the noise from step size s and the range of activations. The range of noise e will vary when the range of activations change, making it complicated to analyze across the whole network in a unified form. Some existing papers (Jiang et al., 2019) also indicate the problem that the additive noise doesn’t take parameters’ magnitude into account and suggest a multiplicative form form (Keskar et al., 2016). To be specific, they claim that:
Perturbing the parameters without taking their magnitude into account can cause many of them to switch signs. Therefore, one cannot apply large perturbations to the model without changing the loss significantly. One possible modification to improve the perturbations is to choose the perturbation magnitude based on the magnitude of the parameter. In that case, it is guaranteed that if the magnitude of perturbation is less than the magnitude of the parameter, then the sign of the parameter does not change.
To eliminate the influence induced by the range of activations, we take the noise of multiplicative form: ˆ a = (1 + u ) · a .
In the quantization background, we denote the fixed-point integer value with respect to a as ¯ a ( a = (¯ a + c ) · s and ˆ a = ¯ a · s ).
And then u can be denoted as:
[Equation]
[Equation]
To be noted, u is derived from the definition of the quantizer and can be equivalently transformed from the generalized noise here. From this formulation, we can find that the range of u is not related to the activation range or step size s and is only influenced by the rounding error and the bit-width. In a word, the multiplicative form has its pysical meaning in quantization background and is beneficial as discussed above. B P ROOF OF L EMMA 1 AND T HEOREM
We demonstrate them by considering the situation of (1) fully connected and (2) convolutional networks separately.
As transformation with fully connected ones has been revealed in main body, we add some extra illustration here and mainly target at the convolutional layers.
Here, we give the proof of Eq. (5) by leveraging the definition of FC layers in Eq. We first look at each input sample and temporarily omit x in the notation below for simplicity.
With activation noise u ,
[Equation]
By taking V ( ℓ ) i,j = u ( ℓ ) j , we have [Equation] Thus, with proper constructed v , the noise on activation ( u ) can be viewed as the perturbation on weight ( v ). For every layer, we can conduct this transformation from u to v .
Therefore, considering operating on the quantized weight and calibration data, optimizing E x ∼D c
[ L ( ˆ w , x , 1 + u ( x ))] can be approximated to optimizing E x ∼D
c
[ L ( ˆ w ⊙ 1 + v ( x )) , x , 1 ] . Then we have
E [Equation]
c [
[Equation]
x
[Equation]
Now, Lemma 1 is proved for the case of fully connected networks.
B. C ONVOLUTIONAL NETWORKS
We first look at each input sample and temporarily omit x in the notation below for simplicity.
One layer in the network G can be interpreted as:
[Equation]
where the f ( · ) is the activation function, ( p, q ) pair represents for one element in weight matrix and ( i, j ) pair represents for one element in activation matrix.
Due to complicated design of convolution structure, based on G we introduce two networks G 1 , Gto make proofs more clearly.
Definition 1.
G 1 means inserting random noise on activations, G 2 means sticking random variables into weight.
[Equation]
Definition 2. We mark losses of network G 1 and G 2 in the following way:
[Equation]
With the definitions
[Equation]
we first prove that G 1 and G 2 share common parts in their first-order derivative to its noise in Lemma
[Equation]
Lemma 2.
Assuming the same weight and taking u = [-
-]0 and v = 0 for G 1 and G 2 , we have:
[Equation]
where
[Equation]
Note that with u = 0 and v = 0 , the activations are the same for the two networks so we don’t use different notations here.
Proof.
[Equation]
Because of taking the same weight, and calculating the derivatives to U ( ℓ ) i,j at u = 0 and derivatives to V ( ℓ ) i,j at v = 0 , we have the same activation values for these two networks.
Therefore, Eq.
(21) holds:
[Equation]
By marking the above value as T ( ℓ ) ( i,j ) , ( p,q )
[Equation]
we can get Lemma 2. Based on Lemma
[Equation]
we further derive Theorem 2 as the pre-condition of the final Lemma 1.
Theorem 2. By taking [Equation] [Equation]
we have
[Equation]
[Equation]
Proof.
According to Lemma [Equation] By taking V [Equation]
Thus Theorem 2 is affirmed. We now prove Lemma 1 equipped with Taylor Expansion technique and Theorem [Equation] Proof.
First, by adopting Taylor Expansions at u = 0 , we can get that:
[Equation]
Then, according to Theorem
[Equation]
the above equation can be rewritten as:
[Equation]
Again, by adopting Taylor Expansions at v = 0 for the right part of Eq.
[Equation]
we arrive at: [Equation]
Finally, apply expectation on Eq. [Equation] and Lemma 1 is proved:
[Equation]
With the Lemma 1 proved [Equation]
we can easily derive Theorem 1 by the following transformation: [Equation]
[Table] C E XPERIMENTS C. S UPPLEMENTARY EXPERIMENTS OF S EC . 3.
To further illustrate that different optimization weight is the key point for accuracy of post-training quantization, we conduct some ablation studies that replacing activation quantization or weight one with setting in quantization-aware training, which usually behaves well, respectively.
In detail, by altering the activation quantization with QAT training, we use the whole ImageNet and the LSQ (Esser et al., 2019) scheme to learn the step size of activation for 5 epochs.
And for weight one, we still keep rounding up or down strategy, which is adopted in PTQ but with the whole training set and 40 optimization epochs.
However, results in Table 8 are surprising that although the optimization space in weight is kept very restricted, leveraging the whole data to implement weight tuning can achieve a large accuracy boost.
And the activation quantization step size might not be as important as weight.
These findings indicate that exploring a quantization-friendly weight may be the fresh insight for accurate posttraining quantization, where this work dedicates to.
In the next part, we will further discuss about it from a flatness perspective.
C. F LATNESS AND P OST -T RAINING Q UANTIZATION
While there are plenty of works exploring the correlation between flatness and generalization and devoting to fruitful techniques to achieve a better loss landscape, the interaction between quantization and flatness has not been exploited much.
In this paper, we first connect flatter weight with activation quantization from PTQ view, as implied in Sec. 3.2.
From this, we conjecture that flatness and quantization might be helpful to each other.
By leveraging some mechanisms such as (Izmailov et al., 2018; Foret et al., 2020), FP32 models with smoother loss surface are adopted to validate the performance on quantization compared with the naive one.
In Table 7, model SWA 20 is enabled by applying the SWA technique (Izmailov et al., 2018) to ResNet-18 and finetuning 20 epochs on the whole ImageNet.
From the table, the observation is that the promotion about generalization can not fully reveal the enhancement induced by SWA about quantization.
With even lower bits thus larger noise, SWA 20 surpasses the naive one by a large margin.
The above experimental results imply that distinct FP32 models with analogous accuracy might contribute to surprisingly disparate outcomes after PTQ, especially for those naive mechanisms without any weight tuning.
In turn, there have been some studies that delve into robustness boost by applying quantization.
(Fu et al., 2021) advocates that quantization can be properly leveraged to enhance DNNs’ robust- ness, even beyond their full-precision counterparts.
They propose a random bit training strategy to accomplish it, which our work illustrates the correlation with bit and perturbation in Appendix A. C. SUPPLEMENTARY EXPERIMENTS OF S EC . 3.
Overfitting phenomenon.
To further check the overfitting phenomenon as discussed in Sec.
3.3, we take ResNet-18 W2A2 as an example and compare the test accuracy and train accuracy among 3 Cases and QD ROP .
Results are listed in Table 9. It can be seen that with extremely low-bit quantization, for Case 2 and Case 3, although accuracy on test data drops a lot, performance on train data remains high. Recall that introducing activation quantization adds an extra term where the noise on weight v ( x ) varies with input data. Combined with the experimental results, we argue that it might be easy to cause some overfitting phenomenon on tuning Term (7-2) in Theorem 1 with only 1024 calibration samples.
The two accuracy of QD ROP also confirmed this phenomeon.
Although QD ROP sacrifices some performance on train data, it covers more diverse directions of perturbation from flatness perspective and is more likely to be flatter on test data.
As for Case 1, which doesn’t take Term (7-2) into account, naturally behaves worst with low-bit activation quantization.
Other choices for perturbation type.
Rethinking Sec. 3.3, there are numerous tricks to promote the flatness on the test set, such as using data augmentation for diverse x , modifying v ( x ) directly even like (Foret et al., 2020), which injects the worst-case weight perturbations or adjusting u ( x ) to acquire v ( x ) from another distribution.
In this part, we inspect other strategies and discover that they all cause some performance drop to some extent except the data augmentation trick.
We hypothesis that it is made by increasing the objective loss described in Eq.
(4), where for asymmetric weight reconstruction, the increased weird loss in one block induced by employing these tricks will be amplified to subsequent ones and lead to a failure.
However, our method, which tries to drop some quantization and keep it in FP32 signal, masks some reconstruction loss thereby has no such worries [Table] [Table] Hessian information.
As Hessian information is known to be a metric to characterize flatness property (Keskar et al., 2016; Dong et al., 2019) [Table] we also calculate the top-1, top-5 Hessian eigenvalues ( λ 1 , λ 5 ) and the Hessian trace ( Tr ) among 3 Cases and QD ROP to better support our analysis.
In table Table
[Table]
QD ROP has the smallest value of Hessian information, which matches with our theoretical framework and observations of loss landscape
[Table] [Table] D R ELATED WORKS Post-training quantization Rounding-to-nearest operation is known to be the direct and easy way for quantizing parameters or activations in PTQ.
Although there is almost no accuracy drop when quantizing to 8-bit, lower bit quantization is yet a hard task and worth exploring.
(Choukroun et al., 2019) transforms quantization to a Minimum Mean Squared Error problem both for weights and activations.
(Nagel et al., 2019) equalizes weight ranges among channels thus be more favorable to per-layer quantization and employs bias correction to absorb the output error induced by quantization. However, such methods neglect the task loss thus lead to a sub-optimal. AdaRound (Nagel et al., 2020), which proposes to learn the rounding mechanism by reconstructing output layer by layer brings more opportunities for 4-bit quantization. Besides layer reconstruction, B RECQ (Li et al., 2021a) discusses more choices and advises to do block reconstruction with better accuracy at 2-bit weight quantization.
Nonetheless, we argue that AdaRound and B RECQ isolate weight quantization and activation one theoretically and experimentally, which might be a key point of failures on extremely low-bit quantization. Flatness The idea of “flat” minima might date back to (Hochreiter & Schmidhuber, 1997), where the benefits are recognized in recent years, such as generalization (Jiang et al., 2019; Keskar et al., 2016) and adversarial training (Wu et al., 2020; Zheng et al., 2021). Some previous works (Izmailov et al., 2018; Foret et al., 2020) devote to improve the flatness of the trained weight for robustness under perturbation or distribution shift. Other ideas try to model flatness or sharpness formally by visualization of loss landscape or mathematical formulas.
And for quantization, which could be viewed as some kind of noise, a flat model has been implied to be preferable, (Dong et al., 2019;
Yang et al., 2019; Kadambi et al., 2020). Despite the natural fact that flatness helps with weight quantization, how does activation quantization involves with smoother loss surface has not been discussed deeply, particularly for post-training quantization. In this work, we introduce noise scheme by randomly dropping activation quantization and achieve a general flatness. Another paper (Fan et al., 2020) also utilizes randomness by adding noise to weight for the simulation of weight quantization.
But they target at reducing the induced bias of Straight Through Estimation (STE) in QAT and also have different motivations and solve different problems from us.
E I MPLEMENTATION DETAILS
Observation.
Here, we give the concrete implementation of experiments in Sec. 3.1.
We actually consider three ways of introducing activation for Case 2, but we find the difference of outcomes among them are negligible thus employing one of them for clearer clarification.
ImageNet.
We extract 1024 training examples from ImageNet dataset as calibration dataset based on the standard pre-process.
Pretrain-models are downloaded from B RECQ ’s open source code.
Hyper-parameters we keep it as B RECQ , such as set batch size 32, learning rate for activation step size 4e-5, learning rate for weight tuning 1e-3, iterations 2x104.
Following B RECQ , we first fold batch normalization layer into convolution before tuning the weight then reconstruct output block- wise, where we tune the weight by rounding up or down, use LSQ (Esser et al.
, 2019)to parameterize activation step size.
As for our extra content, we learn the weight and activation parameters together and use 50% rate to drop some activation quantization.
Object detection.
Here, we also obey B RECQ ’ setting and use the same pretrain-models withtraining samples taken from MS COCO dataset for calibration.
Parameters about resolution is set to 800 (max size 1333) and 600 (max size 1000) for ResNets and MobileNetV2, respectively and batch size is set to 2 while others are the same as classification task.
To be noted, we didn’t quantize the head with applying block reconstruction to backbone and layer reconstruction to neck like B RECQ .
GLUE benchmark and SQuAD.
The BERT fine-tuned models are taken from huggingface group (https://huggingface.co/).
And we sampled 1024 examples from training set.
We keep the maximum sequence length to be 128 for GLUE benchmark but maximum sequence length 384 with doc stride 128 for SQuAD1.1, which is the usual choice.
Also, we quantize all the part in BERT as well as the internal structure of the attention module with only affirming the embedding weight to 8-bit.
Other settings and hyper-parameters are chosen in the same way as ImageNet experiments.
Baselines.
We run baseline methods from open-source codes, such as AdaQuant (Hubara et al., 2020), B RECQ and Adaround.
And we try our best to align some optimal settings like per-channel quantization for fair comparisons.