B RECQ : P USHING THE L IMIT OF P OST -T RAINING Q UANTIZATION BY B LOCK R ECONSTRUCTION
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the challenging task of neural network quantization without end-toend retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantized models than quantization-aware training (QAT).
In this work, we propose a novel PTQ framework, dubbed B RECQ , which pushes the limits of bitwidth in PTQ down to INT for the ﬁrst time.
B RECQ leverages the basic building blocks in neural networks and reconstructs them one-by-one. In a comprehensive theoretical study of the second-order error, we show that B RECQ achieves a good balance between crosslayer dependency and ﬁrst-order approximation.
To further employ the power of quantization, the mixed precision technique is incorporated in our framework by approximating the inter-layer and intra-layer sensitivity.
Extensive experiments on various handcrafted and searched neural architectures are conducted for both image classiﬁcation and object detection tasks.
And for the ﬁrst time we prove that, without bells and whistles, PTQ can attain 4-bit ResNet and MobileNetV comparable with QAT and enjoy 240 × faster production of quantized models.
I NTRODUCTION
The past decade has witnessed the rapid development of deep learning in many tasks, such as computer vision, autonomous driving, etc.
However, the issue of huge computation cost and memory footprint requirements in deep learning has received considerable attention.
Some works such as neural architecture search (Zoph & Le, 2016) try to design and search a tiny network, while others, like quantization (Hubara et al., 2017), and network pruning (Han et al., 2015) are designed to compress and accelerate off-the-shelf well-trained redundant networks.
Many popular quantization and network pruning methods follow a simple pipeline: training the original model and then ﬁnetune the quantized/pruned model.
However, this pipeline requires a full training dataset and many computation resources to perform end-to-end backpropagation, which will greatly delay the production cycle of compressed models. Besides, not all training data are always ready-to-use considering the privacy problem.
Therefore, there is more demand in industry for quantizing the neural networks without retraining, which is called Post-training Quantization.
Although PTQ is fast and light, it suffers from severe accuracy degeneration when the quantization precision is low.
For example, DFQ (Nagel et al., 2019) can quantize ResNet-18 to 8-bit without accuracy loss (69.7% top-1 accuracy) but in 4-bit quantization, it can only achieve 39% top-1 accuracy.
The primary reason is the approximation in the parameter space is not equivalent to the approximation in model space thus we cannot assure the optimal minimization on the ﬁnal task loss.
Recent works like (Nagel et al., 2020) recognized the problem and analyzed the loss degradation by
Taylor series expansion.
Analysis of the second-order error term indicates we can reconstruct each layer output to approximate the task loss degeneration.
However, their work cannot further quantize the weights into INT2 because the cross-layer dependency in the Hessian matrix cannot be ignored when the perturbation on weight is not small enough.
In this work, we analyze the second-order error based on the Gauss-Newton matrix.
We show that the second-order error can be transformed into network ﬁnal outputs but suffer from bad approximation error.
To achieve the best tradeoff, we adopt an intermediate choice, block reconstruction.
In addition, our contributions are threefold:
Based on the second-order analysis, we deﬁne a set of reconstruction units and show that block reconstruction is the best choice with the support from theoretical and empirical evidence.
We also use Fisher Information Matrix to assign each pre-activation with an importance measure during reconstruction. We incorporate genetic algorithm and the well-deﬁned intra-block sensitivity measure to generate latency and size guaranteed mixed precision quantized neural networks, which fulﬁlls a general improvement on both specialized hardware (FPGA) and general hardware (ARM CPU).
We conduct extensive experiments to verify our proposed methods.
We ﬁnd that our method is applicable to a large variety of tasks and models.
Moreover, we show that post-training quantization can quantize weights to INT2 without signiﬁcant accuracy loss for the ﬁrst time.
P RELIMINARIES
Notations Vectors are denoted by small bold letters and matrices (or tensors) are denoted by capital bold letters. For instance, W and w represent the weight tensor and its ﬂattened version. Bar accent denotes the expectation over data points, e.g. ¯ a .
Bracketed superscript w ( (cid:96) ) indicates the layer index.
For a convolutional or a fully-connected layer, we mark its input and output vectors by x and z .
Thus given a feedforward neural network with n layers, we can denote the forward process by
[Equation]
where h ( · ) indicates the activation function (ReLU in this paper).
For simplicity, we omit the analysis of bias b ( (cid:96) ) as it can be merged into activation.
|| · || F denotes the Frobenius norm.
Quantization Background Uniform symmetric quantization maps the ﬂoating-point numbers to several ﬁxed-points.
These points (or grids) have the same interval and are symmetrically distributed.
We denote the set that contains these grids as Q u,sym b =
[Equation] [Equation]
[Equation] [Equation]
Here, s is the step size between two grids and b is the bit-width.
Quantization function, denoted by q ( · ) :
R → Q u,sym b , is generally designed to minimize the quantization error:
[Equation]
Solving this minimization problem, one can easily get the q ( · ) by leveraging the rounding-to-nearest operation (cid:98)·(cid:101) .
Rounding-to-nearest is a prevalent method to perform quantization, e.g. PACT (Choi et al., 2018).
However, recently many empirical or theoretical evidence support that simply minimizing the quantization error in parameter space does not bring optimal task performances.
Speciﬁcally,
Esser et al.
(2020) propose to learn the step size s by gradient descent in quantization-aware training (QAT).
LAPQ (Nahshan et al., 2019) ﬁnds the optimal step size when the loss function is minimized without re-training the weights.
Their motivations are all towards minimizing a ﬁnal objective, which is the task loss, i.e.,
[Equation]
While this optimization objective is simple and can be well-optimized in QAT scenarios, it is not easy to learn the quantized weight without end-to-end ﬁnetuning as well as sufﬁcient training data and computing resources.
In post-training quantization settings, we only have full precision weights that w (cid:63) =
arg min w E [ L ( w )] where w ∈ R and a small subset of training data to do calibration.
Taylor Expansion It turns out that the quantization imposed on weights could be viewed as a special case of weight perturbation.
To quantitatively analyze the loss degradation caused by quantization,
Nagel et al.
(2020) use Taylor series expansions and approximates the loss degradation by
[Equation]
where ¯ g ( w ) =
E [ ∇ w L ] and ¯ H ( w ) =
E [ ∇ 2 w L ] are the gradients and the Hessian matrix and ∆ w is the weight perturbation.
Given the pre-trained model is converged to a minimum, the gradients can be safely thought to be close to 0 . However, optimizing with the large-scale full Hessian is memoryinfeasible on many devices as the full Hessian requires terabytes of memory space.
To tackle this problem, they make two assumptions:
Layers are mutual-independent, thus the Hessian is in the form of layer-diagonal 1 and Kroneckerfactored, i
e., ¯ H ( w ( (cid:96) ) )
=
E [ x ( (cid:96) ) x ( (cid:96) ) , T ⊗ H ( z ( (cid:96) ) ) ] , where ⊗ is the Kronecker product.
The second-order derivatives of pre-activations are constant diagonal matrix ( H ( z ( (cid:96) ) )
=
c
× I ) which is independent of input data points.
At last, the objective is transformed into a practical proxy signal, the change in feature-maps ( z = Wx ) , and the quantized model can be obtained by a layer-by-layer feature map reconstruction
algorithm (with few calibration images).
Recent works, like Bit-Split (Wang et al., 2020) and AdaQuant (Hubara et al., 2020), also take this layer-wise objective to improve the post-training quantization.
However, they failed to quantize weights to INT2.
We think the inherent reason is that when ∆ w grows higher, the former assumptions do not hold and an accurate signal is required.
P ROPOSED M ETHOD 3. C ROSS - LAYER D EPENDENCY
Denote the neural network output z ( n ) =
f ( θ ) , the loss function can be represented by L ( f ( θ ))
where θ = vec[ w (1) , T , . . .
, w ( n )
, T ] T is the stacked vector of weights in all n layers.
The Hessian matrix can be computed by
[Equation]
where z ( n ) ∈ R m .
Since the pretrained full precision model is converged to a local minimum, we can assume the Hessian is positive-semideﬁnite (PSD).
Speciﬁcally, the converged model has ∇ z ( n ) L close to
so the ﬁrst term in Eq.
(5) is neglected and Hessian becomes the Gauss-Newton (GN) matrix G ( θ ) .
GN matrix can be written in matrix form (Botev et al., 2017) as
[Equation]
where J z ( n ) ( θ ) is the Jacobian matrix of the network output with respect to the network parameters.
However, in practice, we cannot explicitly compute and store the Jacobian for each input data point in such a raw form.
To reduce the computation and memory budget, we will transform the secondorder error into the network output, as shown in the following theorem.
Theorem 3.1.
Consider an n -layer feedforward neural network with ReLU activation function.
Assuming all weights are quantized, the second-order error optimization can be transformed by:
[Equation]
Remark 3.1.
The same transformation is also applicable for activation quantization.
The quadratic loss is deﬁned as E [∆ γ T H ( γ ) ∆ γ ] where ∆ γ = vec[∆ x (1) , T , . . .
, ∆ x ( n ) , T ] T .
We prove the theorem using the quadratic form, details can be found in Appendix A.1.
Here we provide a sketch of the proof by matrix form.
The product of perturbation and Jacobian can be thought as the ﬁrst-order Taylor approximation of the change in network output ∆ z ( n ) :
[Equation]
Therefore, combining Eq.
(8) and Eq.
(6) we can transform the large-scale second-order error into the change in network outputs characterized by the output Hessian H ( z ( n ) ) .
The theorem indicates a simple observation, suppose a well-trained teacher model and an initialized student model, we can minimize their discrepancy by reconstructing the network’s ﬁnal output z ( n ) , which coincides with and generalizes the distillation (Hinton et al., 2015; Polino et al., 2018).
LAPQ (Nahshan et al., 2019) also considers the dependency but their optimization does not rely on second-order information.
However, we should emphasize that distillation requires the same computation and data resources as in normal training procedure, which is impractical for PTQ with limited data.
In fact we ﬁnd the network output reconstruction is worse than the layer-by-layer reconstruction in PTQ.
The primary reason for this is approximating the ﬁrst-order error with the ﬁnal output could cost signiﬁcant approximation error (cf.
Eq.
(8)) if we consider all layers in the neural networks, as we show in the following theorem: Theorem 3.2.
Consider the above feedforward neural network we analyzed, if all the layers are quantized, then we have
[Equation]
where f ( θ ) means the output of the network.
ˆ θ is the quantized weights.
stem Network Structure
Body Structure
Stage Structure
Block Structure (w, h, 3) body (w/2, h/2, c 0 ) head (w/32, h/32, c 4 ) (1, 1, #classes) stage(w/2, h/2, c 0 ) stage(w/4, h/4, c 1 ) stage... ...
(w/32, h/32, c 4 ) block(w, h, c i ) block(w/2, h/2, c i+1 ) block n ... ...
(w/2, h/2, c i+1 ) layer(w, h, c i ) layer(w, h, c i *e) layer(w, h, c i *e) (w, h, c i ) + layer-diagonal block-diagonal
Full matrix
Proof is available in Appendix A.2.
It can be seen that the bound of difference between ﬁrst-order error and the ﬁnal output will scale up with the number of layers we considered .
In other words, there is a trade-off problem.
For one thing, we should square up the off-diagonal loss in the Hessian, therefore according to Theorem 3.1, we should consider every cross-layer dependency and optimize the whole quantized network to minimize the ﬁnal output of the network.
But for another, using the ﬁnal output of the whole network to replace the ﬁrst-order error will lead to biased estimation and potentially harm the whole reconstruction process, as described in Theorem 3.2.
From this perspective, to completely eliminate the ﬁrst-order approximation error, we should instead use the layer-wise optimization as mentioned in Sec. 2.
The theoretical analysis inspires us that both network-wise and layer-wise reconstruction have their own drawbacks. There should be a better choice to conduct reconstruction at an intermediate granularity between network-wise and layer-wise optimization. 3. B LOCK R ECONSTRUCTION
The layer-wise optimization corresponds to layer-diagonal Hessian (Fig.
1b blue parts) and the network-wise optimization corresponds to full Hessian (Fig.
1b green parts).
Similarly, we can deﬁne an intermediate block-diagonal Hessian.
Formally, if layer k to layer (cid:96) (where 1 ≤ k < (cid:96) ≤ n ) form a block, the weight vector is deﬁned as ˜ θ = vec[ w ( k ) , T , . . .
, w ( (cid:96) ) , T ] T and the Hessian can be also transformed by ∆˜ θ T ¯ H
(˜ θ ) ∆˜ θ
= E [∆ z ( (cid:96) ) , T H ( z ( (cid:96) ) ) ∆ z ( (cid:96) ) ] . Such block-diagonal Hessian ignores the inter-block dependency and considers the intra-block dependency but its estimation is more accurate than the full Hessian.
Then we can block-by-block reconstruct the intermediate output.
To this end, we deﬁne 2 kinds of intermediate reconstruction granularity : 1.
Stage-wise reconstruction : consider every intra-stage loss and reconstruct each stage’s output by optimizing all parameters in that stage and 2.
Block-wise reconstruction : consider every intra-block loss, ignore inter-block loss and optimize each block with its output 2 .
The relationship between network, stage, block, and layer is illustrated in Fig.
1a.
We test these 4 kinds of reconstruction granularity and ﬁnd that block-wise optimization outperforms others .
We think this is because the main off-diagonal loss in the Hessian is concentrated in each block, as Fig.
1b orange part illustrated, while the inter-block loss is small and can be ignored in the optimization.
Also, the stage-wise or net-wise optimization suffers from the bad estimation of the ﬁrst-order error and degenerate the ﬁnal performances.
We report the quantitative comparison in Sec. 4.1.
We name our algorithm B RECQ , because we choose block as our base reconstruction unit.
It is necessary to point out that our analysis does not give the optimal conﬁguration of the reconstruction granularity.
The choice of block-wise optimization comes from our experiments and we ﬁnd this choice has two merits.
(1) No hyper-parameters included and (2) applicable for all models and all tasks we tested.
3. A PPROXIMATING P RE - ACTIVATION H ESSIAN
With block-diagonal approximated Hessian matrix, we can measure the cross-layer dependency inside each block and transform any block’s second-order error to the output of this block
E [∆ z ( (cid:96) ) , T H ( z ( (cid:96) ) )
∆ z ( (cid:96) ) ] . This objective requires the further computation of the knowledge in the rest of the network, i.e., pre-activation Hessian H ( z ( (cid:96) ) ) .
One way is to follow Nagel et al.
and assume H ( z ( (cid:96) ) )
=
c × I .
Therefore the quadratic loss becomes || ∆ z ( (cid:96) )
|| 2 .
This method might be easy to implement but lose too much information.
We use the diagonal Fisher Information Matrix (FIM) to replace the pre-activation Hessian.
Formally, given a probabilistic model p ( x | θ ) , the FIM is deﬁned as:
[Equation]
The FIM is equal to the negative expected Hessian of the log-likelihood function, therefore, a simple corollary is that the Hessian of task loss will become FIM if the model distribution matches the true data distribution (LeCun et al., 2012).
Although matching true data distribution seems impossible, this is the best we can do since the pretrained model is converged.
The FIM is also related to the asymptotic Kullback-Leibler divergence, see Appendix A.3 for more details.
The diagonal of the pre-activation FIM is equal to the squared gradients of each elements, which is successfully used in Adam (Kingma & Ba, 2014) for the second momentum.
The optimization objective becomes
[Equation]
Compared with the MSE minimization, the above minimization incorporates the squared gradient information.
If the output has higher absolute gradients, it will receive more attention when being reconstructed.
A similar method for pruning the pre-activation has been proposed in Theis et al.
Note that B RECQ is compatible with any optimization method, like STE (Hubara et al., 2017).
Here we adopt adaptive rounding (Nagel et al., 2020) for weights and learned step size (Esser et al., 2020) for activation step size because we observe they generally perform better in PTQ, see details in Appendix C.4.1.
We formulate the overall calibration algorithm for a uniﬁed precision model in algorithm 1.
We should emphasize that we only need a small subset (1024 in our experiments) of the whole training dataset to calibrate the quantized model.
And we can obtain a quantized ResNetwithin 20 minutes on a single GTX 1080TI GPU.
3. M IXED P RECISION
To further push the limit of post-training quantization, we employ mixed precision techniques, which can be formulated by
[Equation]
Here, c is the bit-width vector with the shape of number of layers.
H ( · ) is a hardware performance measurement function, which is used to ensure the mixed precision model has the same or lower hardware performance (e.g., memory and speed) than a predeﬁned threshold δ .
We choose 2, 4, 8-bit for mixed precision because they are most common in practical deployment.
Regarding the training loss L , we ﬁnd that nearly all existing literature (Cai et al., 2020;
Hubara et al., 2020; Dong et al., 2019) uses layer-wise measurement.
They all assume the total loss can be summed together.
Therefore, the mixed precision problem becomes an integer programming problem.
However, we argue that the loss measurement should contain two parts: diagonal loss and off-diagonal loss, the ﬁrst is the same with previous works and measure the sensitivity of each layer independently, while the off-diagonal loss is used to measure the cross-layer sensitivity.
Theoretically, we should examine all permutations, which results in 3 n possibilities and prohibits the search algorithm.
Our ﬁrst attempt is to reduce the off-diagonal loss into the block-level as we mentioned that the Hessian can be approximated to a block-diagonal matrix.
Granted, we still ﬁnd the search space is large, for example, if a block has four layers, then we have to consider the 3 4 =permutations for a single block.
Based on our preliminary experiments, we ﬁnd that 4-bit and 8bit quantization nearly do not drop the ﬁnal accuracy.
Hence we only take 2-bit permutations into consideration and drastically reduce the search space.
We use genetic algorithm (Guo et al., 2020) to search the optimal bitwidth conﬁguration with hardware performance threshold, the algorithm is located in algorithm 2.
Due to space limits, we put related works in Appendix B. Readers can refer to related works for a brief discussion on quantization and second-order analysis.
E XPERIMENTS
In this section, we report experimental results for the ImageNet classiﬁcation task and MS COCO object detection task.
We provide our source codes in Google Drive anonymously.
The detailed implementation of the experiments can be found in the Appendix C.4.4.
The rest of this section will contain ablation study on reconstruction granularity, classiﬁcation and detection results, mixed precision results and comparison with quantization-aware training.
In Appendix C, we conduct more experiments, including the impact of the ﬁrst and the last layer, the impact of calibration dataset size and data source.
4. A BLATION S TUDY
We test four kinds of reconstruction granularity: Net-wise, Stage-wise, Block-wise, and Layer-wise
[Table]
reconstruction
[Table]
We conduct experiments using
MobileNetV2 and ResNet-18 with 2-bit weight quantization for all layers except for the ﬁrst and the last layer.
It can be seen from Table 1 that
Block-wise optimization outperforms other methods.
This result implies that the approximation error in net-wise and stage-wise optimization outweighs their off-diagonal loss.
In ResNet-18, we ﬁnd the difference is not signiﬁcant, this can be potentially attributed to that ResNet-18 only haslayers in the body and the block size, as well as the stage size, is small, therefore leading to indistinct results.
4. I MAGE N ET
We conduct experiments on a variety of modern deep learning architectures, including ResNet (He et al., 2016) with normal convolution, MobileNetV2 (Sandler et al., 2018) with depthwise separable convolution and RegNet (Radosavovic et al., 2020) with group convolution.
Last but not least important, we also investigate the neural architecture searched (NAS) models, MNasNet (Tan et al., 2019).
In Table 2, we only quantize weights into low-bit integers and keep activations full precision.
We compare with strong baselines including Bias Correction, optimal MSE, AdaRound, AdaQuant, and Bit-split.
Note that the ﬁrst and the last layer are kept with 8-bit.
While most of the existing methods have good performances in 4-bit quantization, they cannot successfully quantize the model into 2-bit.
Our method consistently achieves the lowest accuracy degradation for ResNets (within 5%) and other compact models.
We further quantize activations into 4-bit to make the quantized model run on integer-arithmetic hardware platforms.
We ﬁnd that 4-bit activation quantization can [Table] [Table] have a huge impact on RegNet and MobileNet.
Nonetheless [Table] our methods produce higher performance than other state-of-the-arts. To be noted
[Table]
B RECQ is the ﬁrst to promote the 2W4A accuracy of PTQ to a usable level while all other existing methods crashed
[Table] 4. C OMPARISON WITH Q UANTIZATION - AWARE T RAINING [Table] In this section
[Table]
we compare our algorithm (post-training quantization) with some quantization-aware training methods, including PACT (Choi et al., 2018), DSQ
(Gong et al., 2019), LSQ (Esser et al., 2020)
[Table]
and a mixed precision technique HAQ (Wang et al., 2019).
Table 4 shows that although
B RECQ is a PTQ method with limited available data
[Table]
it can achieve comparable accuracy results with existing quantization-aware training models.
In addition, our method can surpass them in 4bit MobileNetV2 while using less than one training GPU hours.
Our method also has comparable accuracy with HAQ, which is a training-based mixed precision method.
Note that our GPU hours include 3 uniﬁed precision training (2-, 4-, 8-bit respectively) and further mixed-precision training only needs to check the lookup table.
Instead, HAQ would end-to-end search for each hardware performance threshold from scratch.
[Table]
[Figure]
t A
[Figure]
4. MS COCO
To validate the effectiveness of B RECQ on other tasks [Figure] we conduct object detection on the twostage Faster-RCNN (Ren et al., 2015) and the one-stage RetinaNet (Lin et al., 2017).
ResNet-18, 50 as well as MobileNetV2 are adopted as backbones for the detection model.
Results in Tabledemonstrate our method nearly does not drop the performance in 4-bit weight quantization and 8-bit activation.
Even when the weight bit decreases to 2, the model still achieves near-to-original mAP.
4. M IXED P RECISION
In this section [Figure] we test (1) model-size guaranteed mixed precision and (2) FPGA latency guaranteed mixed precision 3 to unleash the potential of mixed precision and further push the limit of
PTQ. We choose ResNet-18, MobileNetV2, and RegNetX-600MF to validate the efﬁcacy of our algorithm.
Note that in this section, we keep activation in 8-bit because we only compare the discrepancy between the uniﬁed and mixed precision in weights.
We omit 3-bit weight quantization in uniﬁed precision because it is usually unfriendly to the hardware.
Latency settings can be found in Appendix C.4.3.
From Fig.
2 we ﬁnd that (1) mixed precision consistently outperforms uniﬁed precision, especially when using extremely low-bit, e.g., up to 10% accuracy increase with the same latency as the 2-bit model.
(2) mixed precision can produce many bit conﬁgurations that can adapt to plenty of hardware requirements while uniﬁed precision can only have 2 ﬁxed models.
C ONCLUSION
In this paper, we propose B RECQ , a post-training quantization framework by analyzing the secondorder error.
We show that the reconstruction of quantization at the block granularity arrives at a good balance of cross-layer dependency and ﬁrst order approximation, especially in 2-bit weight quantization where no prior works succeed to quantize.
B RECQ is compatible with mixed precision and can reduce the search cost.
To our best knowledge, B RECQ reaches the highest performance in post-training quantization and is the ﬁrst to be on a par with quantization-aware training using 4-bit.
A PPENDIX FOR B RECQ A M AIN P ROOFS A. P ROOF OF T HEOREM 3.
Proof.
We will prove the theorem using quadratic form.
Denote the weight vector shape as θ ∈ R d , and the network output vector shape as z ( n ) ∈ R m .
The quadratic form of the ∆ θ T H ( θ ) ∆ θ can be represented by:
[Equation]
where L is the cross-entropy loss.
Based on Eq.
[Equation]
we have
[Equation]
Substituting above equation in Eq
[Equation]
[Equation]
we have
[Equation]
where we deﬁne the J [ xy ] is the Jacobian matrix of x w.r.t.
y .
To this end, we use the ﬁrst-order Taylor expansion as we did in Eq.
(8) to approximate the change in network output, i.e.,
[Equation]
Therefore, the ﬁnal objective is transformed to ∆ z ( n ) , T H ( z ( n ) ) ∆ z [Equation]
n [Equation]
A. P ROOF OF T HEOREM 3.
In order to prove Theorem 3.2, we ﬁrst introduce the following lemma:
Lemma A.1.
Given an n -layer feedforward neural network with ReLU activation, and a single input image x
( 1 ) , we have:
[Equation]
Remark A.1.
The above lemma holds for any activation that satisﬁes h ( x )
=
h (cid:48)
( x ) x where h (cid:48) ( · ) is the derivative, e.g., leaky ReLU.
Proof.
Recall that θ = vec[ w (1) , T , w (2) , T , . . .
, w ( n )
, T ] T is the stacked vector of weights in each layer.
We ﬁrst prove that for weights in any layer, w ( i )
J
[ z ( n ) w ( i ) ] =
z ( n ) holds, then we can prove the lemma by adding the equation for each layer.
Consider the Jacobian matrix J [ z ( i +1)
z ( i ) ] , it can computed by
[Equation]
where B ( i ) =
diag( h (cid:48)
( z ( i ) )) is the gradient of activation function.
Therefore, according to the property of (leaky) ReLU activation h ( x ) = h (cid:48)
( x ) x , we have
[Equation]
For any i < n , i.e., any layer except the last layer, we have J [ z ( n ) w ( i ) ]
=
J [ z ( i ) w ( i ) ]
J [ z ( n ) z ( i ) ] .
The ﬁrst term is equal to x ( i ) .
Hence, we have
[Equation]
For i = n , it is easy to get w ( n ) J
[ z ( n ) w ( n ) ] = w ( n ) x ( n ) =
z ( n ) , therefore:
[Equation]
Proof.
Based on the above lemma, we have ∂f ( θ )
∂θ θ =
nf ( θ ) , where f is the neural network function and f ( θ ) denotes the ﬁnal output.
Therefore, we have
[Equation]
where the last line follows | x + y | ≤ | x | + | y | .
A. Q UADRATIC FORM OF KL DIVERGENCE
In this section, we provide the derivation of quadratic form of the KL divergence.
First, we will show that Fisher is the negative expected Hessian of log-likelihood ( E p ( x | θ )
[ H log p ( x | θ ) ] ) and then show that the Hessian of KL divergence is equal to E p ( x | θ )
[ H log p ( x | θ ) ] .
Consider a probabilistic model parameterized by θ with likelihood function p ( x | θ ) , the FIM is deﬁned as F ( θ ) =
E p ( x | θ ) [ ∇ θ log p ( x | θ ) ∇ θ log p ( x | θ ) T ] .
The Hessian is therefore given by
[Equation]
) p ( x | θ ) − ∇ p ( x | θ ) ∇ p ( x | θ ) T ) p ( x | θ ) p ( x | θ ) (cid:21) (23b) =
E p ( x | θ ) (cid:20) ∇∇ p ( x | θ ) p ( x | θ ) (cid:21) − E p ( x | θ ) (cid:34)(cid:18) ∇ p ( x | θ ) p ( x | θ ) (cid:19) (cid:18) ∇ p ( x | θ ) p ( x | θ ) (cid:19) T (cid:35) (23c) =
(cid:90) ∇∇ p ( x | θ ) p ( x | θ ) p ( x | θ )
d x − E p ( x | θ ) [log p ( x | θ ) log p ( x | θ ) T ] (23d) = ∇∇ (cid:90) p ( x | θ )d x − F ( θ ) (23e) =
∇∇ 1 − F ( θ ) = − F ( θ ) (23f)
Before taking Taylor series expansion to the KL divergence, we will show the ﬁrst- and second-order derivative of the KL divergence:
[Equation]
The Hessian of the KL divergence is given by:
[Equation]
where the last line is the result of Eq.
To this end
[Equation]
we can directly use Taylor series expansion on the KL divergence given a small perturbation ∆ θ :
[Equation] B R ELATED W ORKS B. Q UANTIZATION Quantization can be classiﬁed into two categories: Quantization-aware Training (QAT) and Posttraining Quantization (PTQ).
QAT methods usually train a quantized model from scratch or ﬁnetune it from the pre-trained full precision models.
Rounding ﬂoating-point numbers to ﬁxed-points numbers will produce 0 gradients almost everywhere.
Therefore, most QAT methods employ the
Straight-Through Estimator (STE) for gradients approximation.
Gong et al.
(2019) uses a differentiable tanh function to gradually approach the step function.
Choi et al.
ﬁrst introduces parameterized clipping thresholds to learn it from task loss.
Esser et al.
(2020) propose the Learned
Step Size Quantization by learning the step size with STE.
Apart from uniform quantization, some works like Li et al.
(2019) argue that non-uniform quantization has better performance than uniform quantization while keeping its efﬁciency.
Despite the promising results given by QAT methods, they usually need more than 100 GPU hours to get it.
Meanwhile, we cannot assure the full training data is ready-to-use.
In that case, PTQ plays an important role which is what we focus on in this paper.
Generally, most deep learning models can be safely quantized to 8-bit without re-training.
Data-Free Quantization Nagel et al.
(2019) even do layer-wise 8-bit PTQ without any data.
However, in 4-bit quantization, most parameter space-based methods cannot obtain good performances.
Bias Correction Finkelstein et al.
and LAPQ Nahshan et al.
(2019) narrow the gap but still cannot completely recover the full precision performances in 4-bit quantization.
Recently, Nagel et al.
(2020) propose to do layer-wise calibration and made huge progress in 4-bit quantization.
Our work continues its analysis on Taylor expansion and considers the off-diagonal loss.
Another perspective of quantiﬁcation is the precision allocation scheme.
It is generally believed that each layer has its own redundancy and hardware performances.
Hardware-aware Quantization (HAQ Wang et al.
leverages reinforcement learning to search the optimal bitwidth conﬁguration.
Hessian-aware Weight Quantization (HAWQ) (Dong et al., 2019) utilizes the second-order information to decide the bitwidth.
Mixed precision also appears in PTQ, such as the Pareto frontier method in ZeroQ (Cai et al., 2020) and the Integer Programming method in AdaQuant (Hubara et al., 2020).
However, apart from end-to-end search like HAQ, most methods like HAWQ, ZeroQ, and AdaQuant do not consider the inter-layer sensitivities, i.e., they assume the choice of bit-width is independent of bit-width for other layers.
Our work approximates the inter-layer sensitivity by intra-block sensitivity.
B. S ECOND - ORDER A NALYSIS AND O PTIMIZATION
The history of second-order information in perturbation analysis can be traced to the 1990s like Optimal Brain Surgeon (Hassibi & Stork, 1993; Dong et al., 2017).
The Hessian matrix is essential for pruning and quantization.
As aforementioned, HAWQ uses the largest eigenvalue of Hessian to determine the sensitivity.
Hessian matrix is also important for second-order optimization like
Newton’s method as it consists of the curvature information.
However, calculating the real full Hessian is prohibitive on today’s deep learning architectures because the Hessian will scale up with the number of parameters rapidly.
Therefore, approximations are made to simplify the calculation and make the storage more ﬂexible, e.g., Gauss-Newton optimization with Kronecker-factored recursive approximation Botev et al.
Hessian-Free optimization (Martens, 2010) avoids the explicit computation of the Hessian matrix by solving the linear system g = Hv .
Second-order optimization with FIM is called Natural Gradient Descent (Amari, 1998).
Adam Kingma & Ba (2014) uses the moving average of the diagonal FIM to represent the curvature information.
K-FAC (Martens & Grosse, 2015) utilizes the layer-diagonal FIM and the approximation of the expected Kronecker product to compute the curvature information.
C E XPERIMENTS C. E FFECT OF THE F IRST AND THE L AST L AYER
Many papers claim that the ﬁrst and the last layer can have a huge impact on the ﬁnal accuracy.
In this section, we investigate this phenomenon as well as the impact of the ﬁrst and the last layer on hardware performances.
We test ResNet-18, MobileNetV2 as well as RegNet-600MF.
Our observations include:
In terms of accuracy, the 4-bit quantization is essentially good, both of these two layers won’t drop too much accuracy (with 0.2%).
But in 2-bit quantization, the last fully connected layer is far more important than the ﬁrst layer.
We also observe that the ﬁrst layer in MobileNetV2 and RegNet (3 × 3 kernels, 32 channels) is slightly more sensitive than that in ResNet-18 (7 × 7 kernel, 64 channels).
In terms of model size, the ﬁrst layer merely has a minor impact because the input images only have 3 channels, while the last layer contains many weight parameters and greatly affects the memory footprint.
We should point out that just the model size in the ﬁrst layer is low doesn’t mean the memory burden is low, because the input image will cost huge memory space.
In terms of latency, the situation depends on the architecture.
For example, in ResNet-18 the ﬁrst layer has a huge impact on the latency, while in MobileNetV2 and RegNet-600MF the last layer is more important than the ﬁrst layer.
This is because the latency is affected by multiple factors, such as the input size of the featuremap, the FLOPs, and the weight memory size.
The arithmetic intensity (OPs/byte) greatly affects the latency.
We ﬁnd that the operations with high arithmetic intensity, i.e., shallow layers in the network, generate a less latency gap between different bitwidths.
[Table]
(cid:51) (cid:51) 71. 2.26 MB 32.80 ms 59.
[Table]
MB
[Table]
40 ms (cid:55) (cid:51) 71.
2.26 MB 32.64 ms 59.
[Table]
MB 30.24 ms (cid:51) (cid:55) 71.
1.65 MB 31.52 ms 56.
0.83 MB 28.48 ms (cid:55) (cid:55) 71.
[Table]
65 MB 31.36 ms 55.
[Table]
MB
[Table]
32 ms
RegNet
[Table]
600MFFP
[Table]
[Table]
(cid:51) (cid:51)
[Table]
[Table]
.19 MB
[Table]
ms 65.
[Table]
MB 23.20 ms (cid:55) (cid:51)
[Table]
[Table]
.19 MB
[Table]
ms
[Table]
1.85 MB 22.88 ms (cid:51) (cid:55) 72.
[Table]
MB 31.
[Table]
ms 62.
[Table]
[Table]
47 MB 22.40 ms (cid:55) (cid:55)
[Table]
[Table]
MB
[Table]
04 ms
[Table] [Table]
MB 22.08 ms
In conclusion, we ﬁnd that keeping the ﬁrst and the last layer 8-bit is unnecessary.
Especially in ResNet-18, we ﬁnd that setting all layers to 4-bit results in 53.3 ms latency and is faster than the 59.8 ms in 2-bit quantization (with ﬁrst and last layer 8-bit), but the accuracy is even 4% higher.
Such phenomenon indicates the potential power of the mixed precision.
C. E FFECT OF D ATA
[Figure]
We evaluated the inﬂuence of the size of calibration dataset and the source of the data on ResNet-18.
We test different numbers of input data point and ﬁnd that the improvement in 4-bit quantization is trivial.
Yet in 2-bit quantization we can see that the accuracy increases 5% when #data points increase.
We also test the distilled data introduced in ZeroQ (Cai et al., 2020).
Distilled data is learned from pretrained models’ BN statistics, i.e., x 1distilled = arg min x ∈R (cid:80) ni =1 (( µ i − ˆ µ i )
+( ς i − ˆ ς i )) where µ i and ς i is the original mean and variance in BN statistics of the ( i ) -th layer.
We ﬁnd that distilled data performs good in 4-bit quantization but still has a large margin with the original ImageNet dataset in 2-bit quantization.
We also ﬁnd the ﬁnal accuracy does not beneﬁt much from the increase of number of distilled data, this might because the distilled data are minimized by a same objective and has low diversity.
C.
[Figure] IXED
[Figure]
[Figure] In this section, we test the mobile CPU latency guaranteed mixed precision.
The latency lookup table is tested using the technique in Gong et al.
We only validate it on ResNet-18 and ResNetbecause the current low-bit General Matrix Multiply (GEMM) implementation only supports normal convolution. The results concur with Fig. Below 4-bit, the mixed precision can achieve better task performance than the uniﬁed precision models.
For ResNet-50, the improvement is lower than that for ResNet-18 and any other mixed precision models.
We think this is because the sensitivity in ResNet-50 is not distinct and therefore the improvement brought by mixed precision is trivial.
C. I MPLEMENTATION C.4. L EARNING S TRATEGIES
In this work, we mainly focus on developing optimization objective rather than optimization strategies.
We observe adaptive rounding performs well in post-training quantization.
A brief introduction on AdaRound is given below, the detailed algorithm can be found in Nagel et al.
Traditional quantization function is performed by rounding-to-nearest operation: ˆ w = s × clip( (cid:98) w /s (cid:101) , n, p ) .
AdaRound optimizes the rounding policy in post-training quantization.
Specifically, all weights are initially rounded by ﬂoor operation, and a learnable variable v determines the ﬁnal rounding result to be ﬂooring or ceiling.
A sigmoid-like function σ ( · ) keeps the learnable variable v moving between 0 and 1 and a regularization term assures the σ ( v ) can converged to either 0 or 1.
The formulation is given by
[Equation]
The minimization problem together with the regularization is given by
[Equation]
(28) where progressively decreasing β in the calibration ensures the σ ( v ) converged to binary values.
The activations cannot be quantized using adaptive rounding because they vary with different input data points.
Thus, we can only adjust its quantization step size Esser et al.
Denoting the quadratic loss in above equation as L q , the gradients of step size is given by
[Equation]
[Equation]
where all step size in the block will be optimized.
C.4. G ENETIC A LGORITHM FOR M IXED P RECISION
t . . . , T th iteration
PE Array
[Figure]
Weight Buffer
[Figure]
A A A Multiply
Replicate
ElementwiseMultiply
Produce
Accu-mulate
Accumulate A11 A12 A
A21 A22 A
A31 A32 A
A11 A11 A11 B11 B12 B
A21 A21 A21 B11 B12 B
A31 A31 A31 B11 B12 B
B11 B12 B
B21 B22 B
B31 B32 B
B11 B12 B
B11 B12 B
B11 B12 B
C11 C12 C
C21 C22 C
C31 C32 C
C11 C12 C
C21 C22 C
C31 C32 C
[Figure]
A
[Figure]
C
Results
[Figure]
We test the latency of quantized neural networks on a self-developed simulator of a precisionvariable accelerator for NN. The basic architecture of this accelerator is inspired by typical systolicmatrix multiplication. The accelerator supports the per-channel quantization parameter.
The precision of each layer of a NN is highly conﬁgurable in this accelerator, supporting 9 types of precision combination: activation: 2-, 4-, 8-bit × weight:
2-, 4-, 8-bit, see Fig. 5a. With the support of scalable function-unit (Sharma et al., 2018), the peak performance of the accelerator is able to achieve corresponding linear improvement as the precision decreases. For example, the peak performance of this accelerator is 256 GMAC/s in 8-bit × 8-bit precision, and it scales to 512 GMAC/s in 8bit × 4-bit precision and 4 TMAC/s in 2-bit × 2-bit precision.
Although this accelerator provides considerable computation resources especially in low precision, the parallelism of the speciﬁc layer (like depthwise convolution) and the bandwidth of on-chip buffer is limited.
Consequently, actual performance may not scale accurately along with the peak performance, and the ﬁnal performance differs according to the size and type of layers.
The simulator performs cycle-accurate simulation and evaluation for a given NN executed on the accelerator, so we can get an equivalent evaluation by using this simulator.
The simulator is available in the provided source codes.
For the acquisition of mobile ARM CPU latency, we adopt the redesigned low-bit GEMM implementation in Han et al.
Fig.
5b shows a brief overview of the low-bit GEMM implementation.
Since there is no instruction supporting the bit-width below 8 on ARM architecture, we can not get a higher computation efﬁciency for extremely low-bit such as 2-bit and 4-bit.
But we can acquire a better memory access efﬁciency.
The primary speedup comes from the reduction of data movement.
Speciﬁcally, we can conduct more times of addition in the same 8-bit register before we have to move it to a 16-bit register to avoid overﬂow.
The lower bit-width is used, the less movement is needed.
Together with the optimized data packing and data padding, we can run mixed precision quantization on Raspberry Pi 3B, which has a 1.2 GHz 64-bit quad-core ARM Cortex-A53.
Note that this implementation is not optimized for depthwise separable or group convolution, therefore we only verify the latency on ResNets.
C.4. I MPLEMENTATION D ETAILS
The ImageNet dataset consists of 1.2M training images and 50K test images.
We follows standard pre-process (He et al., 2016) to get 1024 224 × 224 input images as the calibration dataset.
We fold the batch normalization layer into convolution and freeze the BN statistics before post-training quantization.
We use Adam optimizer (Kingma & Ba, 2014) to learn the weight rounding and activation range to reconstruct the block output.
Note that some layers are not a component of any block, such as the ﬁrst convolutional layer and the last fully connected layer and the last convolutional layer in the MobileNetV2.
These layers use naive layer reconstruction.
The batch size of learning is set toand each block will be optimized for 2 × 10 4 iterations.
The learning rate is set to 10 − 3 during the whole learning process.
Other hyper-parameters such as the temperature β are kept the same with Nagel et al.
For activation step size, we also use Adam optimizer and set the learning rate to 4e-5.
Note that we do not implement the gradient scale as introduced in the original paper (Esser et al., 2020).
After reconstruction, we will store the sensitivity measured on the calibration dataset.
Note that we will store intra-block sensitivity in 2-bit quantization.
The sensitivity, as well as hardware performances for each layer, will be stored in a lookup table.
When calculating the ﬁtness value and determining the hardware performances in a genetic algorithm, we will check the lookup table.
For genetic algorithm, we set the population size to 50 and evolve 100 iterations to obtain the best individual.
The ﬁrst population is initialized by Gaussian distribution and we round the samples to integers in [0, 1, 2], corresponding to bit-width [2, 4, 8].
The mutation probability is set to 0.1.
The genetic algorithm usually completes the evolution in only about 3 seconds.
For object detection tasks, we use 256 training images taken from the MS COCO dataset for calibration.
The image resolution is set to 800 (max size 1333) for ResNet-18 and ResNet-50, while the image resolution for MobileNetV2 is set to 600 (max size 1000).
Note that we only apply block reconstruction in the backbone because other parts of the architecture, such as Feature Pyramid Net, do not have the block structure. Therefore a naive layer reconstruction is applied to the rest of the network.
Learning hyper-parameters are kept the same with ImageNet experiments. We run baseline methods from open-source codes, such as ZeroQ (Cai et al., 2020) 4 , AdaQuant (Hubara et al., 2020) 5 , and LAPQ (Nahshan et al., 2019) 6 . For all baseline experiments, 1024 training (distilled) data are generated and serve as the calibration dataset except we mention it.