GQ-N ET : T RAINING Q UANTIZATION -F RIENDLY D EEP N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Network quantization is a model compression and acceleration technique that has become essential to neural network deployment.
Most quantization methods perform ﬁne-tuning on a pretrained network, but this sometimes results in a large loss in accuracy compared to the original network.
We introduce a new technique to train quantization-friendly networks, which can be directly converted to an accurate quantized network without the need for additional ﬁne-tuning.
Our technique allows quantizing the weights and activations of all network layers down to 4 bits, achieving high efﬁciency and facilitating deployment in practical settings.
Compared to other fully quantized networks operating at 4 bits, we show substantial improvements in accuracy, for example 66.68% top-1 accuracy on ImageNet using
ResNet-18, compared to the previous state-of-the-art accuracy of 61.52% Louizos et al.
(2019) and a full precision reference accuracy of 69.76%.
We performed a thorough set of experiments to test the efﬁcacy of our method and also conducted ablation studies on different aspects of the method and techniques to improve training stability and accuracy.
Our codebase and trained models are available on
GitHub.
I NTRODUCTION
Neural network quantization is a technique to reduce the size of deep networks and to bypass computationally and energetically expensive ﬂoating-point arithmetic operations in favor of efﬁcient integer arithmetic on quantized versions of model weights and activations.
Network quantization has been the focus of intensive research in recent years (Rastegari et al., 2016;
Zhou et al., 2016;
Jacob et al., 2018; Krishnamoorthi, 2018; Jung et al., 2018; Louizos et al., 2019; Nagel et al., 2019;
Gong et al., 2019), with most works belonging to one of two categories.
The ﬁrst line of work quantizes parts of the network while leaving a portion of its operations, e .
g . computations in the ﬁrst and last network layers in ﬂoating point.
While such networks can be highly efﬁcient, using bitwidths down to 5 or 4 bits with minimal loss in network accuracy (Zhang et al., 2018; Jung et al., 2018), they may be difﬁcult to deploy in certain practical settings, due to the complexity of extra ﬂoating point hardware needed to execute the non-quantized portions of the network.
Another line of work aims for ease of real world deployment by quantizing the entire network, including all weights and activations in all convolutional and fully connected layers; we term this scheme strict quantization .
Maintaining accuracy under strict quantization is considerably more challenging.
While nearly lossless 8-bit strictly quantized networks have been proposed (Jacob et al., 2018), to date state-of-the-art 4 bit networks incur large losses in accuracy compared to full precision reference models.
For example, the strict 4-bit ResNet-18 model in Louizos et al.
(2019) has 61 .
52% accuracy, compared to 69 .
76% for the full precision model, while the strict 4-bit MobileNet-v2 model in Krishnamoorthi (2018) has 62 .
00% accuracy, compared to 71 .
88% accuracy in full precision.
To understand the difﬁculty of training accurate low-bitwidth strictly quantized networks, consider a common training procedure which begins with a pre-trained network, quantizes the model, then applies ﬁne-tuning using straight-through estimators (STE) for gradient updates until the model achieves sufﬁcient quantized accuracy.
This process faces two problems.
First, as the pre-trained model was not initially trained with the task of being subsequently quantized in mind, it may not be “quantization-friendly”.
That is, the ﬁne-tuning process may need to make substantial changes to the initial model in order to transform it to an accurate quantized model.
Second, ﬁne-tuning a model, especially at low bitwidths, is difﬁcult due to the lack of accurate gradient information provided
[Figure]
by STE.
In particular, ﬁne-tuning using STE is done by updating a model represented internally with ﬂoating point values using gradients computed at the nearest quantizations of the ﬂoating point values.
Thus for example, if we apply 4 bit quantization to ﬂoating point model parameters in the range [0 , 1] , a random parameter will incur an average round-off error of 1 / 32 , which will be incorporated into the error in the STE gradient for this parameter, leading to possibly ineffective ﬁne-tuning.
To address these problems, we propose GQ-Net , a guided quantization training algorithm.
The main goal of GQ-Net is to produce an accurate and quantization-friendly full precision model, i .
e .
a model whose quantized version, obtained by simply rounding each full precision value to its nearest quantized point, has nearly the same accuracy as itself.
To do this, we design a loss function for the model which includes two components, one to minimize error with respect to the training labels, and another component to minimize the distributional difference between the model’s outputs and the outputs of the model’s quantized version.
This loss function has the effect of guiding the optimization process towards a model which is both accurate, by virtue of minimizing the ﬁrst loss component, and which is also similar enough to its quantized version due to minimization of the second component to ensure that the quantized model is also accurate.
In addition, because the ﬁrst component of the loss function deals only with ﬂoating point values, it provides accurate gradient information during optimization, in contrast to STE-based optimization which uses biased gradients at rounded points, which further improves the accuracy of the quantized model.
Since
GQ-Net directly produces a quantized model which does not require further ﬁne-tuning, the number of epochs required to train GQ-Net is substantially less than the total number of epochs needed to train and ﬁne-tune a model using the traditional quantization approach, leading to signiﬁcantly reduced wall-clock training time.
We note that GQ-Net’s technique is independent of and can be used in conjunction with other techniques for improving quantization accuracy, as we demonstrate in Section 4.3.
Finally, we believe that the guided training technique we propose can also be applied to other neural network structural optimization problems such as network pruning.
We implemented GQ-Net in PyTorch and our codebase and trained models are publicly available.
We validated GQ-Net on the ImageNet classiﬁcation task with the widely used ResNet-18 and compact MobileNet-v1/v2 models, and also performed a thorough set of ablation experiments to study different aspects of our technique.
In terms of quantization accuracy loss compared to reference ﬂoating point models, GQ-Net strictly quantized using 4-bit weights and activations surpasses existing state-of-the-art strict methods by up to 2 .
7 × , and also improves upon these methods even when they use higher bitwidths.
In particular, 4-bit GQ-Net applied to ResNet-18 achieves 66 .
68% top-1 accuracy, compared to 61 .
52% accuracy in Louizos et al.
(2019) and a reference ﬂoating point accuracy of 69 .
76% , while on MobileNet-v2 GQ-Net achieves 66 .
15% top-1 accuracy compared to 62 .
00% accuracy in Krishnamoorthi (2018) and a reference ﬂoating point accuracy of 71 . 88% .
Additionally, GQ-Net achieves these results using layer-wise quantization, as opposed to channel-wise quantization in Krishnamoorthi (2018), which further enhances the efﬁciency and practicality of the technique.
R ELATED WORKS
Neural network quantization has been the subject of extensive investigation in recent years.
Quantization can be applied to different part of neural networks, including weights, activations or gradients.
Courbariaux et al.
(2015), Hou et al.
(2016), Zhou et al.
, Hou & Kwok (2018) and other works quantized model weights to binary, ternary or multi-bit integers to reduce model size.
Wei et al.
(2018) quantized activations of object detection models for knowledge transfer.
Alistarh et al.
(2016), Hou et al.
(2019) quantized model gradients to accelerate distributed training.
Another line of work quantizes both weights and activations to accelerate model inference by utilizing ﬁx-point or integer arithmetic.
These works include Courbariaux et al.
(2016), Rastegari et al.
Gysel et al.
(2016), Krishnamoorthi (2018), Choi et al.
(2018), Zhang et al.
(2018), Jung et al.
A large set of methods have been proposed to improve training or ﬁne-tuning for network quantization.
Straight through estimators (Bengio et al., 2013) (STE) propagate gradients through non-differentiable operations with the identity mapping.
Other training methods “soften” nondifferentiable operations to similar differentiable ones in order for gradients to pass through, then gradually anneal to piecewise continuous functions by applying stronger constraints.
This line of works include Louizos et al.
(2019), Gong et al.
, Bai et al.
Some works regard quantization as a stochastic process that produces parameterized discrete distributions, and guides training using gradients with respect to these parameters Soudry et al.
(2014), Shayer et al.
Another line of works does not require ﬁne tuning, and instead re-calibrates or modiﬁes the original network to recover accuracy using little or even no data He & Cheng (2018), Nagel et al.
(2019), Meller et al.
Several recent works have focused on quantizing all parts of a network, typically in order to support deployment using only integer arithmetic units and avoiding the cost and complexity of additional ﬂoating point units.
Gysel et al.
(2016) proposed performing network inference using dynamic ﬁxed-point arithmetic, where bitwidths for the integer and mantissa parts are determined based on a model’s weight distribution.
Jacob et al.
; Krishnamoorthi (2018) proposed the quantization training and deployment algorithm behind the Tensorﬂow-Lite quantization runtime, which generates strictly quantized networks that can be easily implemented in hardware.
Louizos et al.
(2019) proposed a training method for strictly quantized models based on annealing a smooth quantization function to a piecewise continuous one.
There has also been recent work on using parameterized quantizers which are optimized during quantization training.
Choi et al.
(2018) introduced learnable upper bounds to control the range of quantization.
Zhang et al.
(2018) proposed quantizers with a learnable basis which an be executed using ﬁxed-point arithmetic.
Jung et al.
(2018) proposed to optimize weight scaling and quantization ranges jointly from task losses.
GQ-N ET
In this section we describe the architecture of our proposed GQ-Net and then discuss components of the architecture which can be tuned to improve performance.
3. GQ-N ET A RCHITECTURE
The major components of GQ-Net include the following, and are illustrated in Figure 1:
An L -layer neural network h W ( · ) with all computations performed using full precision ﬂoating point arithmetic.
Here W =
{ W 1 , . .
, W L } denotes the parameter (weights) of the model, with W i , i ∈ 1 . . .
L being the weights in layer i and expressed in ﬂoating point.
The quantized model ˆ h W,Q ( · ) built from h W ( · ) .
Here Q =
{ Q w 1 , . . .
, Q wL , Q a 0 , . . .
, Q aL } is a set of quantizers , i .
e .
mappings from ﬂoating point to (scaled) integer values; the quantizers may be parameterized, and we describe how to optimize these parameters in Section 3.2.
Q wi quantizes weights W i and Q ai quantizes activations in layer i .
Let x 0 denote an input to h W .
To construct output ˆ h W,Q ( x 0 ) of the quantized network, we proceed layer by layer.
We ﬁrst quantize the weights in layers i = 1 , . . .
, L as ˆ w i =
Q wi ( w i ) , and also quantize the input by setting ˆ x 0 =
Q a 0 ( x 0 ) .
we compute the quantized activations ˆ x i in layer i iteratively for i = 1 , . . .
, L using ˆ x i = Q ai (˜ x i ) , where ˜ x i =
g i ( ˆ w i ∗ ˆ x
i − 1 ) , and g i ( · ) denotes the nonlinearity function in layer i and ∗ denotes convolution.
Note that since ˆ w i and ˆ x i − 1 are quantized, ˜ x i can be computed using integer or ﬁxed point arithmetic.
Next, we construct a loss function L incorporating both the training loss L f of the full precision model h W and a loss L q capturing the difference between h W and the quantized model ˆ h W,Q .
[Equation]
Here ω f , ω q ∈ R are parameters capturing the relative importance of training loss versus distributional loss.
In this paper, we focus on image classiﬁcation networks, and thus we set L f to be the cross-entropy loss between outputs from h W and the training labels.
In addition, we set L q = D KL ( σ ( h W ( · ))
||
σ (ˆ h W,Q ( · ))) , where σ denotes the softmax function, to be the KL divergence between distributions σ ( h W ) and σ (ˆ h W,Q ) on each input.
Hence, minimizing the second term in L corresponds to pushing the ﬂoating point and quantized models to behave similarly to each other.
Since the weight parameters W appear in both terms in L , the two terms can give conﬂicting signals for updating W during the optimization of L , causing the optimization to be unstable.
We discuss how to deal with this problem in Section 3.2.
To train GQ-Net, we successively take mini-batches of training samples and labels and use them to compute L during the forward pass and propagate gradients with respect to W and the parameters of Q during the backward pass in order to minimize L .
After L has converged sufﬁciently, we take the quantized weights in ˆ h W,Q ( · ) as the quantized model.
3. O PTIMIZING GQ-N ET
We now describe how different components of GQ-Net can be optimized to improve accuracy and training stability.
Weight scheduling for L f and L q
Parameters ω f and ω q capture the relative importance of the cross entropy and KL divergence errors during training.
A large ω f ignores the similarity between the ﬂoating point and quantized models and may result in a model that is accurate but not quantization-friendly.
Conversely, a large ω q ignores guidance on accuracy from the ﬂoating point model and may result in similar but poorly performing ﬂoating point and quantized models.
We tested different schemes for weighting L f and L q , and found that using ﬁxed values such as ω f = ω q = 0 .
5 already yields better results than many current methods, as discussed in Section 4.
However, further experimentation showed that scheduling , i .
e .
dynamically modifying the values of ω f , ω q during training can produce higher accuracy than using static values.
For example, consider a schedule as shown in Figure 2a, which initially sets ω f = 1 ,
ω q = 0 , then alternates between setting ω q = 1 and ω q = 0 several times.
Schedules of this sort can be understood as initially favoring model accuracy so that the ﬂoating point model is driven to a high accuracy region of model space, before increasing the importance of model similarity so that a quantization-friendly model can be found in the high accuracy region.
This is repeated several times, leading to increasingly more accurate full precision models whose accuracy is then transferred to the quantized model.
As demonstrated in Section 4, this schedule results in better performance than static ones.
LR w_q w_f (a) ϲ ϰ Ϯ Ϭ Ϯ ϰ Ϭ ϭϬϬϬ ϮϬϬϬ ϯϬϬϬ ϰϬϬϬ ϱϬϬϬ ϲϬϬϬ ϳϬϬϬ ůĂǇĞƌϮ͘Ϭ͘ĚŽǁŶƐĂŵƉůĞ͘Ϭ ĨƉ Ƌ (b)
Reducing interference between L f and L q
The loss function L includes terms in L f and L q , where the former is a function of the ﬂoating point parameters W , and the latter involves both W and the quantized version
ˆ W of W parameterized by θ .
We discovered that directly optimizing L results in reduced accuracy, which we attribute to conﬂicting updates to W produced by gradients from the L f and L q terms.
Ideally, we would like update W to minimize L f , while independently updating ˆ W to minimize
L q .
While this is clearly not possible due to the dependency between W and ˆ W , we found that a heuristic based on this idea helped improve accuracy.
In particular, let x L = h W ( x 0 ) and ˜ x L
= ˆ h W,Q ( x 0 ) be the output of the full precision and quantized networks on input x 0 , so that L q =
D KL ( σ ( x L ) ||
σ (˜ x L )) , as in Section 3.1.
During back propagation we compute L f from x L and derive ∇ x L L f as usual, and use this via the chain rule to update W .
However, when computing L q from x L and ˜ x L , we treat x L as a constant tensor which does not produce any gradients, and only derive ∇ ˜
x
L L q and use this via the chain rule to update W .
We can implement this behavior using the detach operator in PyTorch or the stop gradient operator in TensorFlow.
Parameterized quantizer GQ-Net can use any type of quantizer Q ( · ) :
R → T for a discrete set T .
Motivated by recent work such as PACT (Choi et al., 2018), we adopt layer-wise linear quantizers with learnable boundaries in GQ-Net.
In particular, for each layer i , we use one weight quantizer function Q wi,θ wi ( · ) for all weights in the layer, and one activation quantizer function Q ai,θ ai ( · ) for all activations.
Here, θ wi and θ ai represent learnable parameters; for expository simplicity we drop a, w and i in the following and denote all parameters by θ .
θ consists of k , the quantization bitwidth, and lb and ub representing the lower and upper quantization boundaries.
We use uniform quantization, which is substantially easier to implement in hardware, and set
[Equation]
Where clamp( x, a, b ) = max( a, min( x, b )) , and (cid:98)·(cid:101) represents the round operator.
During training, gradients propagate through the nondifferentiable (cid:98)·(cid:101) operator using the straight though estimator (STE), i .
e .
∂ (cid:98) x
(cid:101) ∂x
= 1 .
Parameters lb , ub are updated by the gradients propagated from the loss function L , and thus the quantizers will learn to set the appropriate quantization boundaries to improve accuracy.
Alternatingly optimizing W and θ
The accuracy of the quantized model depends both on the weights W of the ﬂoating point model as well as how these are quantized using the quantizers parameterized by θ .
We found that jointly optimizing W and θ in each iteration resulted in unstable training and poor accuracy.
Performance was substantially improved by alternatingly optimizing
W and θ .
That is, in each training epoch we update either the values in W or θ while freezing the values of the other set of parameters, then switch the updated and frozen sets in the next epoch.
The reason alternating optimization improved training is that both W and θ affect the values of quantized weights, so that updating both simultaneously may cause suboptimal changes to quantized weights.
Multi-domain batch normalization Batch normalization is a critical component in large model training.
However, since GQ-Net trains a ﬂoating point and quantized model simultaneously, we need to adjust the way batch normalization is performed to achieve good accuracy.
In particular, as illustrated in Figure 2, activations from the ﬂoating point and quantized models follow different distributions.
Thus, normalizing them with same set of running statistics can hinder training.
Instead, we regard activations in different numerical precision as being from different domains, similar to as in multi-domain transfer learning, and normalize them with separate statistics { µ f , σ f } and { µ q , σ q } : ¯ x = x − µ f σ f , ¯ˆ x
= ˆ x − µ q σ q .
This modiﬁcation only introduces minor storage overhead, while it signiﬁcantly improves GQ-Net’s accuracy in both full-precision and quantized settings.
E XPERIMENTS
To validate the effectiveness of GQ-Net and assess its different components, we conducted a series of comparisons and ablation studies using the ImageNet classiﬁcation task.
We used the ILSVRC 2012 dataset consists of 1.2M training samples and 10K validation samples from 1K categories, and evaluated our system using top-1 and top-5 validation accuracies.
4. I MPLEMENTATION DETAILS
Network settings We used the ResNet-18, MobileNet-v1 and MobileNet-v2 architectures in the
ImageNet experiments.
All MobileNets used channel expansion ratio 1 . 0 .
Unlike some recent works, we did not modify the order of the Conv, BN and ReLU layers.
We replaced the BatchNorm layers in these models with SyncBatchNorm, i .
e .
we used mini-batch statistics µ f , µ q and σ f , σ q computed from all distributed GPUs during training.
Quantization settings Unless otherwise speciﬁed, all of the following experiments were conducted with parameterized linear quantizers using a bitwidth of 4 .
The weight and activation quantizers each have their own parameters θ = { lb, ub } , which are initialized at the iteration right before the quantization error penalty L q is enabled.
Speciﬁcally, for weight quantizers, { lb, ub } are initialized by the minimum and maximum elements in the weight tensors of each layer.
For activation quantizers, { lb, ub } are initialized by the upper and lower 99 .
9% percentile values, computed from 5 mini-batches sampled from the training set.
Weights and activations of all layers were quantized, including the ﬁrst and last layers.
Training protocol We used the same training protocol for all architectures and quantization settings.
Training was performed on 32 distributed GPUs, each with a mini-batch size of 64, and stopped after 120 epochs on the training set.
Model weights W and quantization parameters θ were optimized with different optimization settings.
Model weights W were randomly initialized using the Kaiming-normal scheme (He et al., 2015) without using a pre-trained model, and optimized by
SGD with 0 .
9 Nesterov momentum and 10 − 4 weight decay.
The learning rate warmed-up from 0 .to 0 .
8 in the ﬁrst 4 epochs, and decayed twice by a factor of 0 .
1 at epochs 60 and 90.
Quantization parameters θ were optimized using Adam without weight decay, with coefﬁcients for ﬁrst and second momentums set to β 1 = 0 .
9 and β 2 = 0 .
999 , and learning rate ﬁxed to 10 − 3 during the entire training process.
Following standard practice, training samples were resized and randomly cropped to 224 × 224 pixels, followed by random horizontal ﬂipping and normalization.
Validation samples were centrally cropped to 224 × 224 pixels, followed by normalization. [Table]
4. C OMPARISON WITH OTHER STRICT QUANTIZATION METHODS
We validated the effectiveness of GQ-Net by comparing it with several other state-of-the-art quantization methods.
As GQ-Net seeks to fully quantize a network and execute it using only integer operations, we perform comparisons with other strict quantization methods.
The comparison baselines include results from (Krishnamoorthi, 2018), (Jacob et al., 2018) and (Louizos et al., 2019), indicated as White Paper, Integer-only and RelaxedQuant resp.
in Table [Table]
The ﬁrst row in the table contains full precision accuracy results evaluated using reference implementations 2 , while the second row contains the full precision accuracy for models trained with our training protocol.
For the ResNet-18 model which is widely studied in the network compression literature, we signiﬁcantly outperform the state-of-the-art strict quantization method RelaxedQuant in an equal bitwidth setting ( + 5 .
16 % top-1 accuracy improvement with 4-bit weights and activations).
Our 4-bit model even outperforms the comparison methods when they use higher bitwidths ( +1 .
58% compared with 5-bit RelaxedQuant, +2 .
04% compared with 5-bit Integer-only).
For the compact MobileNets family, our method also achieves higher accuracy using lower bitwidths.
For example, compared with the White Paper method at 8-bit weights and 4-bit activations, our 4-bit
W/A GQ-Net MobileNet-v2 model achieves + 8 .
15 % top-1 accuracy improvement, and our GQNet MobileNet-v1 model obtains +1 .
04% higher top-1 accuracy in the same setting.
4. A BLATION STUDIES
We also validated the effectiveness of the different components of GQ-Net discussed in Section 3.2, by progressively adding them to the vanilla quantization-friendly training protocol, and applying these protocols to the ResNet-18 architecture with its weights and activations quantized to 4-bits.
For the vanilla setting, W and θ were optimized jointly at each training step, loss weights were set to ω f = ω q = 0 .
5 through the entire training process, gradients from L q propagated through both σ ( x L ) and σ (˜ x L ) , and full precision and quantized activations were normalized using the same set of running statistics.
For a more complete comparison, we also evaluated these settings using the full precision GQ-Net model.
The numerical results are given in Table 2.
By alternatingly updating model weights W and quantizer parameters θ between training steps, quantization accuracy improved by +4 .
24% over the vanilla protocol.
This indicates that although gradients ∇ W L and ∇ θ L both guide their respective parameters to minimize training loss, combining them in a training step makes the quantized weights ˆ W derived from these parameters suboptimal.
Dynamically adjusting the weights ω f and ω q during different parts of the training process as described in Section 3.2 improved quantization accuracy by +0 .
This suggests that the importance of the accuracy loss L f and the distributional loss L q may not be equal at different stages of [Table] the training process.
The schedule we used, alternating between short periods of ω q = 0 and longer periods of ω q = 1 , suggests we should ﬁrst allow the ﬂoating point model settle into a reasonably accurate state before enforcing its similarity to the quantized model.
Blocking the gradient from L q caused by σ ( x L ) further improved quantization accuracy by +0 .
This indicates that although the full precision logits x L and quantized logits ˜ x L are both derived from the same set of parameters W , it is useful to heuristically separate their effects during backpropagation.
Normalizing full precision and quantized activations by different sets of running statistics in the BatchNorm layers improved both full precision accuracy ( +1 . 69% ) and quantized accuracy ( +0 . 60% ).
This highlights the difference in the mean and variance of the full precision and quantized activations, despite the similarity of the full precision and quantized models in terms of KL divergence.
Lastly, we considered the effectiveness of using parameterized quantizers.
We tested replacing the parameterized quantizers in GQ-Net with naive linear quantizers using ﬁxed weight and activation quantization ranges set to the 99.9% percentile values derived from 5 initial training mini-batches.
We found that using ﬁxed quantizers signiﬁcantly lowered accuracy, by 3.59%.
We note that RelaxedQuant also uses learned quantizers, and that replacing these with ﬁxed quantizers may also result in decreased accuracy.
C ONCLUSION
In this paper we presented GQ-Net, a novel method for training accurate quantized neural networks.
GQ-Net uses a loss function balancing full precision accuracy as well as similarity between the full precision and quantized models to guide the optimization process.
By properly tuning the weights of these two factors, we obtained fully quantized networks whose accuracy signiﬁcantly exceeds the state of the art.
We are currently studying additional ways to adjust GQ-Net components to further improve accuracy.
We are also interested in combining GQ-Net with complementary quantization techniques, and in applying similar methodologies to other neural network optimization problems.