F8N ET : F IXED -P OINT 8- BIT O NLY M ULTIPLICATION FOR N ETWORK Q UANTIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural network quantization is a promising compression technique to reduce memory footprint and save energy consumption, potentially leading to real-time inference.
However, there is a performance gap between quantized and fullprecision models.
To reduce it, existing quantization approaches require highprecision INT32 or full-precision multiplication during inference for scaling or dequantization.
This introduces a noticeable cost in terms of memory, speed, and required energy.
To tackle these issues, we present F8Net, a novel quantization framework consisting in only ﬁxed-point 8-bit multiplication.
To derive our method, we ﬁrst discuss the advantages of ﬁxed-point multiplication with different formats of ﬁxed-point numbers and study the statistical behavior of the associated ﬁxedpoint numbers.
Second, based on the statistical and algorithmic analysis, we apply different ﬁxed-point formats for weights and activations of different layers.
We introduce a novel algorithm to automatically determine the right format for each layer during training.
Third, we analyze a previous quantization algorithm— parameterized clipping activation (PACT)—and reformulate it using ﬁxed-point arithmetic.
Finally, we unify the recently proposed method for quantization ﬁnetuning and our ﬁxed-point approach to show the potential of our method.
We verify F8Net on ImageNet for MobileNet V1/V2 and ResNet18/50.
Our approach achieves comparable and better performance, when compared not only to existing quantization techniques with INT32 multiplication or ﬂoating point arithmetic, but also to the full-precision counterparts, achieving state-of-the-art performance.
I NTRODUCTION
Real-time inference on resource-constrained and efﬁciency-demanding platforms has long been desired and extensively studied in the last decades, resulting in signiﬁcant improvement on the tradeoff between efﬁciency and accuracy (Han et al., 2015;
Liu et al., 2017; 2018; Mei et al., 2019; Tanaka et al., 2020; Ma et al., 2020; Krishnamoorthi, 2018; Mishra et al., 2020; Liang et al., 2021; Nagel et al., 2021).
As a model compression and inference acceleration technique, quantization is promising compared to other methods, such as network pruning (Han et al., 2015; Tanaka et al., 2020; Ma et al., 2020) and slimming (Liu et al., 2017; 2018), as achieves a large compression ratio (Krishnamoorthi, 2018; Nagel et al., 2021) and is computationally beneﬁcial for integer-only hardware.
The latter one is especially important because many hardwares (e.g., most brands of DSPs (Ho, 2015; Inc., 2019)) only support integer or ﬁxed-point arithmetic for accelerated implementation and cannot deploy models with ﬂoating-point operations.
However, the drop in performance, such as classiﬁcation accuracy, caused by quantization errors, restricts wide applications of such methods (Zhu et al., 2016).
To address this challenge, many approaches have been proposed, which can be categorized into simulated quantization, integer-only quantization, and ﬁxed-point quantization (Gholami et al., 2021).
Fig.
1 shows a comparison between these implementations.
For simulated quantization, previous works propose to use trainable clipping-levels (Choi et al., 2018), together with scaling techniques on activations (Jin et al., 2020b) and/or gradients (Esser et al., 2019), to facilitate training for the quantized models.
However, some operations in these works, such as batch normalization (BN), are conducted with full-precision to stabilize training (Jin et al., 2020b; Esser et al., 2019), limiting the practical application of integer-only hardware.
Meanwhile, integer-only quantization, where the model inference can be implemented with integer multiplication, addition, and bit shifting, has shown
[Table]
Accumulation
[Table]
Int32)
Output (Int8)
Int32 Multiplication (Int8)
Weight (Int8) Input (Int8)
FP32 Multiplication (FP32)
Output
[Table]
Accumulation (FP32)
FP
Weight (FP32) Input (FP32)
[Table]
[Figure]
Int32 Accumulation (Int32)
Requantization
Output (
[Figure]
Int32 Multiplication (Int8)
Dequantization FP
Weight (Int8) Input (Int8)
FP32 Multiplication (FP32)
Output (FP32)
Accumulation (FP32)
FP
Weight (FP32) Input (FP32)
[Figure]
signiﬁcant progress in recent studies (Jacob et al., 2018; Yao et al., 2021; Kim et al., 2021).
Albeit ﬂoating-point operations are removed to enable models running on devices with limited support of operation types, INT32 multiplication is still required for these methods.
On the other hand, ﬁxed-point quantization, which also applies low-precision logic for arithmetic, does not require
INT32 multiplication or integer division.
For example, to replace multiplication by bit shifting, Jain et al.
(2019) utilize trainable power-of-2 scale factors to quantize the model.
In this work, we adopt ﬁxed-point quantization.
Our work differs from previous efforts (Jain et al., 2019) in three major aspects.
First, to determine the minimum error quantization threshold, we conduct statistical analysis on ﬁxed-point numbers.
Second, we unify parameterized clipping activation (PACT) and ﬁxed-point arithmetic to achieve high performance and high efﬁciency.
Third, we discuss and propose quantization ﬁne-tuning methods for different models.
We dub our method as
F8Net, as it consists in only F ixed-point 8 -bit multiplication employed for Net work quantization.
We thoroughly study the problem with ﬁxed-point numbers, where only INT8 multiplication is involved, without any INT32 multiplication, neither ﬂoating-point nor ﬁxed-point types.
Throughout this paper we focus on 8-bit quantization, the most widely supported case for different devices and is typically sufﬁcient for efﬁciency and performance requirements.
Our contribution can be elaborated as follows.
• We show 8-bit ﬁxed-point number is able to represent a wide range of values with negligible relative error, once the format is properly chosen (see Fig.
3 and Fig. 4).
This critical characteristic enables ﬁxed-point numbers a much stronger representative capability than integer values.
• We propose a method to determine the ﬁxed-point format, also known as fractional length, for weights and activations using their variance.
This is achieved by analyzing the statistical behaviors of ﬁxed-point values of different formats, especially those quantized from random variables with normal distribution of different variances.
The analysis reveals the relationship between relative quantization error and variance, which further helps us build an approximated formula to determine the fractional length from the variance.
• We develop a novel training algorithm for ﬁxed-point models by unifying ﬁxed-point quantization and PACT (Choi et al., 2018).
Besides, we show the impact of fractional length sharing for residual blocks, which is also important to obtain good performance for quantized models.
• We validate our approach for various models, including MobileNet V1/V2 and ResNet18/50 on ImageNet for image classiﬁcation, and demonstrate better performance than existing methods that resort to 32-bit multiplication.
We also integrate the recent proposed ﬁne-tuning method to train quantized models from pre-trained full-precision models
with ours for further veriﬁcation.
R ELATED W ORK
Quantization is one of the most widely-used techniques for neural network compression (Courbariaux et al., 2015; Han et al., 2015; Zhu et al., 2016;
Zhou et al., 2016; 2017; Mishra et al., 2017; Park et al., 2017; Banner et al., 2018), with two types of training strategies: Post-Training Quantization directly quantizes a pre-trained full-precision model (He & Cheng, 2018; Nagel et al., 2019; Fang et al.,
2020a;b; Garg et al., 2021); Quantization-Aware Training uses training data to optimize quantized models for better performance (Gysel et al., 2018; Esser et al., 2019; Hubara et al., 2020; Tailor et al., 2020).
In this work, we focus on the latter one, which is explored in several directions.
One area uses uniform-precision quantization where the model shares the same precision (Zhou et al., 2018;
Wang et al., 2018; Choukroun et al., 2019;
Gong et al., 2019; Langroudi et al., 2019; Jin et al., 2020a; Bhalgat et al., 2020; Chen et al., 2020; Yang et al., 2020; Darvish Rouhani et al., 2020; Oh et al., 2021).
Another direction studies mixed-precision that determines bit-width for each layer through search algorithms, aiming at better accuracy-efﬁciency trade-off (Dong et al., 2019;
Wang et al., 2019; Habi et al., 2020; Fu et al., 2020; 2021; Yang & Jin, 2020;
Zhao et al., 2021a;b; Ma et al., 2021).
There is also binarization network, which only applies 1-bit (Rastegari et al., 2016; Hubara et al., 2016;
Cai et al., 2017; Bulat et al., 2020; Guo et al., 2021).
Despite the fact that quantization helps reduce energy consumption and inference latency, it is usually accompanied by performance degradation.
To alleviate this problem, several methods are proposed.
One type of effort focuses on simulated quantization.
The strategy is to leave some operations, e.g.,
BN, in full-precision for the stabilized training of quantized models (Choi et al., 2018; Esser et al., 2019; Jin et al., 2020b).
Nevertheless, these methods limit the application of the quantized models on resource-demanding hardware, such as DSP, where full-precision arithmetic is not supported for accelerated computing (Inc., 2019; Ho, 2015).
To completely eliminate ﬂoating-point operations from the quantized model, integer-only quantization techniques emulate the full-precision multiplication by 32-bit integer multiplication followed by bit shifting (Jacob et al., 2018;
Zhu et al., 2020; Wu et al., 2020; Yao et al., 2021;
Kim et al., 2021).
However, the calculation of INT32 multiplication in these works requires one more operation, which results in extra energy and higher latency (Gholami et al., 2021).
In parallel, recent work (Jain et al., 2019) proposes to restrict all scaling factors as power-ofvalues for all weights and activations, which belongs to ﬁxed-point quantization methods (Lin et al., 2016; Jain et al., 2019; Kim & Kim, 2021;
Mitschke et al., 2019;
Enderich et al., 2019b; Chen et al., 2017; Enderich et al., 2019a; Zhang et al., 2020; Goyal et al., 2021).
This enables the model to only incorporate INT8 or even INT4 multiplications, followed by INT32 bit shifting.
However, there still a lack of a thorough study of the beneﬁts of using ﬁxed-point arithmetic.
Also, the power-of-2 scaling factors are directly determined from the training data without theoretical analysis and guidance.
In this work, we give an extensive analysis, especially on the potential and theoretical principle of using ﬁxed-point values for quantized models, and demonstrate that with proper analysis and design, a model quantized with only INT8 multiplication involved is able to achieve comparable and even better performance to the integer-only methods implemented with INT32 multiplication.
A NALYSIS OF F IXED -P OINT R EPRESENTATION
In this section, we ﬁrst introduce the ﬁxed-point multiplication (Smith et al., 1997; Tan & Jiang, 2018) and analyze the distribution of weight from different layers in a well-trained full-precision model (Sec. 3.1).
We then investigate the statistical property of ﬁxed-point numbers, and demonstrate the potential of approximating full-precision values by 8-bit ﬁxed-point numbers with different formats (Sec. 3.2).
After that, we study the relationship between standard deviation of random variables and the optimal ﬁxed-point format with the smallest quantization error.
Finally, we derive an approximated formula relating the standard deviation and ﬁxed-point format, which is veriﬁed empirically and employed in our ﬁnal algorithms (Sec. 3.3).
3. A DVANTAGES OF F IXED -P OINT A RITHMETIC
Fixed-point number is characterized by its format, which includes both the word length indicating the whole bit-width of the number and the fractional length (FL) characterizing the range and resolution of the represented values (Smith et al., 1997).
Fixed-point arithmetic—especially ﬁxedpoint multiplication—is widely utilized for applications in, e.g., digital signal processing (Smith et al., 1997; Tan & Jiang, 2018).
Compared with integer or ﬂoating-point multiplication, ﬁxed-point multiplication has two major characteristics:
First, multiplying two ﬁxed-point numbers is more efﬁcient than multiplying two ﬂoating-point numbers, especially on resource-constrained devices such as DSP.
Second, it is more powerful than its integer counterpart due to its versatility and the representative ability of ﬁxed-point numbers (there can be tens of different implementations for ﬁxed-point multiplication but only one for integer and ﬂoating-point ones (Smith et al., 1997)).
This
[Figure] [Figure]
point formats.
Both are experimented on zero-mean Gaussian random variables (R.V.), with ReLU applied on (b).
efﬁciency and versatility make ﬁxed-point quantization a more appealing solution than integer-only quantization.
Speciﬁcally, as shown in Fig.
2a, the scales of weights from different layers in a pre-trained full-precision model can vary in orders, ranging from less than 0 .
1 to nearly 4 .
Direct quantization with only integers inevitably introduces considerable quantization error, unless more precision and more operations are involved, such as using INT32 multiplication together with bit shifting for scaling as shown in Fig. 1c.
On the other hand, employing ﬁxed-point numbers has the potential to reduce quantization error without relying on high-precision multiplication, as weights and activations from different layers have the extra degree of using different formats during quantization.
Indeed, as shown in Fig.
2b for a well-trained MobileNet V2 with 8-bit ﬁxed-point numbers, the fractional lengths for weights and activations vary from layer to layer.
This raises the question of how to determine the formats for each layer.
In the following, we study this for 8-bit ﬁxed-point models.
3. S TATISTICAL A NALYSIS FOR F IXED -P OINT F ORMAT
For a predeﬁned bit-width, integer, which is a special case of ﬁxed-point numbers with zero fractional length, has a predeﬁned set of values that it can take, which severely constrains the potential of integer-only quantization.
On the other hand, ﬁxed-point numbers, with an extra degree of freedom, i.e., the fractional length, are able to represent a much wider range of full-precision values by selecting the proper format, and thus they are more suitable for quantization.
As an example, Fig.
3 shows the relative quantization error with 8-bit ﬁxed-point values using different formats for a set of random variables, which are sampled from normal distributions (both signed and unsigned, with the latter processed by ReLU before quantization) with zero-mean and different standard deviations σ (more experimental details in Appx. 7.2).
From the experiments, we make the following two observations.
Observation 1 : Fixed-point numbers with different formats have different optimal representing regions, and the minimum relative error and optimal standard deviation (annotated as a star) varies
[Figure] [Figure]
[Figure] [Figure] for different fractional lengths (Fig. 3).
This is because the format controls the value magnitude and the representation resolution (the least signiﬁcant bit).
Observation 2: Larger fractional lengths are more robust to represent smaller numbers, while smaller fractional lengths are more suitable for larger ones .
For a given standard deviation, using small fractional length has the risk of underﬂow, while large fractional length might cause overﬂow issue.
Speciﬁcally, integers (black curves in Fig. 4) are much more prone to underﬂow issues and have large relative errors for small enough values to quantize.
3. C HOOSING O PTIMAL F IXED -P OINT F ORMAT
With the above observations, we are interested in answering two questions: ( 1) Can we achieve a small ﬁxed- point quantization
error
for a wide
range
of full-
precision values by always
using the
optimal fractional length
corresponding to the smallest relative error To answer this, we ﬁrst plot the smallest possible relative error amongst all the candidate ﬁxed-point formats against the standard deviation.
As shown in red lines from Fig.
4a and Fig.
4c, for zero-mean normal distribution, by always choosing the optimal ﬁxed-point format, we are able to achieve a relative quantization error smaller than 1% for standard deviation with a range of order of at least around 3 .
For example, for signed quantization, the standard deviation can range from 0 .
1 to around 40 to achieve less than 1% error, and for unsigned quantization, the standard deviation can range from 0 .
1 to 100 .
The experiments verify our presumption that using ﬁxed-point values with the optimal formats is able to achieve negligible quantization error.
(2) Can we have a simple way to determine the optimal fractional length?
To answer this, we plot the optimal fractional length from the statistics of the full-precision values against the standard deviation, as shown in the blue lines in Fig.
4a and Fig.
4c.
We ﬁnd that the threshold σ value corresponding to the jumping point is almost equidistant on the log scale of the standard deviation.
This is expected as the representing region of different formats are differed by a factor of 2’s exponents.
Plotting the threshold standard deviation (on a log-scale) against the corresponding optimal fractional length (Fig. 4b and Fig. 4d), we ﬁnd their relationship is almost linear, leading to the following semi-empirical approximating formulas to determine the optimal fractional length FL ∗ from the standard deviation (more discussion in Appendix 7.7):
[Equation]
In the following, unless speciﬁcally stated, we use (1) to determine the fractional length for both weight and activation quantization.
Note that we only calculate the standard deviation during training.
M ETHODS
In this section, we discuss our proposed training technique for neural network quantization with ﬁxed-point numbers, where the formats of weights and activations in each layer are determined based on (1) during training.
We ﬁrst analyze how to unify PACT and ﬁxed-point quantization (Sec. 4.1).
Then we show how to quantize weights and activations, especially updating for BN running statistics and fractional lengths (Sec. 4.2).
Finally, we discuss the necessity of relating scaling factors from two adjacent layers to calculate the effective weights for quantization, especially for residual blocks where some layers have several layers following them (Sec. 4.3).
4. U NIFYING PACT AND F IXED -P OINT Q UANTIZATION
To quantize a positive value x with unsigned ﬁxed-point number of format (WL , FL) , where WL and FL denotes word length and fractional length for the ﬁxed-point number, respectively, we have the quantization function ﬁx quant as:
[Equation]
where clip is the clipping function, and 0 ≤ FL ≤ WL for unsigned ﬁxed-point numbers.
Note that ﬁxed-point quantization has two limitations: overﬂow, which is caused by clipping into its representing region, and underﬂow, which is introduced by the rounding function.
Both of these introduce approximation errors.
To minimize the error, we determine the optimal fractional length for each layer based on the analysis in Sec. 3.3.
To achieve a better way to quantize a model using ﬁxed-point numbers, we take a look at one of the most successful quantization techniques, PACT (Choi et al., 2018), which clips on the full-precision value with a learned clipping-level α before quantization:
[Equation]
where M is a pre-deﬁned scale factor mapping the value from [0 , 1] to [0 , M ] .
The formal similarity between (2) and (3) inspires us to relate them with each other as (more details in the Appx.
[Equation]
where we have set M = 2 WL − 1 , which is the typical setting.
With this relationship, we can implement PACT and train the clipping-level α implicitly with ﬁxed-point quantization.
4. U PDATING BN AND F RACTIONAL L ENGTH
Double Forward for BN Fusion .
To quantize the whole model with only 8-bit ﬁxed-point multiplication involved, we need to tackle the scaling factor from BN layer, including both the weight and running variance.
Speciﬁcally, we need to quantize the effective weight that fuses the weight of convolution layers with the weight and running variance from BN (Jacob et al., 2018; Yao et al., 2021).
This raises the question of how to determine the running statistics during training.
To solve this problem, we apply forward computation twice.
For the ﬁrst forward , we apply the convolution using quantized input yet full-precision weight of the convolution layer, and use the output to update the running statistics of BN.
In this way, the effective weight to quantize is available.
Note there is no backpropagation for this step.
For the second forward , we quantize the combined effective weight to get the ﬁnal output of the two layers of convolution and BN and do the backpropagation.
Updating Fractional Length .
Different from existing work that directly trains the fractional length (Jain et al., 2019), we deﬁne the fractional length for weight on-the-ﬂy during training by inferring from current value of weight, using (1).
For the fractional length of activation, we use a buffer to store and update the value with a momentum of 0 . 1 , similar to how to update BN running statistics.
Once the fractional lengths are determined after training, we keep them ﬁxed for inference.
4. R ELATING S CALING F ACTORS BETWEEN A DJACENT L AYERS
As shown in (4), there are still two extra factors during the quantization operation, which we denote as a ﬁx scaling factor η ﬁx :
[Equation]
Now α is a trainable parameter with full-precision, which means the ﬁx scaling factor is also in full-precision.
To eliminate undesired extra computation, we absorb it into the above-mentioned effective weights for quantization (Sec. 4.2).
However, the ﬁx scaling factor occurs twice, one for rescaling after quantization ( η ﬁx ) and the other for scaling before quantization ( 1 /η ﬁx ).
To completely absorb it, we need to relate two adjacent layers.
In fact, for a mapping that includes convolution, BN, and ReLU (more details are shown in Appx. 7.5), we apply PACT quantization to relate the activation between two adjacent layers as:
[Equation]

[Equation]
(6) where q is the ﬁxed-point activation, W the full-precision weight of the convolution layer, i and j the spatial indices, n the total number of multiplication, and the superscript ( l ) indicates the l -th block consisting of convolution and BN.
γ , β , σ , µ are the learned weight, bias, running standard deviation, and running mean for the BN layer, respectively.
Also, we set WL = 8 for all layers.
As can be seen from (6), to obtain the ﬁnal effective weight for ﬁxed-point quantization, for the l -th Conv-BN block, we need to access the ﬁx scaling factor, or equivalently, the clipping-level α and the activation fractional length FL , from its following ( l + 1) -th block(s).
To achieve this, we apply two techniques.
Pre-estimating Fractional Length .
As mentioned above, we determine the activation fractional length from its standard deviation.
Also, (5) indicates that the ﬁx scaling factor relies on such fractional length for each layer. However, in (6), we need the ﬁx scaling factor from the next layer to determine the effective weight under quantization, which we have not yet updated.
Thus, when calculating the effective weights during training, we use the activation fractional length stored in the buffer, instead of the one for quantizing the input of the next layer.
Clipping-Level Sharing .
As shown in Fig.
5, for residual blocks, some layers have two following layers (which we also name as child layer).
To avoid the problem of inconsistent ﬁx scaling factor when computing the effective weight for the parent layer, we deﬁne one layer as master and force all its siblings to share its clipping-level.
In fact, the best way is to share both the clipping-level and the fractional length, but we ﬁnd sharing fractional length leads to considerable performance drop, especially for deep models such as MobileNet V2 and ResNet50.
This is because the fractional lengths play two roles here: one is for the ﬁx scaling factor, and the other is for the representing region (or equivalently the clipping-level).
Using different fractional lengths effectively enables different clipping-levels (although only differ by a factor of power-of-2, see Appx. 7.6), which can be beneﬁcial because the activation scales might vary from layer to layer.
Moreover, breaking the constraint of sharing activation fractional length does not introduce much computational cost, as the value only differs in storing format, and typically the values are stored in 32-bit, i.e., the accumulation results are only quantized into 8-bit for multiplication.
Note that when computing the effective weight of the parent layer, we only use the master child’s activation fractional length.
For effective weight of each child layer and ﬁxed-point quantization on its input, we use its own fractional length.
P a r e n t L a y e r
C h il d L a y e r #
C h il d L a y e r
# α = 3.54 FL =α = 3.
FL =
P a r e n t
L a y e r
C h il d L a y e r #
N e x t L a y e r
C h il d L a y e r
P a r e n t L a y e r
C h il d L a y e r #
N e x t
[Figure]
a y
[Figure]
r
[Figure]
d
[Figure]
P a r e n t L a y e r
C h il d L a y e r # C h il d L a y e r #
[Figure] [Figure]
P a r e n t
L a y e r
C h il d L a y e r #
N e x t L a y e r
C h il d L a y e r
[Figure]
r
[Figure]
y
[Figure]
r
[Figure]
y
[Figure]
r #
[Figure]
r
[Figure]
all its sibling layers use the master layer’s clipping level.
On the other hand, since using different fractional length only cause bit shifting or different ﬁxed-point quantization formats, and the values are stored in 32-bit before quantized into 8-bit, we do not share the fractional formats to allow more degrees of freedom.
The two ﬁgures show the case of direct residual connection (a) and that with downsampling convolution layer (b).
E XPERIMENTS
V1/V2b.
Following Yao et al.
, we abbreviate Integer-Only Quantization as “Int”,
INT8-Multiplication-Only Quantization as “8-bit”, the Baseline Accuracy as “BL”
[Table]
and
Top-1 Accuracy as “Top-1”.
All models are for 8-bit weight and activation quantization.
For MobileNet V2, we are using MobileNet
V2b version as it is the most typical one.
[Table]
In this section
[Table]
we present our results for various models on ImageNet (Deng et al., 2009) for classiﬁcation task and compare the results with previous works that focus on quantization-aware training to verify the effectiveness of our method.
We show the results for two sets of training.
First, we discuss the conventional training method following Jin et al.
(2020b).
Second, we unify our method with one recent ﬁne-tuning method that quantizes full-precision models with high accuracy (Yao et al., 2021).
More detailed experimental settings are described in Appx.
Conventional training .
We ﬁrst apply our method using conventional training (Choi et al., 2018; Esser et al., 2019; Jin et al., 2020b; Fu et al., 2021), where the quantized model is trained with the simplest setting as those for full-precision model (more details in
Appx. 7.1).
To verify the effectiveness of our method, we perform experiments on several models including
ResNet18 and MobileNet V1/V2.
As shown in Table 1, our method achieves the state-of-the-art results for all models.
Additionally, we obtain comparable or even better performance than the full-precision counterparts.
Compared with previous works on simulated quantization (Choi et al., 2018; Park et al., 2018; Esser et al., 2019; Jin et al., 2020b; Fu et al., 2021) that requires full-precision rescaling after INT8 convolution, our approach is not only more efﬁcient but also achieves better performance.
On the other hand, compared with previous ﬁxed-point quantization (Jain et al., 2019), our approach gives better results.
This might partially due to that our method is based on a more systematic analysis, as explained above in Section 3.3.
To further understand the signiﬁcance of our method, we plot the fractional lengths for weight and activation for each layer.
Illustrated in Fig.
2b for MobileNet V2, we ﬁnd that the fractional lengths for both weight and activation vary from layer to layer.
Speciﬁcally, for weight quantization, since some layers have relatively large value range of effective weight, especially some depthwise layers, small fractional length is necessary to avoid overﬂow issue.
On the other hand, for layers with small weight scale, large fractional length has more advantages to overcome the underﬂow problem.
The same conclusion also applies for the fractional length for activation.
Indeed, for some early layers in front of depthwise convolution layer, the activation fractional length needs to be small, yet for the later-stages, larger fractional length is desired.
This further veriﬁes our ﬁnding that using different fractional lengths for layers with the same parent is critical for good performance, because layers at different depths might be siblings and requires different fractional lengths (see Fig. 5).
Following Yao et al.
, we abbreviate Integer-Only Quantization as “Int”, INT8-Multiplication-Only
Quantization as “8-bit”, Layer-Wise Quantization as “Layer”, the baseline accuracy as “BL”, Top-1 Accuracy as “Top-1”, and Top-1 Accuracy
Drop with respect to the baseline as “Drop”.
We use two baselines for ResNet50, one from PytorchCV (S´emery, 2021)
(Baseline #1) and another from Nvidia (Nvidia, 2021) (Baseline #2), and we use ResNet50b version.
Note that the OMPQ (Ma et al., 2021) is mixed-precision quantization.
[Table]
Tiny ﬁne-tuning on full-precision model .
Recent work (Yao et al., 2021) focus on investigating the potential of neural network quantization.
To this end, they suggest to tiny ﬁne-tune on a well-pretrained full-precision model with high accuracy.
In this way, it might help to avoid misleading conclusion coming from improper comparison between weak full-precision models with strong quantized model.
To further investigate the power of our method and compare it with these advanced techniques, we also apply our method and ﬁne-tune on several full-precision models with high accuracy.
Also, given the number of total ﬁne-tuing steps is very small, we apply grid search to determine the optimal fractional lengths for this experiment.
The results are listed in Table 2, and we can ﬁnd that our method is able to achieve better performance than previous method (Yao et al., 2021), without time- and energy-consuming high-precision multiplication (namely dyadic scaling shown in Fig. 1c).
Our method reveals that the high-precision rescaling, no matter implemented in fullprecision, or approximated or quantized with INT32 multiplication followed by bit-shifting (a.k.a. dyadic multiplication), is indeed unnecessary and is not the key part for quantized model to have good performance.
This is not well-understood in previous literature.
Specifically, we demonstrate that by properly choosing the formats for weight and activation in each layer, we are able to achieve comparable and even better performance with 8-bit ﬁxed-point numbers, which can be implemented more efﬁciently on speciﬁc hardwares such as DSP that only supports integer operation.
C ONCLUSION
Previous works on neural network quantization typically rely on 32-bit multiplication, either in full-precision or with INT32 multiplication followed by bit-shifting (termed dyadic multiplication), mainly for the scaling part.
This raises the question of whether high-precision multiplication is critical to guarantee high-performance for quantized models, or whether it is possible to eliminate it to save cost.
In this work, we study the opportunities and challenges of quantizing neural networks with 8-bit only ﬁxed-point multiplication, via thorough statistical analysis and novel algorithm design.
We extensively validate our method with the most recent techniques in the literature on several widely used models, including ResNet18/50 and MobileNet V1/V2 on ImageNet classiﬁcation.
With our method, we achieve the state-of-the-art performance without 32-bit multiplication, and the quantized model is able to achieve comparable or even better performance than their full-precision counterparts.
Our method demonstrates that high-precision multiplication, implemented with either ﬂoating-point or dyadic scaling, is not necessary for model quantization to achieve good performance.
One future direction is to perform an in-depth statistical analysis of ﬁxed-point numbers with smaller word-lengths and to employ them for neural network quantization.
A PPENDIX 7. M ORE E XPERIMENTAL D ETAILS
More Details for Conventional Training .
For conventional training method, we train the quantized model initialized with a pre-trained full-precision one.
The training of full-precision and quantized models shares the same hyperparameters, including learning rate and its scheduler, weight decay, number of epochs, optimizer, and batch size.
For ResNet18 and MobileNet V1, we use an initial learning rate of 0 . 05 , and for MobileNet V2, it is 0 .
We ﬁnd the value of learning rate, i.e., 0 .and 0 .
05 , does not have much impact on the ﬁnal performance.
Totally, 150 epochs of training are conducted, with cosine learning rate scheduler without restart.
The warmup strategy is adopted with linear increasing ( batchsize / 256 × 0 . 05 )
(Goyal et al., 2017) during the ﬁrst ﬁve epochs before cosine learning rate scheduler.
The input image is randomly cropped to 224 × 224 and randomly ﬂipped horizontally, and is kept as 8-bit unsigned ﬁxed-point numbers with FL = 8 and without standardization.
For ResNet18 and MobileNet V1/V2, we use batch size of 2048 and run the experiments on 8 A100 GPUs.
The parameters are updated with SGD optimizer and Nesterov momentum with a momentum weight of 0 .
9 without damping.
The original structure of MobileNet
V2 uses ReLU6 as its activation.
Since our uniﬁed PACT and the ﬁxed-point quantization already has clipping operation, and can be equivalently formulated with ReLU6 by rescaling weight or activation, we eliminate ReLU6 in our implementation.
Discussion for Weight Decay .
We set weight decay to 4 × 10 − 5 , and ﬁnd the weight decay scheme is critical for good performance, especially for the quantized model.
We analyze weight decay for different models as follows:
• For ResNet18, we apply weight decay on all layers, including convolution, fully-connected, and BN layers.
• For MobileNet V1, previous methods only apply weight decay on conventional convolution and fully-connected layers, but not on depthwise convolution and BN
(Howard et al., 2017).
We ﬁnd
this leads to the overﬁtting problem
, making some early
convolution layers have large weights, which is not friendly for
quantization.
We further
observe
that some channels of some depthwise
convolution layers have all zero inputs
due
to some channels of previous layer become all
negative
and ReLU is applied afterwards, making the running statistics of the corresponding channels in the
following
BN layer
almost zero
This breaks the regularization effect of BN (Luo et al., 2018)
Since each output channel only depends on one input channel for depthwise convolution layers, the weights connecting them become uncontrolled, and the effective weights become large, leading to an
overﬁtting
problem.
Applying weight decay on the depthwise convolution and BN layers helps to alleviate this problem, and the resulting effective
weights become small
• For MobileNet V2, we ﬁnd overﬁtting plays the role of reducing the validation error (although the training error is lower), and applying weight decay on depthwise convolution or BN weights impairs the training procedure.
The underlying reason might be related to the residual connecting structure of this model (note MobileNet V1 does not use residual connection).
In summary, we apply weight decay on all layers, including depthwise convolution and BN layers
for
ResNet
and
MobileNet V
, and do not apply weight decay on depthwise convolution and BN layers for MobileNet V2.
More Details for Tiny Fine-tuning .
For tiny ﬁne-tuning on full-precision models, we follow the same strategy proposed in Yao et al.
Speciﬁcally, we use a constant learning rate of 10 − 4 , with 500 iterations of ﬁne-tuning (or equivalently data ratio of around 0 .
05 with batch size of 128 ).
Different from (Yao et al., 2021), we ﬁnd ﬁxed BN is not helpful, and we allow it to update during the whole ﬁne-tuning step.
As mentioned in Sec.
5, we apply grid search to determine the fractional lengths for both weight and input, as the training cost is very small and applying grid search does not introduce too much effort or training time.
Also, since the original full-precision model uses the normalized input, we also apply normalization on the images and quantize images with signed ﬁxedpoint numbers (and format determined with grid search) before being fed into the ﬁrst convolution layer of the model.
7. M ORE D ETAILS FOR S TATISTICAL A NALYSIS
For the toy example in Fig.
3, we sample 10 , 000 zero-mean Gaussian random variables with different standard deviations, and apply ReLU activation for the rectiﬁed Gaussian variables with unsigned quantization.
The variables are then quantized with ﬁxed-point quantization given in (2) and (9), respectively.
We calculate the relative quantization error and plot against the standard deviation for each ﬁxed-point format.
Note that zero-mean is a reasonable simplifying assumption if we assume to neglect the impact of bias in BN for analysis purposes.
7. DERIVATION FOR FIXED - POINT AND PACT RELATION
Here we derive the relationship between PACT and ﬁxed-point quantization shown in (4).
Speciﬁcally, the PACT quantization in (3) can be formulated as follows for positive α :
[Equation]
For M = 2 WL − 1 , which is the typical setting for quantization, we have:
[Equation] [Equation]
Comparing with the expression for ﬁxed-point quantization (2), we can immediately get (4).
7. D OUBLE S IDE Q UANTIZATION FOR W EIGHT AND M OBILE N ET V
In (2), we only give the formula for ﬁxed-point quantization of unsigned case.
For weight and activation from some layer without following ReLU nonlinearity (such as some layers in MobileNet
V2), signed quantization is necessary, and the expression is similarly given as:
[Equation]
where clip is the clipping function, and 0 ≤ FL ≤ WL − 1 .
7. D ERIVATION OF E FFECTIVE W EIGHT
Here we derive the equation of effective weights relating two adjacent layers in Sec. 4.3.
Speciﬁcally, for a Conv-BN-ReLU block with conventional PACT quantization using input clipping, quantization and dequantization, the general procedure can be described as
[Equation]
Input
[Equation]
10b
[Equation]
Input
[Equation]
10d
[Equation] [Equation]
where x is the input before clipping, (cid:98) q is the integer input after quantization, (cid:101) q is the full-precision input after dequantization, clip is the clipping function, α is the clipping-level, M = 2 WL − 1 is the scaling for quantization, W ij is weight from convolution layer, and γ , β , σ , µ are weight, bias, running standard deviation, and running mean from BN layer, respectively, and i and j are spatial indices.
We ﬁrst note that (10a), (10b) and (10c) can be combined as:
[Equation]
where q is the ﬁxed-point activation and we have used the relationship given by (4) and the deﬁnition in (5).
From this we can derive that:
[Equation]
which is just (6)
[Equation]
7. P RIVATE F RACTIONAL L ENGTHS E NABLING D IFFERENT C LIPPING -L EVELS
Here we analyze the effect of using private fractional lengths between sibling layers to indicate that this effectively enables private clipping-levels for them.
In fact, the original PACT quantization step is given as
[Equation]
where we have omitted layer and spatial indices for simpliﬁcation.
Now if we use private fractional lengths for sibling layers while require them to share the same clipping level, and use the master child’s fractional length for calculating the effective weight in (6), denoting the fractional length of the master layer as FL m , the above function becomes
[Equation]
where α (cid:48) =
FL m − FL α .
From this we see that using private fractional lengths effectively enables different clipping-levels between sibling layers, and the cost is only some bit shifting. Layer
F r a c t i o n a l L e n g t h
#Weight FL
#Activation FL
[Table]
7. M ORE D ISCUSSION OF THE O PTIMAL F RACTIONAL L ENGTH
Here we give some further discussion of using standard deviation to determine the optimal fractional length.
The main reason is that standard deviation is a more robust statistics than others, such as dynamic range, and is an easily-estimated parameter for Gaussian distributed weights and preactivations.
Considering depth-wise convolution layers that contain much fewer weights and inputs, using robust statistics becomes essential as these layers might include weights or inputs with strange behavior, e.g. , the pre-activation values of some channels become all negative with large magnitude.
Therefore, the standard deviation is more suitable and robust than the dynamic range.
7. F RACTIONAL L ENGTH FOR R ES N ET
Here we provide more results of fractional lengths distribution in Fig.
6 for the well-trained ResNet with 8-bit ﬁxed-point numbers ﬁnetuned from the Baseline #2 in Table 2b.
As we can see, the optimal fractional lengths are layer-dependent and their distribution is highly different from those in
MobileNet V2 (as shown in Fig. 2b).
Speciﬁcally, for MobileNet V2, some layers have vanishing weight fractional lengths and less than 4% of all layers have an activation fractional length less than 4, while for ResNet50, more than 88% of all layers have an activation fractional length that is less or equal to 4.
7. A NALYZING S EARCHING S PACE OF F RACTIONAL L ENGTHS
In the main paper, we adopt the largest possible searching space for the fractional lengths of 8-bit ﬁxed-point.
As shown in Fig.
2b, many layers have a fractional length less than 4 , either for input or weight.
Here we study whether it is possible to use only fractional lengths between 6 and 8 .
To this end, we ﬁnetune on ResNet50b using the Baseline #1.
The results are listed in Table 3, from which we ﬁnd that restricting the fractional lengths between 6 to 8 signiﬁcantly impacts the performance of the ﬁnal quantized model, as the top-1 accuracy drops from 77 .
6% to 72 .