E XPLORING THE P OTENTIAL OF L OW - BIT T RAINING OF C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a low-bit training framework for convolutional neural networks.
Our framework focuses on reducing the energy and time consumption of convolution kernels, by quantizing all the convolutional operands (activation, weight, and error) to low bit-width.
Speciﬁcally, we propose a multi-level scaling (MLS) tensor format, in which the element-wise bit-width can be largely reduced to simplify ﬂoating-point computations to nearly ﬁxed-point.
Then, we describe the dynamic quantization and the low-bit tensor convolution arithmetic to efﬁciently leverage the MLS tensor format.
Experiments show that our framework achieves a superior trade-off between the accuracy and the bit-width than previous methods.
When training ResNet-20 on CIFAR-10, all convolution operands can be quantized to 1-bit mantissa and 2-bit exponent, while retaining the same accuracy as the full-precision training.
When training ResNet-18 on ImageNet, with 4-bit mantissa and 2-bit exponent, our framework can achieve an accuracy loss of less than 1% .
Energy consumption analysis shows that our design can achieve over 6 .
8 × higher energy efﬁciency than training with ﬂoating-point arithmetic.
I NTRODUCTION
Convolutional neural networks (CNNs) have achieved state-of-the-art performance in many computer vision tasks, such as image classiﬁcation (Krizhevsky et al., 2012) and object detection (Redmon et al., 2016;
Liu et al., 2016).
However, deep CNNs are both computation and storage-intensive.
The training process could consume up to hundreds of ExaFLOPs of computations and tens of
GBytes of storage (Simonyan & Zisserman, 2014), thus posing a tremendous challenge for training in resource-constrained environments.
At present, the most common training method is to use GPUs, but it consumes much energy.
The power of a running GPU is about 250W, and it usually takes more than 10 GPU-days to train one CNN model on ImageNet (Deng et al., 2009).
It makes AI applications expensive and not environment-friendly.
[Table]
Reducing the precision of NNs has drawn great attention since it can reduce both the storage and computational complexity.
It is pointed out that the power consumption and circuit area of ﬁxed-point multiplication and addition units are greatly reduced compared with ﬂoating-point ones (Horowitz, 2014).
Many studies (Jacob et al., 2017a; Dong et al., 2019; Banner et al., 2018b) focus on amending the training process to acquire a reduced-precision model with higher inference efﬁciency.
Besides the studies on improving inference efﬁciency, there exist studies that accelerate the training process.
Wang et al.
and Sun et al.
(2019) reduce the ﬂoating-point bit-width to 8 during the training process.
Wu et al.
(2018) implements a full-integer training procedure to reduce the cost but fails to get acceptable performance.
As shown in Tab. 1, Conv in the training process accounts for the majority of the operations.
Therefore, this work aims at simplifying convolution to low-bit operations, while retaining a similar performance with the full-precision baseline.
The contributions of this paper are:
This paper proposes a low-bit training framework to improve the energy efﬁciency of CNN training.
We design a low-bit tensor format with multi-level scaling (MLS format), which can strike a better trade-off between the accuracy and bit-width, while taking the hardware efﬁciency into consideration.
The multi-level scaling technique extracts the common exponent of tensor elements as much as possible to reduce the element-wise bitwidth, thus improving the energy efﬁciency.
To leverage the MLS format efﬁciently, we develop the corresponding dynamic quantization and the MLS tensor convolution arithmetic.
Extensive experiments demonstrate the effectiveness of our low-bit training framework.
One only needs 1-bit mantissa and 2-bit exponent to train ResNet-20 on CIFAR-10 while retaining the same accuracy as the full-precision training.
On ImageNet, using 4-bit mantissa and 2-bit exponent is enough for training ResNet-18, with a precision loss within 1% .
Our method achieves higher energy efﬁciency using fewer bits than previous ﬂoating-point training methods and better accuracy than previous ﬁxed-point training methods.
We estimate the hardware energy that implements the MLS convolution arithmetic.
Using our MLS tensor format, the energy efﬁciency of convolution can be improved by over 6 . 8 × , than the full-precision training, and over 1 . 2 × than previous low-bit training methods.
R ELATED WORK 2. P OST -T RAINING Q UANTIZATION
Earlier quantization methods like (Han et al., 2015) focused on the post-training quantization, and quantized the pre-trained full-precision model using the codebook generated by clustering or other criteria (e.g., SQNR Lin et al.
, entropy Park et al.
Banner et al.
(2018b) selected the quantization bit-width and clipping value for each channel through the analytical investigation.
Jacob et al.
(2017b) developed an integer arithmetic convolution for efﬁcient inference, but it’s hard to be used in training because the scale of the output tensor should be known before calculation.
These quantization methods need pretrained models, and cannot accelerate the training process.
2. Q UANTIZE -A WARE T RAINING
Quantize-aware training considered quantization effects in the training process.
Some methods trained an ultra low-bit network like binary (Rastegari et al., 2016) or ternary (Li et al., 2016) networks, with a layer-wise scaling factor.
Despite that the follow-up studies (Liu et al., 2020; Qin et al., 2019) have been proposing training techniques to improve the performance of binary networks, the extremely low bit-width still causes notable performance degradation.
Other methods sought to retain the accuracy with relatively higher precision, such as 8-bit (Jacob et al., 2017a).
Gysel et al.
(2018) developed a GPU-based training framework to get dynamic ﬁxed-point models.
These methods focus on accelerating the inference process and the training process is still using ﬂoating-point operations.
2. L OW -B IT T RAINING
To accelerate the training process, studies have been focusing on design a better ﬂoating-point data format.
Dillon et al.
(2017) proposed a novel 16-bit ﬂoating-point format that is more suitable for
CNN training, while K¨oster et al.
(2017) proposed the Flexpoint that contains 16-bit mantissa and 5-bit tensor-shared exponent (scale), which is similar to the dynamic ﬁxed-point format proposed by
Gysel et al.
Recently, 8-bit ﬂoating-point (Wang et al., 2018; Sun et al., 2019) was used with chunk-based accumulation and hybrid format to solve swamping.
[Figure]
Some studies used ﬁxed-point in both the forward and backward processes (Zhou et al., 2016).Wu et al.
; Yang et al.
(2020) implemented a full-integer training framework for integer-arithmetic machines.
However, their methods caused notable accuracy degradation.
Banner et al.
(2018a) used 8-bit and 16-bit quantization based on integer arithmetic (Jacob et al., 2017b) to achieve a comparable accuracy with the full-precision baseline.
But it’s not very suitable for training as we discussed earlier.
These methods reduced both the training and inference costs.
In this paper, we seek to strike a better trade-off between accuracy and bit-width.
M ULIT - LEVEL S CALING L OW - BIT T ENSOR F ORMAT
In this paper, we denote the ﬁlters and feature map of the convolution operation as weight (W) and activation (A), respectively.
In the back-propagation, the gradients of feature map and weights are denoted as error (E) and gradient (G), respectively.
3. M APPING F ORMULA OF THE Q UANTIZATION S CHEME
In quantized CNNs, ﬂoating-point values are quantized to use the ﬁxed-point representation.
In a commonly used scheme (Jacob et al., 2017b), the mapping function is float
= scale × ( Fix + Bias ) , in which scale and Bias are shared in one tensor.
However, since data distribution changes over time during training, one cannot simplify the Bias calculation as Jacob et al.
(2017b) did.
Thus, we adopt an unbiased quantization scheme,and extend the scaling factor to three levels for better representation ability.
The mapping formula of our quantization scheme is
[Equation]
where [ · ] denotes the indexing operation, S s is a sign tensor, S t is a tensor-wise scaling factor shared in one tensor, and S g is a group-wise scaling factor shared in one group, which is a structured subset of the tensor.
Our paper considers three grouping dimensions: 1) grouping by the 1-st dimension of tensor, 2) the 2-nd dimension of tensor, or 3) the 1-st and the 2-nd dimensions simultaneously.
S t ,
S g , and ¯ X use the same format, which we refer to as (cid:104) E, M (cid:105) , a customized ﬂoating-point format with E-bit exponent and M-bit mantissa (no sign bit).
A value in the format (cid:104) E, M (cid:105) is
[Equation]
where Man and Exp are the M-bit mantissa and E-bit exponent, and Frac ∈
[1 , 2) is a fraction.
3. D ETAILS ON THE S CALING F ACTORS
The ﬁrst level tensor-wise scaling factor S t is set to be an ordinary ﬂoating-point number ( (cid:104) E t , M t (cid:105) =
(cid:104) 8 , 23 (cid:105) ), which is the same as unquantized data in training to retain the precision as much as possible.
Considering the actual hardware implementation cost, there are some restrictions on the second level group-wise scaling factor S g .
Since calculation results using different tensor groups need to be aggregated, using S g in an ordinary ﬂoating-point format will make more expensive conversions and operations necessary in the hardware implementation.
We proposed two hardware-friendly group-wise scaling scheme, whose formats can be denoted as (cid:104) E g , 0 (cid:105) , and (cid:104) E g , 1 (cid:105) .
The scaling factor in the (cid:104) E g , 0 (cid:105) format is simply a power of two, which can be implemented easily as shifting on the hardware.
From Eq.
2, a S g
= I 2 F ( Man g , Exp g ) value in the (cid:104) E g , 1 (cid:105) format can be written as
[Equation]
which is a sum of two shifting, and can be implemented with small hardware overhead.
The third level scaling factor S x = I 2 F (0 , Exp x ) = 2 − Exp x is the element-wise exponent in ¯
X =
S x
(1 + Man x) , and we can see that the elements of ¯ X in Eq.
1 are in a (cid:104) E x , M x (cid:105) format.
The speciﬁc values of E x and M x determine the type and the cost of the basic multiplication and accumulation (MAC) operation, which will be discussed later in Sec. 5.2.
[Figure]
L OW - BIT T RAINING F RAMEWORK OF CNN
As shown in Fig.
2, the convolution operation (Conv) is followed by batch normalization (BN), nonlinear operations (e.g. ReLU, pooling).
Since Convs account for the majority computational cost, we apply quantization right before Convs in the training process, including three types:
Conv(W,
A), Conv(W, E), and Conv(A, E).
Note that the output data of Convs is in ﬂoating-point format, and other operations operate on ﬂoating-point numbers.
An iteration of this low-bit training process is summarized in Appendix Alg. 2, in which the major differences from the vanilla training process are the dynamic quantizaiton procedure DynamicQuantization and the low-bit tensor convolution arithmetic LowbitConv .
4. D YNAMIC Q UANTIZATION
The dynamic quantization converts ﬂoating-point tensors to MLS tensors by calculating the scaling factors S s , S t , S g and the elements ¯ X , as shown in Alg. 1.
Exponent ( · ) and Fraction ( · ) are to obtain the exponent (an integer) and fraction (a fraction ∈ [0 , 1) ) of a ﬂoating-point number.
While calculating the quantized elements ¯ X , we adopt the stochastic rounding (Gupta et al., 2015), and implement it by using a uniformly distributed random tensor r ∼ U [ − 12 , 12 ] .
StochasticRound
[Equation]
Note that Alg.
1 describes how we simulate the dynamic quantization process using ﬂoating-point arithmetic.
While in the hardware design, the exponent and mantissa are obtained directly, and the clip/quantize operations are done by taking out some bits from a machine number.
Input: X : ﬂoat 4-d tensor; Axis : grouping dimension; R : U [ − 12 , 12 ] distributed random tensor; (cid:104) E g , M g (cid:105) : format of group-wise scaling factors; (cid:104) E x , M x (cid:105) : format of each element
Output: S s : sign tensor; S t : tensor group-wise scaling factors; S g : group-wise scaling factors; ¯
X : quantized elements /* calculating scaling factors */ 1 S s
=
Sign ( X ) 2 S r =
Max ( Abs ( X ) , axis =
Axis ) //
Group-wise maximum magnitude 3 S t =
Max ( S r ) //
Tensor-wise maximum magnitude 4 S gf = S r ÷ S t // Group-wise scaling factors before quantization <
Exp g , F rac g
=
Exponent ( S gf ) , Fraction ( S gf )
Exp g =
Clip ( Exp g , 1 − 2 E g , 0) //
Clip Exp g to E g bits 7 F rac g
= Ceil ( F rac g × 2 M g )
÷ 2 M g //
Quantize Frac g to M g bits 8 S g
= F rac g ×
Exp g // Group-wise scaling factors after quantization /* calculating elements */ 9 X f = Abs ( X )
÷ S g
÷ S t //
Dividing the scaling factors 10 Exp x , F rac x = Exponent ( X f ) , Fraction ( X f ) /
* Quantize F rac x to M x bits with underﬂow handling † */ 11 E xmin = 1 − 2 E x 12 F rac xs =
F rac x × 2 M x if not underﬂow, else F rac x × 2 M x − E xmin + E x 13 F rac xint =
Clip ( StochasticRound ( F rac xs , R )) , 0 , 2 M x − 1) 14 F rac x = F rac xint × 2 − M x if not
4. L OW - BIT T ENSOR C ONVOLUTION A RITHMETIC
In this section, we describe how to do convolution with two low-bit MLS tensors.
Denoting the input channel number as C and the kernel size as K , the original formula of convolution in training is:
[Equation]
We take Conv(W, A) as the example to describe the low-bit convolution arithmetic, and the other two types of convolution can be implemented similarly.
Using the MLS data format and denoting the corresponding values (scaling factors S , exponents Exp , fractions F rac ) of W and A by the superscript ( w ) and ( a ) , one output element Z [ n, co, x, y ] of Conv ( W, A ) can be calculated as:
[Equation]
Eq.
6 shows that the accumulation consist of intra-group MACs that calculates P [ n, co, ci ] and inter-group MACs that calculates Z .
And the intra-group calculation of P [ n, co, ci ] is:
P [ n, co, ci ] =
[Equation]
[Table]
where F rac , Exp are fractions and exponents, whose precision is ( M x + 1) -bits and E x -bits, respectively.
Thus the intra-group calculation contains ( M x + 1) -bit multiplication, (2 E x +1 − 4) -bit shifting, and the (2 M x + 2 E x +1 − 2) -bit integer results are accumulated with enough bit-width to get the partial sum P .
And the accumulator has to be ﬂoating-point in some previous work (Wang et al., 2018; Sun et al., 2019), since they use E x = 5 .
As for the inter-group calculation, each element in S ( p ) is a (cid:104) E, 2 (cid:105) number obtained by multiplying two (cid:104) E, 1 (cid:105) numbers.
Omitting the n index for simplicity, the calculation can be written as:
[Equation]
ci
[Equation]
  
P [ x, ci ]2 − Exp ( p ) [ co,ci ] if Man ( p ) [ co, ci ] = P [ x, ci ]2 − Exp ( p ) [ co,ci ] + P [ x, ci ]2 − Exp ( p ) [ co,ci ] −if Man ( p ) [ co, ci ] = 01 / P [ x, ci ]2 1 − Exp ( p ) [ co,ci ] + P [ x, ci ]2 − Exp ( p ) [ co,ci ] −if Man ( p ) [ co, ci ] =
Due to the special format of S ( p ) , the calculation of the ﬂoating-point Z can be implemented efﬁciently on hardware, in which no ﬂoating-point multiplication is involved.
E XPERIMENTS
We train ResNet (He et al., 2016) on CIFAR-10 (Krizhevsky, 2010) and ImageNet (Deng et al., 2009) with our low-bit training framework.
We experiment with the MLS tensor formats using different (cid:104) E x , M x (cid:105) conﬁgurations.
And we adopt the same quantization bit-width for W, A, E, thus that hardware design is simple.
The training results on CIFAR-10 and ImageNet are shown in
Tab. 2.
We can see that our method can achieve smaller accuracy degradation using lower bit-width.
Previous study (Zhou et al., 2016) found that quantizing E to a low bit-width hurt the performance.
However, our method can quantize E to M x = 1 or 2 on CIFAR-10, with a small accuracy drop from 92.45% to 91.97%.
On ImageNet, the accuracy degradation of our method is rather minor under 8-bit quantization (0.6% accuracy drop from 69.1% to 68.5%).
In the cases with lower bit-width, our method achieves a higher accuracy (66.5%) with only 4-bit than Banner et al.
(2018a) who uses 8-bit (66.4%).
With (cid:104) 2 , 4 (cid:105) data format, the accuracy loss is less than 1%.
In this case, the bit-width
[Table] [Table]
of the intermediate results is 2 M x + 2 E x +1 − 2 =
[Table]
which means that the accumulation can be conducted using 16-bit integers, instead of ﬂoating-points (Mellempudi et al., 2019).
5. A BLATION S TUDIES 5.1. G ROUPING D IMENSION
Group-wise scaling is beneﬁcial because the data ranges vary across different groups.
We compare the average relative quantization error of using the three grouping dimensions (Sec. 3.1) with (cid:104) 8 , 1 (cid:105) group-wise scaling format and (cid:104) 0 , 3 (cid:105) element format.
The ﬁrst row of Fig.
3 shows that the AREs are smaller when each tensor is split to N × C groups.
Furthermore
[Figure]
we compare these grouping dimensions in the training process.
The results in the ﬁrst section of Tab. 3 show that the reduction of AREs is important to the accuracy of low-bit training.
And when tensors are split to N × C groups, the low-bit training accuracy is higher.
And we can see that M g = 1 is important for the performance, especially with low M x (e
[Figure]
g., when M x = 1 )
[Figure]
5.1. E LEMENT - WISE E XPONENT
To demonstrate the effectiveness of the element-wise exponent, we compare the AREs of quantization with different E x without group-wise scaling, and the results are shown in the second row of Fig.
We can see that using more exponent bits results in larger dynamic ranges and smaller AREs.
And with larger E x , the AREs of different layers are closer.
Besides the ARE evaluation, Tab. 3 also shows that larger E x achieves better accuracies, especially when M x is extremely small.
As shown in Fig.
3 Row 3 and Tab. 3, when jointly using the group-wise scaling and the element-wise exponent, the ARE and accuracy are further improved.
And we can see that the group-wise scaling is essential for simplifying the ﬂoating-point accumulator to ﬁxed-point, since one can use a small E x (e.g., 2) by using the group-wise scaling technique.
[Figure]
5. H ARDWARE E NERGY E STIMATION
Fig.
4 shows a typical convolution hardware architecture, which consists of three main components: local multiplication, local accumulation, and addition tree.
Our algorithm mainly improves the local
MAC.
Compared with the full-precision design, we simplify the FP MUL to use a bit-width less than 8 and the local FP Acc to use 16-bit integer.
According to the data reported by (Yang et al., 2020), the energy efﬁciencies are at least 7 × and 20 × higher than full-precision design, respectively.
While our method could signiﬁcantly reduce the cost of the convolution, it also introduces some overhead: 1) Group-wise maximum statistics (Line 2) and scaling factors division (Line 9) in Alg.accounts for the main overhead of dynamic quantization .
The cost of these two operations is comparable with that of a batch normalization operation, which is relatively small compared with convolution since the number of operations is fewer by orders of magnitude (Tab. 1).
The groupwise scaling factors introduce additional scaling.
Fortunately, when using the (cid:104) E g , 0 (cid:105) or (cid:104) E g , 1 (cid:105) format, we can implement it efﬁciently with shifting (see Eq. 3).
To summarize, the introduced overhead is small compared with the reduced cost.
According to the numbers of different operations in the training process (Tab. 1) and the energy consumption of each operation (Appendix Tab. 3) (Horowitz, 2014), we can estimate that our convolution arithmetic is over 6.8 × more energy-efﬁcient than full precision one when training ResNet (see Appendix for the details).
Due to the simpliﬁed integer accumulator, our energy efﬁciency is at least 24% higher than other low-bit ﬂoating-point training algorithms (Mellempudi et al., 2019; Wang et al., 2018).
C ONCLUSION
This paper proposes a low-bit training framework to enable training a CNN with lower bit-width convolution while retaining the accuracy.
Speciﬁcally, we design a multi-level scaling (MLS) tensor format, and develop the corresponding quantization procedure and low-bit convolution arithmetic.
Experimental results and the energy analysis demonstrate the effectiveness of our method.
A L OW - BIT T RAINING F RAMEWORK
Input: L layers
W
L t learning
Output: W 1: L t + updated ﬂoat weights /* forward propagation */ 1 for l in 1 : L do q W l = DynamicQuantization ( W l ) q
[Equation]
l
[Equation]
=
DynamicQuantization ( A l − 1 )
[Equation]
LowbitConv (
[Equation]
l
[Equation]
q
[Equation]
l
[Equation]
Y
[Equation]
BatchNorm (
[Equation]
l )
[Equation]
Activation (
[Equation]
) 7 ∂loss∂
[Equation]
Criterion ( A
[Equation]
, T ) /* backward propagation */ for l in L : 1 do
[Equation]
[Equation] [Equation]
[Equation] [Equation]
if is not then
[Equation]
LowbitConv
[Equation] [Equation]
16 Return W 1: L t +
B E XPERIMENTAL S ETUP
In all the experiments, the ﬁrst and the last layer are left unquantized following previous studies (Zhou et al., 2016; Mellempudi et al., 2019; Sun et al., 2019).
For both CIFAR-10 and ImageNet, SGD with momentum 0.9 and weight decay 5e-4 is used, and the initial learning rate is set to 0.1.
We train the models for 90 epochs on ImageNet, and decay the learning rate by 10 every 30 epochs.
On
CIFAR-10, we train the models for 160 epochs and decay the learning rate by 10 at epoch 80 and 120.
[Figure]
x
[Figure]
l u e
[Figure]
a x v a l u e
[Figure]
e
[Figure]
e
[Figure]
a x v a l u e m a x v a l u e
[Figure]
C G ROUP - WISE S CALING
Group-wise scaling is beneﬁcial because the data ranges vary across different groups, as shown in
Fig.
The blue line shows the max value in each group when A and E are grouped by channel or sample.
If we use the overall maximum value (green lines in Fig. 5) as the tensor-wise scaling, many small elements will be swamped.
And usually, there are over half of the groups, in which all elements are smaller than half of the overall maximum (red line).
Thus, to fully exploit the bit-width, it is natural to use group-wise scaling factors.
[Figure]
D E LEMENT - WISE E XPONENT
Fig.
6 shows the performances of training ResNet20 on CIFAR-10 with different (cid:104) E x , M x (cid:105) conﬁgurations.
We can see that, when the mantissa bit-width M x is extremely low (e.g., 1), the element-wise exponent bit-width E x is essential for achieving an acceptable performance.
E E NERGY E FFICIENCY E STIMATION
Tab. 4 (Horowitz, 2014) reported that the energy consumption of a FP32 multiplication (FP32 MUL) is about 4 times that of a FP32 addition (FP32 ADD).
Denoting the energy consumption of FP
ADD as C and FP32 MUL as 4 C , according to (Yang et al., 2020), we can estimate the cost of FP
MUL and INT16 ADD as 4 / 7 C and 1 / 20 C , respectively.
Then, using the operation statistics in
Tab. 1, we can calculate the energy consumption of one training iteration, and estimate the energy efﬁciency improvement ratio:
[Equation]
[Table]
In order to evaluate energy efﬁciency advantage more accurately, we have implemented the RTL design of the two MAC units in Fig.4, and used Design Compiler to simulate the area and power, the power results are shown in Tab.5.
We can see that the simulated results of RTL implemented is similar with our estimation above, both showing the energy efﬁciency of our framework.
[Table]