Is Integer Arithmetic Enough for Deep Learning Training?
Abstract
The ever-increasing computational complexity of deep learning models makes their training and deployment difﬁcult on various cloud and edge platforms.
Replacing ﬂoating-point arithmetic with low-bit integer arithmetic is a promising approach to save energy, memory footprint, and latency of deep learning models.
As such, quantization has attracted the attention of researchers in recent years.
However, using integer numbers to form a fully functional integer training pipeline including forward pass, back-propagation, and stochastic gradient descent is not studied in detail.
Our empirical and mathematical results reveal that integer arithmetic seems to be enough to train deep learning models.
Unlike recent proposals, instead of quantization, we directly switch the number representation of computations.
Our novel training method forms a fully integer training pipeline that does not change the trajectory of the loss and accuracy compared to ﬂoating-point, nor does it need any special hyper-parameter tuning, distribution adjustment, or gradient clipping.
Our experimental results show that our proposed method is effective in a wide variety of tasks such as classiﬁcation (including vision transformers), object detection, and semantic segmentation.
Introduction
Recently, deep learning models have evolved to deliver acceptable performance in many areas such as computer vision, natural language processing, and speech recognition.
However, the number of parameters and computational complexity of most deep learning applications have increased signiﬁcantly.
This ever-increasing computational complexity makes the deployment of deep neural networks harder on edge devices.
Furthermore, the energy required to train a deep learning model increases proportionately with computational complexity.
Thus, the training cost of these large and complex models also increases signiﬁcantly on the high-end cloud servers.
Although, quantization techniques are commonly used to accelerate inference of deep learning models, here we target a fully integer training pipeline (i.e. forward propagation, back-propagation and SGD) .
Unlike recent proposals in quantized back-propagation, we directly change the number representation of ﬂoatingpoint values.
Furthermore, we show empirically and theoretically that our proposed integer training methodology is effective in training deep learning models with integer-only arithmetic .
Despite the fact that accelerated training using integer back-propagation seems intriguing, there are some challenges associated with integer gradient computation.
For example, (i) gradients must be scaled correctly in order to be adapted to the limited dynamic range of the integer number (e.g. int8 : [  128 , 127] ).
(ii)
The numerical error of the gradient must be unbiased in order to preserve the convergence trajectory of the training algorithm.
A small numerical error can be accumulated through the course of training and change the convergence behaviour.
(iii)
The integer training must be distribution independent , i.e. the training method should not depend on the distribution of the gradients, weights, or training data.
We propose a novel integer training method designed to address aforementioned challenges (i), (ii) and (iii) simultaneously.
Having a linear dynamic ﬁxed-point mapping coupled with a non-linear inverse mapping is the key to full integer training.
To this end, we make the following contributions:
• A hardware-friendly integer training method is proposed based on extracting the maximum ﬂoating-point exponent of the tensors as the scale .
We propose linear ﬁxed-point mapping of tensors, while the corresponding inverse mapping for ﬂoating-point is non-linear.
Our proposed method addresses all the previously mentioned challenges: (i) it computes the scales dynamically, and moreover the scale does not need to be adjusted in the course of training; (ii) it provides an unbiased estimation of the gradients and consequently, its convergence trajectory closely follows the ﬂoating-point version; (iii) our proposed representation mapping does not depend on the distribution of the training data, weights, or gradients.
• We study the optimality gap of our proposed integer training algorithm and show it is analogous to its ﬂoating-point counterpart in the course of training ( Theorem 1 ).
Our analysis of stochastic gradient descent with our proposed ﬁxed-point gradients shows the original ﬂoating-point optimality gap is only shifted by a negligible amount ( Remark 3 ).
• Our proposed method effectively performs all operations required in modern neural networks using integer-only arithmetic .
For instance, the computation of linear layer, convolutional layer, batch-norm and layer-norm, residual connections and also stochastic gradient descent (including gradients, momentum, weight decay, and weight update) are all performed in integer arithmetic.
To the best of our knowledge, this is the ﬁrst time that back-propagation of a batch-norm, and the computation of stochastic gradient descent (SGD) is performed in integer arithmetic with negligible loss in the accuracy for large datasets such as ImageNet.
The rest of this paper is structured as follows.
Section 2 reviews some previous works in the ﬁeld of quantized back-propagation and quantiﬁes the similarities and differences with our representation mapping method.
Section 3 discusses our integer training methodology in detail.
Theoretical aspects of our integer training method on SGD are studied in Section 4.
Experimental results supporting our integer training methodology and convergence theory are presented in Section 5.
Related works
Banner et al.
[1], proposed a bifurcated back-propagation method where the gradient of weights are in ﬂoating-point format and input gradients are in 8-bit ﬁxed-point format.
Moreover, they proposed a range-based batch-norm which is more tolerant to quantization noise.
The main difference of our proposed algorithm with Banner et al.
[1] is that our integer training method does not demand bifurcation.
In other words, our gradients with respect to weights and inputs are represented and computed using int8 format.
Furthermore, in our method, the back-propagation of batch-norm, as well as stochastic gradient descent (SGD) algorithm, are also represented and computed using integer values.
Zhang et al.
[2] have proposed a layer-wise “precision-adaptive” quantization method that does not affect the distribution of the data.
In their method, the quantization error is measured and the quantization scale is adjusted over the course of training accordingly.
Similarly, in Zhao et al.
[3] a distribution adaptive quantization method is proposed that takes into account the distribution of gradients in the channel dimension.
In addition, they introduced a gradient clipping strategy in order to normalize the magnitude of the gradients in the back-propagation.
The scale of the quantization is adapted to the distribution of the gradients and adjusted iteratively over the course of training.
Unlike the distribution adaptive methods such as [2, 3], our proposed integer training method does not depend on the distribution of training data, weights, and gradients.
Thus, there is no need to adjust the scale iteratively in the course of training.
In Zhu et al.
[4], a “direction sensitive gradient clipping method” is proposed to avoid inappropriate updates in the back-propagation.
Moreover, they proposed a learning rate scaling that tackles the problem of unbiased gradient error accumulation.
In a similar work, Sakr and Shanbhag [5] proposed a precision assignment methodology to improve the convergence of the quantized back-propagation.
This precision assignment is based on some criterion on internal accumulator noise, quantization noise of backward and forward propagation, and gradient clipping.
Unlike [4, 5], our proposed method enjoys having the advantage of unbiased gradients , hence the convergence trajectory closely follows the ﬂoating-point counterpart.
Thus, our method does not need any speciﬁc gradient correction, gradient clipping strategy, and learning rate corrections as opposed to [4, 5].
Jin et al.
[6] proposed a uniﬁed ﬁxed-point and parameterized clipping activation to achieve high accuracy.
Furthermore, they proposed a method that directly trains the fractional length (i.e. scale) of the ﬁxed-point quantization.
Additionally, they use a double forward batch-norm fusion to determine the scaling factor of the batch-norm layer in the forward propagation, while the back-propagation of batch-norm remains ﬂoating-point.
In contrast, our proposed integer training method is capable of implementing integer variant of the batch-norm layer in forward propagation without double fusion as opposed to [6].
We also perform batch-norm’s back-propagation in integer arithmetic which was ignored in the previous works.
Note that back-propagation of batch-norm and layer-norm is sensitive to quantization, as such, naive quantization leads to training divergence.
Additionally, we perform stochastic gradient descent (SGD) computation including weight update, weight decay, and momentum in integer arithmetic with no signiﬁcant loss of accuracy.
Wang et al.
[7] explored an integer-arithmetic implementation of some preliminary CNN architectures such as
AlexNet, however they experienced considerable accuracy drop for ImageNet dataset.
Moreover, more complicated architectures such as ResNet and transformers are not considered.
In our proposed method, we used dynamic ﬁxed-point number format [8] as the primary number format for integer training.
This speciﬁc ﬁxed-point number format (also known as block ﬂoating-point format) has been used previously to quantize inference of deep learning models in Courbariaux et al.
[8], Drumond et al.
[9], and Rouhani et al.
Methodology
Common quantization methods, which use division and clipping techniques (refer to Appendix A.6), are inefﬁcient for back-propagation.
Therefore, we decided to go one step deeper and change the number format directly.
Here, we propose a hardware-friendly number representation mapping using the dynamic ﬁxed-point number format where the scale is deﬁned per tensor .
In this approach, each tensor can be represented by its int8 version while it is multiplied by a shared scale, as opposed to other methods that allow multiple shared scales for different partitions of the tensor (see Rouhani et al.
[10, Figure 4]).
One shared scale per tensor makes the computations easier in the computing hardware (e.g. CPU or GPU).
However, the training algorithm diverges if the representation mapping is not executed properly.
To tackle this problem, we suggest two subtle changes; (i) we propose to perform ﬁxed-point mapping of a tensors in a linear fashion while the inverse mapping is non-linear as explained in Sections 3.1 and 3.2.
This is the key to success of our algorithm since a linear ﬁxed-point mapping allows monotonic conversion of ﬂoating-point format to ﬁxed-point, while a non-linear inverse mapping allows preserving information.
(ii) We suggest to use stochastic rounding in the back-propagation in a way that preserves the expected value of the tensors as well as their vital statistics, such as the mean.
Stochastic rounding in conjunction with representation mapping is crucial to keep the integer training loss trajectory close to its ﬂoating-point counterpart.
3.1 Linear ﬁxed-point mapping
Our proposed integer training method is based on manipulating the ﬂoating-point number format.
This is a very simple and effective way of converting ﬂoating-point values to ﬁxed-point values as opposed to other commonly used methods that involve division operation.
Our method essentially uses shift and round operations to convert ﬂoating-point to ﬁxed-point format, see Figure 1(a).
In this method, a tensor comprising n ﬂoating-point numbers ( f 1 , f 2 , ..., f n ) is converted to a ﬁxedpoint tensor.
First, as shown in Figure 1(a), sign, exponent, and mantissa of each ﬂoating-point number are extracted using the unpack to integer function.
For instance, f 1 is unpacked to s 1 , e 1 , mwhere ( s, e, m ) is used to denote ( sign , exponent , mantissa ).
We simply ﬁnd the maximum element of e 1 , e 2 , . . .
e n i.e. e max = max( e 1 , e 2 , . . .
e n ) to extract the shared scale of the tensor.
Subsequently, the original mantissas are shifted to the right according to the difference of their exponent and e max , to get the individual mantissas.
For instance, m 1 is shifted to right by e max  e 1 which is denoted by m 1 >> ( e max  e 1 ) .
By shifting mantissas to right, we intentionally push the small values to the sub-normal region, where the most signiﬁcant bit of mantissa is not 1 in binary format i.e. (1) 2 , see
Zuras et al.
Pushing the mantissas to the sub-normal region is performed to align their exponents f f fn ...
Unpack To Integer
Float Tensor ...
e max m 1 >>e max - e 1 m 2 >>e max - em n >> e max - e n
Shifted 24-bitMantissas S n S 2 S
S n S 2 S
Stochastic Rounding ...
Signed 8-bit Mantissas
Send to Integer Layer
Singed 8-bit Mantissas 8-bitScale 24-bitMantissa 8-bitScale
S 1 e 1 m
S 2 e 2 m
S n e n m n
Signedmantissas Sharedexponent
Rounding /
Extracting sign
Sign
Exponent Mantissa
Floating point alignment (Using LZA unit)
Pack to float
Float Tensor
L aye r
C o m pu t a t i on e .
g .
L i n ea r L aye r
N on - li n ea r
I n ve r se M a
pp i ng
Add
Integer Linear Layer (GEMM)
Reconstructing individual exponents
[Figure]
with e max and this uniﬁes the scale for ﬁxed-point values in a tensor.
In the next step, to create int mantissas, the 24-bit single ﬂoating-point mantissas (i.e. 23-bit mantissa + 1 implicit hidden bit) are further rounded to 7-bit mantissas to construct signed int8 values i.e 7-bit unsigned integer and 1 sign bit.
As an example, let us assume the shifted mantissa is (0 . 01011001010101010100000) 2 , then it is going to be randomly rounded to either (0 . 010110)
2 or (0 . 010111) 2 based on a probability (Refer to A.1 for details).
3.2 Non-linear inverse mapping
Our proposed ﬁxed-point mapping method is performed by pushing mantissa values to the sub-normal region and rounding them stochastically.
Hence, the inverse mapping must be performed using a ﬂoating-point alignment module that normalizes the mantissas.
A normalized ﬂoating-point mantissa is required to start with the most signiﬁcant bit (1) 2 .
For instance, an alignment module converts 2 127 ⇥ (0 . 0101) 2 to 2 127  2 ⇥ (1 . 0100) 2 .
The alignment module is a very well-known logic circuit that is available in the commodity hardware using Leading Zero Anticipator (LZA) block [12].
Coupling a linear ﬁxed-point mapping with a non-linear inverse mapping in this way keeps the number format close to ﬂoating-point.
Moreover, combining them with stochastic rounding, results in having unbiased ﬁxed-point variant that forces the trajectory of the integer training closely follow the ﬂoating-point counterpart.
Figure 1(b) shows the inverse mapping unit.
This unit receives a tensor of mantissas and a single value of shared exponent computed by an integer layer.
Then a tensor of exponents is reconstructed by repeating the value of the shared exponent.
The size of this tensor is equal to the mantissa tensor’s size.
Moreover, the mantissa tensor is rounded and its sign is extracted.
Note that the rounding is used here because the output of the previous layer might have some excessive mantissa bits; this normally happens in matrix multiplication and convolution operations.
Then the sign, exponent, and mantissa tensors are sent to an alignment unit that shifts the mantissa and adjust the exponent in order to normalize the ﬂoating-point number format [11].
Finally, the result is packed and carried out to the next layer.
3.3 Integer layer computations
When the ﬁxed-point mapping of ﬂoating-point values is completed, the ﬁxed-point values are used to perform the integer-only layer computations.
For instance, let us consider a linear layer which consists of a General Matrix Multiplication (GEMM) operation, but, the idea can be generalized to other types of layers.
Figure 2 demonstrates the layer-wise integer computations of a linear layer where the shared exponents and integer mantissas are treated separately.
As shown in the Figure 2, in order to perform an integer-only GEMM operation, we need to multiply scales (2 e max1 ⇥ 2 e max2 ) .
This multiplication performs an integer addition operation on the exponents ( e max1 + e max2 ) .
Furthermore, the integer mantissas are sent to an integer GEMM module to compute the output mantissa tensor.
[Figure]
L i n ea r F i xe d - po i n t
M a
pp i ng
L aye r
C o m pu t a t i on e .
g .
L i n ea r L aye r
[Figure]
Also note that in our implementation, when the mantissa tensor is in int8 format, multiplication is in int16 format and accumulation is in int32 format.
3. Understanding the representation mapping
Here we provide an intuition of how our proposed representation mapping approach works.
Let us denote A as a tensor and ˆ A as its ﬁxed-point version in the way that is introduced earlier.
In addition, random variables
A i and ˆ A
i are i th element of those tensors.
Also note that A i and ˆ A i are different representations of the same real number, but one is in ﬂoating-point format and the other is in dynamic ﬁxed-point format.
Thus, one can relate A i and ˆ A i with a random error term  i as ˆ
A i =
A i +  i .
Since in our training method we used stochastic rounding [13], E { ˆ A i } =
A i , or equivalently E {  i } =
(see Appendix A.1).
In other words, the ﬁxed-point value is on the average equal to the ﬂoating-point value.
Note that here we consider single precision ﬂoating-point values as a surrogate of real values.
Linear and convolutional layers: For these two types of layers, both forward and backward propagation computations are based on inner products.
Thus, it is easy to see that our proposed ﬁxed-point inner product ˆ C ij =
P k ˆ
A ik ˆ B kj is an unbiased estimator of the ﬂoating-point inner product
[Equation]
Residual connections: A residual connection involves element-wise addition of two tensors.
Each ﬁxed-point element ˆ C ij =
ˆ A ij + ˆ B ij is an unbiased estimator of its ﬂoating-point version
[Equation]
Batch-norm: A batch-norm layer is deﬁned as
[Equation]
For this type of layer, it is important to compute signal statistics such as mean ˆ µ and variance ˆ correctly in integer arithmetic.
Thus, the ﬁxed-point mean ˆ µ is also an unbiased estimator of the true mean µ
[Equation]
Then the following derivation holds for ﬁxed-point variance ˆ
[Equation]
where  2 is the true ﬂoating-point variance of the batch and  2  is the variance of the noise introduced by the linear mapping to ﬁxed-point.
Note that the error variance  2  is rather small and can be integrated to ✏ in the denominator of equation (3).
Theoretical analysis of SGD using the proposed method
The generic equation of weight update in the k th iteration of SGD is
[Equation]
6) where g ( w k , ⇠ k ) is the estimated gradients of random samples of the batch generated by the seed ⇠ k , and ↵ k is the learning rate.
We make the following common assumption in the sequel.
Assumption 1 (Lipschitz-continuity).
The loss function L ( w ) is continuously differentiable and its gradients r L ( w ) satisﬁes the following inequality where L > 0 is the Lipchitz constant
[Equation]
Assumption 2.
(i) L ( w k ) is bounded.
(ii)
Estimated gradients g ( w k , ⇠ k ) is an unbiased estimator of the true gradients of the loss function
[Equation]
and (iii,a) there exist scalars M > 0 and M V [-⩾ 
-]> 0 such that for all iterations of SGD V ⇠
[Equation]
Note that here we deﬁne
[Equation]
Also from Assumption 2.
(ii) and (iii,a) , the second moment bound can be derived
[Equation]
Effect of gradient variance on convergence: The quality of the estimated gradients g ( w k , ⇠ k ) directly affects the convergence of the SGD.
The effect of ﬁrst and second moments of gradient are already studied on real numbers in the literature.
Lemma 1.
Suppose Assumption 2 is true, then we have
[Equation]
Proof: See Bottou et al.
[14, Lemma 4.4].
Inequality (9) shows the effect of gradient variance bounds, M and M G , on each iterate of SGD, and shows the greater the variance, the more deterioration in the quality of SGD steps.
The ﬁrst term,  (1  1 2 ↵ k LM G ) ↵ k || r L ( w k ) || 22 contributes to the decrease of the loss function while the second term, 12 ↵ 2 k LM , prevents it.
Upon choosing the correct gradient estimates, the right hand side of inequality (9) is bounded by a deterministic quantity, and asymptotically ensures sufﬁcient descent of the loss L ( w ) .
Note that the expectation E ⇠ k is taken over random samples with seed ⇠ k .
4.1 Fixed-point gradient noise
When performing representation mapping in the back-propagation, the quality of the gradients deteriorate.
Thus, there is a need to consider the effect of ﬁxed-point mapping variance.
Then,
Assumption 2 (iii,a) should be modiﬁed accordingly.
Remark 1.
Note that Assumption 2 (i), (ii) still hold after ﬁxed-point mapping because of stochastic rounding, i.e. the ﬁxed-point gradient remains an unbiased estimator of gradient (refer to Appendix
A.1).
Assumption 2 (iii,b).
When the gradients are in ﬁxed-point format i.e. ˆ g ( w k , ⇠ k ) , there exist scalars
M > 0 , M V > 0 , M q > 0 and M qV > 0 such that for all iterations of SGD
[Equation]
If ˜ M := M + M q and ˜ M V :=
M V + M qV , then Assumption 2 (iii,b) takes the exact form of
Assumption 2 (iii,a) i.e. V ⇠ k { g ( w k , ⇠ k ) } 6 ˜ M + ˜ M V || r L ( w k ) ||
However, here we separated M q and M qV to emphasize the effect of ﬁxed-point mapping on the true gradients.
Remark 2.
If Assumption 2 (iii,b) holds true, inequality (9) can be transformed to its ﬁxed-point version
[Equation]
Inequality (10) shows the effect of added representation mapping variance with bounds, M q and M qG , on each iterate of SGD.
This observation shows that ﬁxed-point mapping degrades the convergence of SGD unless its variance bounds are relatively small, or controlled by the learning rate.
As an example, refer to Appendix A.2 for analytical derivations of M q and M qG for the back-propagation of a linear layer involving a ﬁxed-point inner product.
4.2 Strongly convex and locally convex loss
Assumption 3 (Strong convexity).
The loss function L ( w ) is differentiable and strongly convex.
We recall that strong convexity for differentiable functions is equivalent to the following inequality with some constant c >
[Equation]
A strongly convex function has a unique minimum point at w ⇤ with the loss value L ⇤ =
L ( w ⇤ ) .
Theorem 1.
Suppose Assumptions 1, 2(i), 2(ii), 2 (iii,b), 3 are all true,
then a SGD method running with ﬁxed-point gradients i.e. ˆ g ( w k , ⇠ k ) and a ﬁxed learning rate 0 < ¯ ↵1 L ( M G + M qG ) satisﬁes the following bound for its optimality gap with the minimum loss L ⇤ at the k th iteration
[Equation]
Proof.
See Appendix A.3.
Remark 3.
Note that when k ! 1 , ¯ ↵ LM 2 c is the original optimality gap (i.e. E {L ( w k )  L ⇤ } ) with real gradients, see Bottou et al.
[14, Theorem 4.6], and then, the optimality gap is also increased by ¯ ↵ LM q 2 c due to ﬁxed-point representation.
Here we argue that by keeping the variance bound M q relatively small via choosing the correct number format, we can theoretically achieve the original performance.
On the other hand, optimality gap is related to ¯ ↵ , which means smaller learning rates leads to smaller optimality gap.
Remark 4 (Local convexity).
Strongly convex loss is not a realistic assumption in large deep learning models.
However, local convexity is a more realistic assumption i.e. L is convex around the minimum point w ⇤ .
Hence, inequality (12) still holds around the minimum point w ⇤ of a locally convex loss.
We essentially train our ImageNet ResNet18 classiﬁcation in the same way: ﬁrst we train the network with a ﬁxed learning rate until reaching a certain optimality gap; then the learning rate is reduced to a smaller ﬁxed value in order to shrink the optimality gap.
Empirical evidence: Figure 3(a) demonstrates the locally convex loss landscape of ResNet18 training on CIFAR10 dataset using ﬂoating-point computation.
Figure 3(b) shows the same loss landscape in the ﬁxed-point int8 format.
We perturbed the weights around the w ⇤ in x and y axes using Gaussian noise and evaluated the loss L in the z axis to acquire Figures 3(a) and 3(b).
Comparing these two ﬁgures pronounces our assumption of local convexity in both integer and ﬂoating-point tests.
Figure 3(c) shows a comparison of the loss trajectory for ﬂoating-point and integer training.
The ﬁxed-point gradients are unbiased estimators of the true gradients, so the trajectory of the integer training closely follows the trajectory of its ﬂoating-point counterpart.
Remark 5.
We implemented an integer weight update, hence, the computations of equation (6) is also performed in integer arithmetic.
It is shown in Appendix A.4 that integer weight update with stochastic rounding is an unbiased estimator of the true weight update.
[Figure]
Experimental results
Image classiﬁcation: Table 1 reports the experimental results of our proposed integer training method on the conventional vision classiﬁcation models.
In this set of models, we used int8 linear layer, int8 convolutional layer, int8 batch-norm layer, and int16 SGD to form a fully integer training pipeline.
Note that all forward and back-propagation computations of layers are performed in integer format.
Moreover, we have chosen smaller vision models such as ResNet18 and MobileNetV knowing that smaller models are harder to train.
Our experimental results show that our proposed integer training method is as good as ﬂoating-point with negligible loss of accuracy ( 6 0 .
5% on
ImageNet).
Remark 6.
We compare our integer training results with ofﬁcial Pytorch ﬂoating-point training.
For instance, in conventional vision models (Table 1), we compare the results directly with the accuracy of torchvision models that are reported on the ofﬁcial Pytorch website 1 .
We further emphasize that there is no change in hyper-parameters of the training.
The full set of hyper-parameters of our experiments are reported in Appendix A.5.
Vision transformer:
We validated the applicability of our proposed integer training on the original vision transformer model, notably ViT-B-16-224 [15].
We took the ﬂoating point checkpoint pretrained on ImageNet21K 2 .
We used Huggingface [16] to ﬁne-tune the model on CIFAR10.
In this experiment, we used int8 linear layer, int8 matrix multiplication, int8 convolutional layer, and int8 layer-norm for our integer training pipeline.
The result is reported in Table 1 demonstrating negligible loss of accuracy ( 6 0 .
Note that for this experiment, the computation of softmax in attention mechanism is in ﬂoating point.
Semantic segmentation: To validate our proposed integer training on semantic segmentation task, we used DeepLabV1/V2 3 with ResNet-101 as the backbone.
We trained the models on PASCAL
VOC-2012 [17] and MS COCO 10K [18].
For the PASCAL dataset, as suggested by Chen et al.
[19], the models were initialized with MS COCO checkpoint and data augmentation was used during training.
Likewise, for COCO dataset we used PASCAL checkpoint for initialization.
The batch-norm layers are frozen in our experiments as suggested in [19].
We used linear and convolutional layers in int8 format.
Table 2 shows that the mean intersect of union (mIOU) of our method closely matches the ﬂoating-point baseline.
Object detection: We used Faster R-CNN[20] with ResNet50 as its backbone and Single Shot
Detector (SSD)[21] which relies on VGG-16 for its backbone.
In these experiments, we used int linear and convolutional layers, while batch-norm layers are frozen.
We integrated our integer training method with MMDetection[22] toolbox and performed our experiments on MS COCO 10K[18],
PASCAL VOC-2007[23], VOC-2012[17], and Cityscapes[24] datasets.
As shown in Table 3, the mean average precision (mAP) of our proposed integer training method is very close to ﬂoating-point baseline.
[Table] [Table]
[Table] [Table]
[Table]
Comparison with state of the art: Table 4 provides a comparison between our training method and state of the art across different experiments.
There are some important differences between our proposed method and other works: (i) our proposed integer training method uses a ﬁxed-point batchnorm layer where both forward and back propagation is computed using integer arithmetic, (ii) our proposed training method uses an integer-only SGD, (iii) in our training method, no hyper-parameter is changed while other methods have changed hyper-parameters or used gradient clipping.
Low-bit integer training: Table 5 provides an ablation study of how lowering integer bit-width can affect the training accuracy.
Our experiments shows that training has a signiﬁcant drop of accuracy with int5 and diverges using int4 number formats.
Conclusion
We proposed a hardware-friendly integer training method based on coupling a linear ﬁxed-point mapping with a non-linear inverse mapping.
This method performs the representation mapping directly on the ﬂoating-point number format.
Furthermore, our method uses stochastic rounding and produces unbiased estimators of gradients in the back-propagation.
Thus, beneﬁting from unbiased gradients and effective representation mapping, the training loss trajectory closely follows its ﬂoatingpoint version.
Moreover, there is no need to tune hyper-parameters or perform gradient clipping to correct the inappropriate gradient and weight updates.
Using our proposed technique, we designed the integer version of the most vital components of deep learning such as linear layer, convolutional layer, batch-norm, layer-norm, and stochastic gradient descent (SGD).
Our experimental results show the effectiveness of the proposed method, where the accuracy loss is negligible in a wide variety of the training tasks.
Furthermore, in our classiﬁcation tests all training components are in integer arithmetic.
We theoretically studied the effect of our proposed method on SGD, and demonstrated why the optimality gap of convergence is shifted by a negligible amount.
Acknowledgments and Disclosure of Funding
The authors would like to thank Richard Wu and Vanessa Courville for their constructive comments.
The authors would like to thank Sara Elkerdawy for initiating the object detection and semantic segmentation experiments during her internship at Huawei
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Theorem 1 and Remark 3 in Section 4.2 reﬂecting the effect of representation mapping noise.
Moreover, the accuracy deviation from baseline ﬂoating-point training experiments are reported in Table 1, Table 2, and Table(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The work is purely methodological and related to number format.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.1, Appendix A.2, Appendix A.3, and Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code is proprietary, however we will provide them upon publication.
We ran the experiments on publicly available datasets.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] .
It was too costly to train multiple times, we just ran the experiments once.
Also note the type of training experiments we reported are resilient to random initialization changes.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]