Is Integer Arithmetic Enough for Deep Learning Training?
Anonymous Author(s)
AffiliationAddress email
Abstract
The ever-increasing computational complexity of deep learning models makes theirtraining and deployment difficult on various cloud and edge platforms.
Replacingfloating-point arithmetic with low-bit integer arithmetic is a promising approachto save energy, memory footprint, and latency of deep learning models.
As such,quantization has attracted the attention of researchers in recent years.
However,using integer numbers to form a fully functional integer training pipeline includingforward pass, back-propagation, and stochastic gradient descent is not studiedin detail.
Our empirical and mathematical results reveal that integer arithmeticseems to be enough to train deep learning models.
Unlike recent proposals, insteadof quantization, we directly switch the number representation of computations.
Our novel training method forms a fully integer training pipeline that does notchange the trajectory of the loss and accuracy compared to floating-point, nor doesit need any special hyper-parameter tuning, distribution adjustment, or gradientclipping.
Our experimental results show that our proposed method is effective in awide variety of tasks such as classification (including vision transformers), objectdetection, and semantic segmentation.
Introduction
Recently, deep learning models have evolved to deliver acceptable performance in many areas suchas computer vision, natural language processing, and speech recognition.
However, the numberof parameters and computational complexity of most deep learning applications have increasedsignificantly.
This ever-increasing computational complexity makes the deployment of deep neuralnetworks harder on edge devices.
Furthermore, the energy required to train a deep learning modelincreases proportionately with computational complexity.
Thus, the training cost of these large andcomplex models also increases significantly on the high-end cloud servers.
Although, quantizationtechniques are commonly used to accelerate inference of deep learning models, here we target afully integer training pipeline (i e. forward propagation, back-propagation and SGD) .
Unlike recentproposals in quantized back-propagation, we directly change the number representation of floatingpoint values.
Furthermore, we show empirically and theoretically that our proposed integer trainingmethodology is effective in training deep learning models with integer-only arithmetic .
Despite the fact that accelerated training using integer back-propagation seems intriguing, thereare some challenges associated with integer gradient computation.
For example, (i) gradients mustbe scaled correctly in order to be adapted to the limited dynamic range of the integer number (e.g.int8 : [ − 128 , 127] ).
(ii)
The numerical error of the gradient must be unbiased in order to preservethe convergence trajectory of the training algorithm.
A small numerical error can be accumulatedthrough the course of training and change the convergence behaviour.
(iii)
The integer training must be distribution independent , i.e. the training method should not depend on the distribution of thegradients, weights, or training data.
We propose a novel integer training method designed to address aforementioned challenges (i), (ii)and (iii) simultaneously.
Having a linear dynamic fixed-point mapping coupled with a non-linearinverse mapping is the key to full integer training.
To this end, we make the following contributions:
• A hardware-friendly integer training method is proposed based on extracting the maximumfloating-point exponent of the tensors as the scale .
We propose linear fixed-point mappingof tensors, while the corresponding inverse mapping for floating-point is non-linear.
Ourproposed method addresses all the previously mentioned challenges: (i) it computes thescales dynamically, and moreover the scale does not need to be adjusted in the course oftraining; (ii) it provides an unbiased estimation of the gradients and consequently, its convergence trajectory closely follows the floating-point version; (iii) our proposed representationmapping does not depend on the distribution of the training data, weights, or
gradients.•
We study the optimality gap of our proposed integer training algorithm and show it isanalogous to its floating-point counterpart in the course of training ( Theorem 1 ).
Ouranalysis of stochastic gradient descent with our proposed fixed-point gradients shows theoriginal floating-point optimality gap is only shifted by a negligible amount ( Remark 3 ).•
Our proposed method effectively performs all operations required in modern neural networksusing integer-only arithmetic .
For instance, the computation of linear layer, convolutionallayer, batch-norm and layer-norm, residual connections and also stochastic gradient descent(including gradients, momentum, weight decay, and weight update) are all performed ininteger arithmetic.
To the best of our knowledge, this is the first time that back-propagationof a batch-norm, and the computation of stochastic gradient descent (SGD) is performed ininteger arithmetic with negligible loss in the accuracy for large datasets such as ImageNet.
The rest of this paper is structured as follows.
Section 2 reviews some previous works in the field ofquantized back-propagation and quantifies the similarities and differences with our representationmapping method.
Section 3 discusses our integer training methodology in detail.
Theoretical aspectsof our integer training method on SGD are studied in Section 4.
Experimental results supporting ourinteger training methodology and convergence theory are presented in Section 5.
Related works
Banner et al.
[1], proposed a bifurcated back-propagation method where the gradient of weights arein floating-point format and input gradients are in 8-bit fixed-point format.
Moreover, they proposeda range-based batch-norm which is more tolerant to quantization noise.
The main difference ofour proposed algorithm with Banner et al.
[1] is that our integer training method does not demandbifurcation.
In other words, our gradients with respect to weights and inputs are represented andcomputed using int8 format.
Furthermore, in our method, the back-propagation of batch-norm, aswell as stochastic gradient descent (SGD) algorithm, are also represented and computed using integervalues.
Zhang et al.
[2] have proposed a layer-wise “precision-adaptive” quantization method thatdoes not affect the distribution of the data.
In their method, the quantization error is measured andthe quantization scale is adjusted over the course of training accordingly.
Similarly, in Zhao et al.
[3]a distribution adaptive quantization method is proposed that takes into account the distribution ofgradients in the channel dimension.
In addition, they introduced a gradient clipping strategy in orderto normalize the magnitude of the gradients in the back-propagation.
The scale of the quantizationis adapted to the distribution of the gradients and adjusted iteratively over the course of training.
Unlike the distribution adaptive methods such as [2, 3], our proposed integer training method doesnot depend on the distribution of training data, weights, and gradients.
Thus, there is no need toadjust the scale iteratively in the course of training.
In Zhu et al.
[4], a “direction sensitive gradientclipping method” is proposed to avoid inappropriate updates in the back-propagation.
Moreover, theyproposed a learning rate scaling that tackles the problem of unbiased gradient error accumulation.
Ina similar work, Sakr and Shanbhag [5] proposed a precision assignment methodology to improve theconvergence of the quantized back-propagation.
This precision assignment is based on some criterionon internal accumulator noise, quantization noise of backward and forward propagation, and gradient clipping.
Unlike [4, 5], our proposed method enjoys having the advantage of unbiased gradients ,hence the convergence trajectory closely follows the floating-point counterpart.
Thus, our methoddoes not need any specific gradient correction, gradient clipping strategy, and learning rate correctionsas opposed to [4, 5].
Jin et al.
[6] proposed a unified fixed-point and parameterized clipping activationto achieve high accuracy.
Furthermore, they proposed a method that directly trains the fractionallength (i.e. scale) of the fixed-point quantization.
Additionally, they use a double forward batch-normfusion to determine the scaling factor of the batch-norm layer in the forward propagation, while theback-propagation of batch-norm remains floating-point.
In contrast, our proposed integer trainingmethod is capable of implementing integer variant of the batch-norm layer in forward propagationwithout double fusion as opposed to [6].
We also perform batch-norm’s back-propagation in integerarithmetic which was ignored in the previous works.
Note that back-propagation of batch-normand layer-norm is sensitive to quantization, as such, naive quantization leads to training divergence.
Additionally, we perform stochastic gradient descent (SGD) computation including weight update,weight decay, and momentum in integer arithmetic with no significant loss of accuracy.
Wang et al.[7] explored an integer-arithmetic implementation of some preliminary CNN architectures such as
AlexNet, however they experienced considerable accuracy drop for ImageNet dataset.
Moreover,more complicated architectures such as ResNet and transformers are not considered.
In our proposedmethod, we used dynamic fixed-point number format [8] as the primary number format for integertraining.
This specific fixed-point number format (also known as block floating-point format) has beenused previously to quantize inference of deep learning models in Courbariaux et al. [8], Drumondet al.
[9], and Rouhani et al.
Methodology
Common quantization methods, which use division and clipping techniques (refer to Appendix A.6),are inefficient for back-propagation.
Therefore, we decided to go one step deeper and change thenumber format directly.
Here, we propose a hardware-friendly number representation mapping usingthe dynamic fixed-point number format where the scale is defined per tensor .
In this approach, eachtensor can be represented by its int8 version while it is multiplied by a shared scale, as opposed toother methods that allow multiple shared scales for different partitions of the tensor (see Rouhaniet al.
[10, Figure 4]).
One shared scale per tensor makes the computations easier in the computinghardware (e.g. CPU or GPU).
However, the training algorithm diverges if the representation mappingis not executed properly.
To tackle this problem, we suggest two subtle changes; (i) we propose toperform fixed-point mapping of a tensors in a linear fashion while the inverse mapping is non-linearas explained in Sections 3.1 and 3.2.
This is the key to success of our algorithm since a linearfixed-point mapping allows monotonic conversion of floating-point format to fixed-point, while anon-linear inverse mapping allows preserving information.
(ii) We suggest to use stochastic roundingin the back-propagation in a way that preserves the expected value of the tensors as well as theirvital statistics, such as the mean.
Stochastic rounding in conjunction with representation mapping iscrucial to keep the integer training loss trajectory close to its floating-point counterpart.
3.1 Linear fixed-point mapping
Our proposed integer training method is based on manipulating the floating-point number format.
This is a very simple and effective way of converting floating-point values to fixed-point values asopposed to other commonly used methods that involve division operation.
Our method essentiallyuses shift and round operations to convert floating-point to fixed-point format, see Figure 1(a).
In this method, a tensor comprising n floating-point numbers ( f 1 , f 2 , ..., f n ) is converted to a fixedpoint tensor.
First, as shown in Figure 1(a), sign, exponent, and mantissa of each floating-pointnumber are extracted using the unpack to integer function.
For instance, f 1 is unpacked to s 1 , e 1 , m 1where ( s, e, m ) is used to denote ( sign , exponent , mantissa ).
We simply find the maximum element ofe 1 , e 2 , ...
e n i.e. e max = max( e 1 , e 2 , ...
e n ) to extract the shared scale of the tensor.
Subsequently,the original mantissas are shifted to the right according to the difference of their exponent and e max ,to get the individual mantissas.
For instance, m 1 is shifted to right by e max − e 1 which is denotedby m 1 >> ( e max − e 1 ) .
By shifting mantissas to right, we intentionally push the small values tothe sub-normal region, where the most significant bit of mantissa is not 1 in binary format i.e. (1) 2 ,see Zuras et al.
Pushing the mantissas to the sub-normal region is performed to align theirexponents with e max and this unifies the scale for fixed-point values in a tensor.
In the next step, to f f fn ...
Unpack To Integer
Float Tensor ...
e max m 1 >>e max - em 2 >>e max - em n >> e max - e n
Shifted 24-bit Mantissas S n S 2 S
S n S 2 S
Stochastic Rounding ...
Signed 8-bit Mantissas
Send to Integer Layer
Singed 8-bit Mantissas 8-bitScale 24-bitMantissa 8-bitScale
S 1 e 1 m
S 2 e 2 m
S n e n m n
Signedmantissas Sharedexponent
Rounding /
Extracting sign
Sign
Exponent
Mantissa
Floating point alignment (Using LZA unit)
Pack to float
Float Tensor
L aye r C
o m pu t a t i on e .
g .
L i n ea r L aye r
N on - li n ea r
I n ve r se
M a
pp i ng
Add
Integer Linear Layer (GEMM)
Reconstructing individual exponents
[Figure]
create int8 mantissas, the 24-bit single floating-point mantissas (i.e. 23-bit mantissa + 1 implicithidden bit) are further rounded to 7-bit mantissas to construct signed int8 values i.e 7-bit unsignedinteger and 1 sign bit.
3.2 Non-linear inverse mapping
Our proposed fixed-point mapping method is performed by pushing mantissa values to the sub-normalregion and rounding them stochastically.
Hence, the inverse mapping must be performed using afloating-point alignment module that normalizes the mantissas.
A normalized floating-point mantissais required to start with the most significant bit (1) 2 .
For instance, an alignment module converts2 127 × (0 . 0101) 2 to 2 127 − 2 × (1 . 0100) 2 .
The alignment module is a very well-known logic circuitthat is available in the commodity hardware using Leading Zero Anticipator (LZA) block [12].
Coupling a linear fixed-point mapping with a non-linear inverse mapping in this way keeps thenumber format close to floating-point.
Moreover, combining them with stochastic rounding, resultsin having unbiased fixed-point variant that forces the trajectory of the integer training closely followthe floating-point counterpart.
Figure 1(b) shows the inverse mapping unit.
This unit receives a tensor of mantissas and a singlevalue of shared exponent computed by an integer layer.
Then a tensor of exponents is reconstructedby repeating the value of the shared exponent.
The size of this tensor is equal to the mantissa tensor’ssize.
Moreover, the mantissa tensor is rounded and its sign is extracted.
Note that the rounding isused here because the output of the previous layer might have some excessive mantissa bits; thisnormally happens in matrix multiplication and convolution operations.
Then the sign, exponent, andmantissa tensors are sent to an alignment unit that shifts the mantissa and adjust the exponent in orderto normalize the floating-point number format [11].
Finally, the result is packed and carried out to thenext layer.
3.3 Integer layer computations
When the fixed-point mapping of floating-point values is completed, the fixed-point values are used toperform the integer-only layer computations.
For instance, let us consider a linear layer which consistsof a General Matrix Multiplication (GEMM) operation, but, the idea can be generalized to othertypes of layers.
Figure 2 demonstrates the layer-wise integer computations of a linear layer where theshared exponents and integer mantissas are treated separately.
As shown in the Figure 2, in orderto perform an integer-only GEMM operation, we need to multiply scales (2 e max1 × 2 e max2 ) .
This
Unpack To Integer
[Figure]
Non-linear Inverse Mapping Module or Next Layer
L i n ea r F i xe d - po i n t
M a
pp i ng
L aye r C
o m pu t a t i on e .
g .
L i n ea r L aye r
[Figure]
multiplication performs an integer addition operation on the exponents ( e max1 + e max2 ) .
Furthermore,the integer mantissas are sent to an integer GEMM module to compute the output mantissa tensor.
3. Understanding the representation mapping
Here we provide an intuition of how our proposed representation mapping approach works.
Letus denote A as a tensor and ˆ A as its fixed-point version in the way that is introduced earlier.
Inaddition, random variables
A i and ˆ A
i are i th element of those tensors.
Also note that A i and ˆ A
i aredifferent representations of the same real number, but one is in floating-point format and the otheris in dynamic fixed-point format.
Thus, one can relate A i and ˆ A i with a random error term δ i asˆ
A i
=
A i
+ δ i .
Since in our training method we used stochastic rounding [13], E { ˆ A i } =
A i , orequivalently E { δ i } =
(see Appendix A.1).
In other words, the fixed-point value is on the averageequal to the floating-point value.
Note that here we consider single precision floating-point values asa surrogate of real values.
Linear and convolutional layers: For these two types of layers, both forward and backwardpropagation computations are based on inner products.
Thus, it is easy to see that our proposedfixed-point inner product ˆ C ij = (cid:80)
k
ˆ A ik ˆ B kj is an unbiased estimator of the floating-point innerproduct
[Equation]
Residual connections: A residual connection involves element-wise addition of two tensors.
Eachfixed-point element ˆ C ij =
ˆ A ij + ˆ B ij is an unbiased estimator of its floating-point version
[Equation]
Batch-norm: A batch-norm layer is defined as
[Equation]
For this type of layer, it is important to compute signal statistics such as mean ˆ µ and variance ˆ σ 2correctly in integer arithmetic.
Thus, the fixed-point mean ˆ µ is also an unbiased estimator of the truemean µ
[Equation]
Then the following derivation holds for fixed-point variance ˆ σ
[Equation]
where σ 2 is the true floating-point variance of the batch and σ 2 δ is the variance of the noise introducedby the linear mapping to fixed-point.
Note that the error variance σ 2 δ is rather small and can beintegrated to ϵ in the denominator of equation (3).
Theoretical analysis of SGD using the proposed method
The generic equation of weight update in the k th iteration of SGD is
[Equation]
where g ( w k , ξ k ) is the estimated gradients of random samples of the batch generated by the seed ξ k ,and α k is the learning rate.
We make the following common assumption in the sequel.
Assumption 1 (Lipschitz-continuity).
The loss function L ( w ) is continuously differentiable and itsgradients ∇L ( w ) satisfies the following inequality where L > 0 is the Lipchitz constant
[Equation]
Assumption 2.
(i) L ( w k ) is bounded.
(ii)
Estimated gradients g ( w k , ξ k ) is an unbiased estimator of the true gradients of the loss function
[Equation]
and (iii,a) there exist scalars M ⩾ 0 and M V ⩾
such that for all iterations of SGD 205 V
[Equation]
Note that here we define
[Equation]
Also from Assumption 2.
(ii) and (iii,a) , the second moment bound can be derived
[Equation]
Effect of gradient variance on convergence: The quality of the estimated gradients g ( w k , ξ k )directly affects the convergence of the SGD.
The effect of first and second moments of gradient arealready studied on real numbers in the literature.
Lemma 2.
Suppose Assumption 2 is true, then we have
[Equation]
Proof: See Bottou et al.
[14, Lemma 4.4].
Inequality (9) shows the effect of gradient variance bounds, M and M G , on each iterate of SGD, andshows the greater the variance, the more deterioration in the quality of SGD steps.
The first term,− (1 − 12 α k LM G ) α k ||∇L ( w k ) ||
22 contributes to the decrease of the loss function while the secondterm, 12 α 2 k LM , prevents it.
Upon choosing the correct gradient estimates, the right hand side ofinequality (9) is bounded by a deterministic quantity, and asymptotically ensures sufficient descent ofthe loss L ( w ) .
Note that the expectation E ξ k is taken over random samples with seed ξ k .
4.1 Fixed-point gradient noise
When performing representation mapping in the back-propagation, the quality of the gradientsdeteriorate.
Thus, there is a need to consider the effect of fixed-point mapping variance.
Then,
Assumption 2 (iii,a) should be modified accordingly.
Remark 1.
Note that Assumption 2 (i), (ii) still hold after fixed-point mapping because of stochasticrounding, i.e. the fixed-point gradient remains an unbiased estimator of gradient (refer to Appendix
A.1).
Assumption 2 (iii,b).
When the gradients are in fixed-point format i.e. ˆ g ( w k , ξ k ) , there exist scalars
M ⩾ 0 , M V ⩾ 0 , M q ⩾ 0 and M qV ⩾ 0 such that for all iterations of SGD
[Equation]
If ˜ M := M + M q and ˜ M V :=
M V + M qV , then Assumption 2 (iii,b) takes the exact form of
Assumption 2 (iii,a) i.e. V ξ k { g ( w k , ξ k ) } ⩽ ˜ M + ˜ M V ||∇L ( w k ) || 22 .
However, here we separated M qand M qV to emphasize the effect of fixed-point mapping on the true gradients.
Remark 2.
If Assumption 2 (iii,b) holds true, inequality (9) can be transformed to its fixed-pointversion
[Equation]
with M qG
[Equation]
=
[Equation]
M qV
[Equation]
Inequality (10) shows the effect of added representation mapping variance with bounds, M q and M qG ,on each iterate of SGD.
This observation shows that fixed-point mapping degrades the convergenceof SGD unless its variance bounds are relatively small, or controlled by the learning rate.
As anexample, refer to Appendix A.2 for analytical derivations of M q and M qG for the back-propagation ofa linear layer involving a fixed-point inner product.
4.2 Strongly convex and locally convex loss
[Figure]
Assumption 3 (Strong convexity).
The loss function L ( w ) is differentiable and strongly convex.
Werecall that strong convexity for differentiable functions is equivalent to the following inequality withsome constant c >
[Equation]
A strongly convex function has a unique minimum point at w ∗ with the loss value L ∗ =
L ( w ∗ ) .
Theorem 1.
Suppose Assumptions 1, 2(i), 2(ii), 2 (iii,b), 3 are all true, then a SGD method runningwith fixed-point gradients i.e. ˆ g ( w k , ξ k ) and a fixed learning rate
<
¯ α ⩽ 1 L ( M + M q ) satisfies thefollowing bound for its optimality gap with the minimum loss L ∗ at the k th iteration
[Equation]
Proof.
See Appendix A.3.
Remark 3.
Note that when k −→ ∞ , ¯ αLM 2 c is the original optimality gap (i.e. E {L ( w k ) − L ∗ } ) withreal gradients, see Bottou et al.
[14, Theorem 4.6], and then, the optimality gap is also increasedby ¯ αLM q 2 c due to fixed-point representation.
Here we argue that by keeping the variance bound M qrelatively small, we can theoretically achieve the original performance.
On the other hand, optimalitygap is related to ¯ α , which means smaller learning rates leads to smaller optimality gap.
Remark 4 (Local convexity).
Strongly convex loss is not a realistic assumption in large deep learningmodels.
However, local convexity is a more realistic assumption i.e. L is convex around the minimumpoint w ∗ .
Hence, inequality (12) still holds around the minimum point w ∗ of a locally convex loss.
We essentially train our ImageNet ResNet18 classification in the same way: first we train the networkwith a fixed learning rate until reaching a certain optimality gap; then the learning rate is reduced to asmaller fixed value in order to shrink the optimality gap.
Empirical evidence: Figure 3(a) demonstrates the locally convex loss landscape of ResNet18 trainingon CIFAR10 dataset using floating-point computation.
Figure 3(b) shows the same loss landscape in
[Figure]
the fixed-point int8 format.
We perturbed the weights around the w ∗ in x and y axes using Gaussiannoise and evaluated the loss L in the z axis to acquire Figures 3(a) and 3(b).
Comparing these twofigures pronounces our assumption of local convexity in both integer and floating-point tests.
Figure3(c) shows a comparison of the loss trajectory for floating-point and integer training.
The fixed-pointgradients are unbiased estimators of the true gradients, so the trajectory of the integer training closelyfollows the trajectory of its floating-point counterpart.
Remark 5.
We implemented an integer weight update, hence, the computations of equation (6) isalso performed in integer arithmetic.
It is shown in Appendix A.4 that integer weight update withstochastic rounding is an unbiased estimator of the true weight update.
Experimental results
Image classification: Table 1 reports the experimental results of our proposed integer training methodon the conventional vision classification models.
In this set of models, we used int8 linear layer,int8 convolutional layer, int8 batch-norm layer, and int16 SGD to form a fully integer trainingpipeline.
Note that all forward and back-propagation computations of layers are performed in integerformat.
Moreover, we have chosen smaller vision models such as ResNet18 and MobileNetV2knowing that smaller models are harder to train.
Our experimental results show that our proposedinteger training method is as good as floating-point with negligible loss of accuracy ( ⩽ 0 .
Remark 6.
We compare our integer training results with official Pytorch floating-point training.
Forinstance, in conventional vision models (Table 1), we compare the results directly with the accuracyof torchvision models that are reported on the official Pytorch website 1 .
We further emphasizethat there is no change in hyper-parameters of the training.
The full set of hyper-parameters of ourexperiments are reported in Appendix A.5.
Vision transformer:
We validated the applicability of our proposed integer training on the originalvision transformer model, notably ViT-B-16-224 [15].
We took the floating point checkpoint pretrained on ImageNet21K 2 .
We used Huggingface [16] to fine-tune the model on CIFAR10.
In thisexperiment, we used int8 linear layer, int8 matrix multiplication, int8 convolutional layer, andint8 layer-norm for our integer training pipeline.
The result is reported in Table 1 demonstratingnegligible loss of accuracy ( ⩽ 0 .
Note that for this experiment, the computation of softmax inattention mechanism is in floating point.
Semantic segmentation: To validate our proposed integer training on semantic segmentation task,we used DeepLabV1/V2 3 with ResNet-101 as the backbone.
We trained the models on PASCAL
VOC-2012 [17] and MS COCO 10K [18].
For the PASCAL dataset, as suggested by Chen et {+al.[19], the models were initialized with MS COCO checkpoint and data augmentation was used during html Refer to: https://huggingface.co/google/vit-base-patch16-
 Refer to : https://github.com/kazuto+}
[Table]
[Table] [Table] [Table] training.
Likewise, for COCO dataset we used PASCAL checkpoint for initialization.
The batch-normlayers are frozen in our experiments as suggested in [19].
We used linear and convolutional layers inint8 format.
Table 2 shows that the mean intersect of union (mIOU) of our method closely matchesthe floating-point baseline.
Object detection: We used Faster R-CNN[20] with ResNet50 as its backbone and Single Shot
Detector (SSD)[21] which relies on VGG-16 for its backbone.
In these experiments, we used int8linear and convolutional layers, while batch-norm layers are frozen.
We integrated our integer trainingmethod with MMDetection[22] toolbox and performed our experiments on MS COCO 10K[18],
PASCAL VOC-2007[23], VOC-2012[17], and Cityscapes[24] datasets.
As shown in Table 3, themean average precision (mAP) of our proposed integer training method is very close to floating-pointbaseline.
Comparison with state of the art: Table 4 provides a comparison between our training methodand state of the art across different experiments.
There are some important differences between ourproposed method and other works: (i) our proposed integer training method uses a fixed-point batchnorm layer where both forward and back propagation is computed using integer arithmetic, (ii) ourproposed training method uses an integer-only SGD, (iii) in our training method, no hyper-parameteris changed while they have changed hyper-parameters or used gradient clipping.
Conclusion
We proposed a hardware-friendly integer training method based on coupling a linear fixed-pointmapping with a non-linear inverse mapping.
This method performs the representation mapping directly on the floating-point number format.
Furthermore, our method uses stochastic rounding andproduces unbiased estimators of gradients in the back-propagation.
Thus, benefiting from unbiasedgradients and effective representation mapping, the training loss trajectory closely follows its floatingpoint version.
Moreover, there is no need to tune hyper-parameters or perform gradient clipping tocorrect the inappropriate gradient and weight updates.
Using our proposed technique, we designedthe integer version of the most vital components of deep learning such as linear layer, convolutionallayer, batch-norm, layer-norm, and stochastic gradient descent (SGD).
Our experimental results showthe effectiveness of the proposed method, where the accuracy loss is negligible in a wide varietyof the training tasks.
Furthermore, in our classification tests all training components are in integerarithmetic.
We theoretically studied the effect of our proposed method on SGD, and demonstratedwhy the optimality gap of convergence is shifted by a negligible amount.
Checklist
For all authors...(
a ) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Theorem 1 and Remark 3 in Section4.2 reflecting the effect of representation mapping noise.
Moreover, the accuracy deviation frombaseline floating-point training experiments are reported in Table 1, Table 2, and Table {+
+}
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The
work is purely methodological and related to number format.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.1, Appendix A.2, Appendix A.3, and Appendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[No] The code is proprietary, however
we will provide them upon publication.
We ran the experiments on publicly available datasets.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[No] .
It was too costly to train multiple times, we just ran the experimentsonce.
Also note the type of training experiments we reported are resilient to random initializationchanges.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Appendix A.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa-
tion or offensive content? [N/A] If you used crowdsourcing or conducted research with human subjects... a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] ) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)approvals, if applicable? [N/A] c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A] A Appendix
A.1 Stochastic Rounding
In this paper, stochastic rounding for a variable x where x 1 < x < x 2 is defined as
[Equation]
13) it is easy to see that ˆ x is an unbiased estimator of x :
[Equation]
14) and if we relate x and ˆ x using an error term δ (i.e. ˆ x = x + δ ), then E { δ } = 0 .
A.2 Representation mapping increases the gradients variance: Linear layer example
A linear layer is essentially a matrix multiplication.
Let us denote X as inputs, W as weights and Y as the 439 output of a linear layer where Y = XW .
Following the notation of this paper, we also denote our fixed-pointversion of this layer as ˆ Y =
ˆ X ˆ W .
In our proposed integer back-propagation, the layer receives the upstream gradient ˆ G :
=
∂ ˆ L∂ ˆ Y .
Using thechain-rule, the gradient with respect to weights is
[Equation]
Thus, computation of gradient with respect to weights is essentially a matrix multiplication too.
We are interestedin understanding the relation between variance of ˆ C =
ˆ X ⊤ ˆ G in the integer back-propagation and the truegradients C = X ⊤ G .
For each element ˆ C ij we have
[Equation]
Note that in inequality (16), σ 2 G = max( V { δ Gi,j } ) knowing that error terms δ Gi,j have essentially similardistributions.
Likewise, σ 2 X = max( V { δ Xi,j } ) .
Also note that X and G are matrices of random variables in theback-propagation and || X ⊤ i,
|| 22 denotes the norm-2 of the i th row of X ⊤ and || G ⊤ ,j || 22 denotes the norm-2 of thej th column of G .
Also, ˆ X ⊤ ik denotes ik th element of the matrix ˆ X ⊤ .
By having the following definitions
[Equation]
we can re-organize the inequality (16) as
[Equation] [Equation]
Remark.
Inequality (18) supports our Assumption 2 (iii,b) i.e.
[Equation]
for considering the effect of our representation mapping method on gradients variance.
A.3 Proof of Theorem
The proof goes along the proof of Bottou et al.
[14, Theorem 4.6] in the case that the gradient variance bound 455 increased as stated in Assumption 2 (iii, b) .
A convex function satisfying the inequality (11) given ¯ w, w ∈ R d represents a quadratic model
[Equation]
19) and has a unique minimizer at w ∗
[Equation]
Also remember that the fixed learning rate in our integer back-propagation has the following constraint
[Equation]
Starting from inequality (10) and using inequalities (20) and (21) we can write
[Equation]
[Equation]
By subtracting L ∗ , rearrange, and taking total expectation from both sides of inequality (22) we have
[Equation]
Then by subtracting ¯ αL ( M + M q ) 2 c from both sides
[Equation]
[Equation] Thus, Theorem 1 can be proven by applying inequality (24) repeatedly for k ∈ N .
Also note that using inequality(21)
it is easy to derive that 0 < (1 − ¯ αc ) < 1 because
[Equation]
[Equation]
hence, in Theorem 1 , if k −→ ∞ , then (1 − ¯ αc ) k −→ 0 and the optimality gap of integer training algorithm usinginequality (12) is [Equation]
[Equation]
A.4 Integer weight update
In the integer weight update, the equation (6) transforms to its fixed-point version with integer-only arithmetic
[Equation]
By expanding the error terms for the representation mapping and taking expatiation on both sides we can see the 469 weights are on the average updated equivalent to the true wights.
[Equation]
We used stochastic rounding for our weight update operation, thus E { δ } = 0 and using our proposed method, ˆ gis unbiased estimator of g .
A.5 Experimental setup
Computing resources: We ran our experiments using an in-house developed integer emulator to avoid common 474 floating-point quantization techniques.
In our emulator, we perform the representation mapping within the GPUmemory.
We further developed the integer deep learning modules using Pytorch autograd functionality on top of 476 our integer emulator.
Experimental results of this paper are run using the following number of GPUs.
• ResNet18 on ImageNet requires 4 × V100 GPUs when batch size is 512 and 2 × V100 GPUs whenbatch size is 256.
[Table]
• MobileNetV2 on ImageNet requires 8 × V100 GPUs when batch size is 512.
[Table]
• ResNet18 on CIFAR10 runs on 1 × V100 GPUs when batch size is 128.
[Table]
• Semantic segmentation experiments require 2 × V100 GPUs.
[Table]
• Object detection experiments require 8 × V100 GPUs. [Table] • ViT-B fine-tuning experiment requires 8 × V100 GPUs.
[Table]
Classification: The hyper-parameters of our classification experiments are reported in Table 5.
We used SGD 485 with momentum of 0.9 for conventional classification experiments and AdamW for ViT fine-tuning [Table]
[Table]
Semantic segmentation:
The hyper-parameters for the semantic segmentation experiments are provided in 487 Table6.
For DeepLabV1 and DeepLabV2 one-scale and multi-scale loss was used respectively.
Conditionalrandom field post-processing was used at the network’s output as proposed in Chen et al.
In all the 489 experiments we use SGD with momentum of 0.9 and weight decay of 5 × 10 − 4 .
[Table]
detection: The hyper-parameters are provided in Table 7.
All experiments use an SGD optimizer with a 491 momentum of 0.9 and a weight decay parameter of 10 − 5 .
The experiments indicated with LR Warmup use alinear warm-up function with a ratio of 10 − 3 for the first 500 iterations.
For the rest of training, the learning rateis reduced by 0.1 at the epochs indicated in the LR Reduction Epochs column.
[Table]
A.6 Background on quantization methods
[Table]
Symmetric uniform quantization is normally used in the literature of quantized back-propagation (see [2, 4, 3] )to convert the gradients to 8-bit integers.
Suppose scale s = max( | x | ) , thenx quantized = round
[Equation]
clamp
[Equation]
, with the clipping function clamp( x, s ) defined as:
[Equation]
Furthermore, given the scale factor s , the de-quantization is performed as:
[Equation]
We further emphasize here that our method of representation mapping differs from this quantization method,where we directly manipulate the floating point number format as mentioned in Section 3.1.