Is Integer Arithmetic Enough for Deep Learning Training?
Abstract
The ever-increasing computational complexity of deep learning models makes their training and deployment difficult on various cloud and edge platforms.
Replacing floating-point arithmetic with low-bit integer arithmetic is a promising approach to save energy, memory footprint, and latency of deep learning models.
As such, quantization has attracted the attention of researchers in recent years.
However, using integer numbers to form a fully functional integer training pipeline including forward pass, back-propagation, and stochastic gradient descent is not studied in detail.
Our empirical and mathematical results reveal that integer arithmetic seems to be enough to train deep learning models.
Unlike recent proposals, instead of quantization, we directly switch the number representation of computations.
Our novel training method forms a fully integer training pipeline that does not change the trajectory of the loss and accuracy compared to floating-point, nor does it need any special hyper-parameter tuning, distribution adjustment, or gradient clipping.
Our experimental results show that our proposed method is effective in a wide variety of tasks such as classification (including vision transformers), object detection, and semantic segmentation.
Introduction
Recently, deep learning models have evolved to deliver acceptable performance in many areas such as computer vision, natural language processing, and speech recognition.
However, the number of parameters and computational complexity of most deep learning applications have increased significantly.
This ever-increasing computational complexity makes the deployment of deep neural networks harder on edge devices.
Furthermore, the energy required to train a deep learning model increases proportionately with computational complexity.
Thus, the training cost of these large and complex models also increases significantly on the high-end cloud servers.
Although, quantization techniques are commonly used to accelerate inference of deep learning models, here we target a fully integer training pipeline (i.e. forward propagation, back-propagation and SGD) .
Unlike recent proposals in quantized back-propagation, we directly change the number representation of floatingpoint values.
Furthermore, we show empirically and theoretically that our proposed integer training methodology is effective in training deep learning models with integer-only arithmetic .
Despite the fact that accelerated training using integer back-propagation seems intriguing, there are some challenges associated with integer gradient computation.
For example, (i) gradients must be scaled correctly in order to be adapted to the limited dynamic range of the integer number (e.g. int8 : [ − 128 , 127] ).
(ii)
The numerical error of the gradient must be unbiased in order to preserve the convergence trajectory of the training algorithm.
A small numerical error can be accumulated through the course of training and change the convergence behaviour.
(iii)
The integer training must be distribution independent , i.e. the training method should not depend on the distribution of the gradients, weights, or training data.
We propose a novel integer training method designed to address aforementioned challenges (i), (ii) and (iii) simultaneously.
Having a linear dynamic fixed-point mapping coupled with a non-linear inverse mapping is the key to full integer training.
To this end, we make the following contributions:
• A hardware-friendly integer training method is proposed based on extracting the maximum floating-point exponent of the tensors as the scale .
We propose linear fixed-point mapping of tensors, while the corresponding inverse mapping for floating-point is non-linear.
Our proposed method addresses all the previously mentioned challenges: (i) it computes the scales dynamically, and moreover the scale does not need to be adjusted in the course of training; (ii) it provides an unbiased estimation of the gradients and consequently, its convergence trajectory closely follows the floating-point version; (iii) our proposed representation mapping does not depend on the distribution of the training data, weights, or gradients.
• We study the optimality gap of our proposed integer training algorithm and show it is analogous to its floating-point counterpart in the course of training ( Theorem 1 ).
Our analysis of stochastic gradient descent with our proposed fixed-point gradients shows the original floating-point optimality gap is only shifted by a negligible amount ( Remark 3 ).
• Our proposed method effectively performs all operations required in modern neural networks using integer-only arithmetic .
For instance, the computation of linear layer, convolutional layer, batch-norm and layer-norm, residual connections and also stochastic gradient descent (including gradients, momentum, weight decay, and weight update) are all performed in integer arithmetic.
To the best of our knowledge, this is the first time that back-propagation of a batch-norm, and the computation of stochastic gradient descent (SGD) is performed in integer arithmetic with negligible loss in the accuracy for large datasets such as ImageNet.
The rest of this paper is structured as follows.
Section 2 reviews some previous works in the field of quantized back-propagation and quantifies the similarities and differences with our representation mapping method.
Section 3 discusses our integer training methodology in detail.
Theoretical aspects of our integer training method on SGD are studied in Section 4.
Experimental results supporting our integer training methodology and convergence theory are presented in Section 5.
Related works
Banner et al.
[1], proposed a bifurcated back-propagation method where the gradient of weights are in floating-point format and input gradients are in 8-bit fixed-point format.
Moreover, they proposed a range-based batch-norm which is more tolerant to quantization noise.
The main difference of our proposed algorithm with Banner et al.
[1] is that our integer training method does not demand bifurcation.
In other words, our gradients with respect to weights and inputs are represented and computed using int8 format.
Furthermore, in our method, the back-propagation of batch-norm, as well as stochastic gradient descent (SGD) algorithm, are also represented and computed using integer values.
Zhang et al.
[2] have proposed a layer-wise “precision-adaptive” quantization method that does not affect the distribution of the data.
In their method, the quantization error is measured and the quantization scale is adjusted over the course of training accordingly.
Similarly, in Zhao et al.
[3] a distribution adaptive quantization method is proposed that takes into account the distribution of gradients in the channel dimension.
In addition, they introduced a gradient clipping strategy in order to normalize the magnitude of the gradients in the back-propagation.
The scale of the quantization is adapted to the distribution of the gradients and adjusted iteratively over the course of training.
Unlike the distribution adaptive methods such as [2, 3], our proposed integer training method does not depend on the distribution of training data, weights, and gradients.
Thus, there is no need to adjust the scale iteratively in the course of training.
In Zhu et al.
[4], a “direction sensitive gradient clipping method” is proposed to avoid inappropriate updates in the back-propagation.
Moreover, they proposed a learning rate scaling that tackles the problem of unbiased gradient error accumulation.
In a similar work, Sakr and Shanbhag [5] proposed a precision assignment methodology to improve the convergence of the quantized back-propagation.
This precision assignment is based on some criterion on internal accumulator noise, quantization noise of backward and forward propagation, and gradient clipping.
Unlike [4, 5], our proposed method enjoys having the advantage of unbiased gradients , hence the convergence trajectory closely follows the floating-point counterpart.
Thus, our method does not need any specific gradient correction, gradient clipping strategy, and learning rate corrections as opposed to [4, 5].
Jin et al.
[6] proposed a unified fixed-point and parameterized clipping activation to achieve high accuracy.
Furthermore, they proposed a method that directly trains the fractional length (i.e. scale) of the fixed-point quantization.
Additionally, they use a double forward batch-norm fusion to determine the scaling factor of the batch-norm layer in the forward propagation, while the back-propagation of batch-norm remains floating-point.
In contrast, our proposed integer training method is capable of implementing integer variant of the batch-norm layer in forward propagation without double fusion as opposed to [6].
We also perform batch-norm’s back-propagation in integer arithmetic which was ignored in the previous works.
Note that back-propagation of batch-norm and layer-norm is sensitive to quantization, as such, naive quantization leads to training divergence.
Additionally, we perform stochastic gradient descent (SGD) computation including weight update, weight decay, and momentum in integer arithmetic with no significant loss of accuracy.
Wang et al. [7] explored an integer-arithmetic implementation of some preliminary CNN architectures such as
AlexNet, however they experienced considerable accuracy drop for ImageNet dataset.
Moreover, more complicated architectures such as ResNet and transformers are not considered.
In our proposed method, we used dynamic fixed-point number format [8] as the primary number format for integer training.
This specific fixed-point number format (also known as block floating-point format) has been used previously to quantize inference of deep learning models in Courbariaux et al.
[8], Drumond et al.
[9], and Rouhani et al.
Methodology
Common quantization methods, which use division and clipping techniques (refer to Appendix A.6), are inefficient for back-propagation.
Therefore, we decided to go one step deeper and change the number format directly.
Here, we propose a hardware-friendly number representation mapping using the dynamic fixed-point number format where the scale is defined per tensor .
In this approach, each tensor can be represented by its int8 version while it is multiplied by a shared scale, as opposed to other methods that allow multiple shared scales for different partitions of the tensor (see Rouhani et al.
[10, Figure 4]).
One shared scale per tensor makes the computations easier in the computing hardware (e.g. CPU or GPU).
However, the training algorithm diverges if the representation mapping is not executed properly.
To tackle this problem, we suggest two subtle changes; (i) we propose to perform fixed-point mapping of a tensors in a linear fashion while the inverse mapping is non-linear as explained in Sections 3.1 and 3.2.
This is the key to success of our algorithm since a linear fixed-point mapping allows monotonic conversion of floating-point format to fixed-point, while a non-linear inverse mapping allows preserving information.
(ii) We suggest to use stochastic rounding in the back-propagation in a way that preserves the expected value of the tensors as well as their vital statistics, such as the mean.
Stochastic rounding in conjunction with representation mapping is crucial to keep the integer training loss trajectory close to its floating-point counterpart.
3.1 Linear fixed-point mapping
Our proposed integer training method is based on manipulating the floating-point number format.
This is a very simple and effective way of converting floating-point values to fixed-point values as opposed to other commonly used methods that involve division operation.
Our method essentially uses shift and round operations to convert floating-point to fixed-point format, see Figure 1(a).
In this method, a tensor comprising n floating-point numbers ( f 1 , f 2 , ..., f n ) is converted to a fixedpoint tensor.
First, as shown in Figure 1(a), sign, exponent, and mantissa of each floating-point number are extracted using the unpack to integer function.
For instance, f 1 is unpacked to s 1 , e 1 , mwhere ( s, e, m ) is used to denote ( sign , exponent , mantissa ).
We simply find the maximum element of e 1 , e 2 , ...
e n i.e. e max = max( e 1 , e 2 , ...
e n ) to extract the shared scale of the tensor.
Subsequently, the original mantissas are shifted to the right according to the difference of their exponent and e max , to get the individual mantissas.
For instance, m 1 is shifted to right by e max − e 1 which is denoted by m 1 >> ( e max − e 1 ) .
By shifting mantissas to right, we intentionally push the small values to the sub-normal region, where the most significant bit of mantissa is not 1 in binary format i.e. (1) 2 , see
Zuras et al.
Pushing the mantissas to the sub-normal region is performed to align their exponents f f fn ...
Unpack To Integer
Float Tensor ...
e max m 1 >>e max - e 1 m 2 >>e max - em n >> e max - e n
Shifted 24-bitMantissas S n S 2 S
S n S 2 S
Stochastic Rounding ...
Signed 8-bit Mantissas
Send to Integer Layer
Singed 8-bit Mantissas 8-bitScale 24-bitMantissa 8-bitScale
S 1 e 1 m
S 2 e 2 m
S n e n m n
Signedmantissas Sharedexponent
Rounding /
Extracting sign
Sign
Exponent Mantissa
Floating point alignment (Using LZA unit)
Pack to float
Float Tensor
L aye r
C o m pu t a t i on e .
g .
L i n ea r L aye r
N on - li n ea r
I n ve r se M a
pp i ng
Add
Integer Linear Layer (GEMM)
Reconstructing individual exponents
[Figure]
with e max and this unifies the scale for fixed-point values in a tensor. In the next step, to create int mantissas, the 24-bit single floating-point mantissas (i.e. 23-bit mantissa + 1 implicit hidden bit) are further rounded to 7-bit mantissas to construct signed int8 values i.e 7-bit unsigned integer and 1 sign bit.
As an example, let us assume the shifted mantissa is (0 . 01011001010101010100000) 2 , then it is going to be randomly rounded to either (0 . 010110)
2 or (0 . 010111) 2 based on a probability (Refer to A.1 for details).
3.2 Non-linear inverse mapping
Our proposed fixed-point mapping method is performed by pushing mantissa values to the sub-normal region and rounding them stochastically.
Hence, the inverse mapping must be performed using a floating-point alignment module that normalizes the mantissas.
A normalized floating-point mantissa is required to start with the most significant bit (1) 2 .
For instance, an alignment module converts 2 127 × (0 . 0101) 2 to 2 127 − 2 × (1 . 0100) 2 .
The alignment module is a very well-known logic circuit that is available in the commodity hardware using Leading Zero Anticipator (LZA) block [12].
Coupling a linear fixed-point mapping with a non-linear inverse mapping in this way keeps the number format close to floating-point.
Moreover, combining them with stochastic rounding, results in having unbiased fixed-point variant that forces the trajectory of the integer training closely follow the floating-point counterpart.
Figure 1(b) shows the inverse mapping unit.
This unit receives a tensor of mantissas and a single value of shared exponent computed by an integer layer.
Then a tensor of exponents is reconstructed by repeating the value of the shared exponent.
The size of this tensor is equal to the mantissa tensor’s size.
Moreover, the mantissa tensor is rounded and its sign is extracted.
Note that the rounding is used here because the output of the previous layer might have some excessive mantissa bits; this normally happens in matrix multiplication and convolution operations.
Then the sign, exponent, and mantissa tensors are sent to an alignment unit that shifts the mantissa and adjust the exponent in order to normalize the floating-point number format [11].
Finally, the result is packed and carried out to the next layer.
3.3 Integer layer computations
When the fixed-point mapping of floating-point values is completed, the fixed-point values are used to perform the integer-only layer computations.
For instance, let us consider a linear layer which consists of a General Matrix Multiplication (GEMM) operation, but, the idea can be generalized to other types of layers.
Figure 2 demonstrates the layer-wise integer computations of a linear layer where the shared exponents and integer mantissas are treated separately.
As shown in the Figure 2, in order to perform an integer-only GEMM operation, we need to multiply scales (2 e max1 × 2 e max2 ) .
This multiplication performs an integer addition operation on the exponents ( e max1 + e max2 ) .
Furthermore, the integer mantissas are sent to an integer GEMM module to compute the output mantissa tensor.
[Figure]
L i n ea r F i xe d - po i n t
M a
pp i ng
L aye r
C o m pu t a t i on e .
g .
L i n ea r L aye r
[Figure]
Also note that in our implementation, when the mantissa tensor is in int8 format, multiplication is in int16 format and accumulation is in int32 format.
3. Understanding the representation mapping
Here we provide an intuition of how our proposed representation mapping approach works.
Let us denote A as a tensor and ˆ A as its fixed-point version in the way that is introduced earlier.
In addition, random variables
A i and ˆ A
i are i th element of those tensors.
Also note that A i and ˆ A i are different representations of the same real number, but one is in floating-point format and the other is in dynamic fixed-point format.
Thus, one can relate A i and ˆ A i with a random error term δ i as ˆ
A i =
A i
+ δ i .
Since in our training method we used stochastic rounding [13], E { ˆ A i } =
A i , or equivalently E { δ i } =
(see Appendix A.1).
In other words, the fixed-point value is on the average equal to the floating-point value.
Note that here we consider single precision floating-point values as a surrogate of real values.
Linear and convolutional layers: For these two types of layers, both forward and backward propagation computations are based on inner products.
Thus, it is easy to see that our proposed fixed-point inner product ˆ C ij = (cid:80)
k
ˆ A ik ˆ B kj is an unbiased estimator of the floating-point inner product
[Equation]
Residual connections: A residual connection involves element-wise addition of two tensors.
Each fixed-point element ˆ C ij =
ˆ A ij + ˆ B ij is an unbiased estimator of its floating-point version
[Equation]
Batch-norm: A batch-norm layer is defined as
[Equation]
For this type of layer, it is important to compute signal statistics such as mean ˆ µ and variance ˆ σcorrectly in integer arithmetic.
Thus, the fixed-point mean ˆ µ is also an unbiased estimator of the true mean µ
[Equation]
Then the following derivation holds for fixed-point variance ˆ σ[-
-]
[Equation]
where σ 2 is the true floating-point variance of the batch and σ 2 δ is the variance of the noise introduced by the linear mapping to fixed-point.
Note that the error variance σ 2 δ is rather small and can be integrated to ϵ in the denominator of equation (3).
Theoretical analysis of SGD using the proposed method
The generic equation of weight update in the k th iteration of SGD is
[Equation]
6) where g ( w k , ξ k ) is the estimated gradients of random samples of the batch generated by the seed ξ k , and α k is the learning rate.
We make the following common assumption in the sequel.
Assumption 1 (Lipschitz-continuity).
The loss function L ( w ) is continuously differentiable and its gradients ∇L ( w ) satisfies the following inequality where L > 0 is the Lipchitz constant
[Equation]
Assumption 2.
(i) L ( w k ) is bounded.
(ii)
Estimated gradients g ( w k , ξ k ) is an unbiased estimator of the true gradients of the loss function
[Equation]
and (iii,a) there exist scalars M ⩾ 0 and M V ⩾
such that for all iterations of SGD V
[Equation]
Note that here we define
[Equation]
Also from Assumption 2.
(ii) and (iii,a) , the second moment bound can be derived
[Equation]
Effect of gradient variance on convergence: The quality of the estimated gradients g ( w k , ξ k ) directly affects the convergence of the SGD.
The effect of first and second moments of gradient are already studied on real numbers in the literature.
Lemma 1.
Suppose Assumption 2 is true, then we have
[Equation]
Proof: See Bottou et al.
[14, Lemma 4.4].
Inequality (9) shows the effect of gradient variance bounds, M and M G , on each iterate of SGD, and shows the greater the variance, the more deterioration in the quality of SGD steps.
The first term, − (1 − 12 α k LM G ) α k ||∇L ( w k ) || 22 contributes to the decrease of the loss function while the second term, 12 α 2 k LM , prevents it.
Upon choosing the correct gradient estimates, the right hand side of inequality (9) is bounded by a deterministic quantity, and asymptotically ensures sufficient descent of the loss L ( w ) .
Note that the expectation E ξ k is taken over random samples with seed ξ k .
4.1 Fixed-point gradient noise
When performing representation mapping in the back-propagation, the quality of the gradients deteriorate.
Thus, there is a need to consider the effect of fixed-point mapping variance.
Then,
Assumption 2 (iii,a) should be modified accordingly.
Remark 1.
Note that Assumption 2 (i), (ii) still hold after fixed-point mapping because of stochastic rounding, i.e. the fixed-point gradient remains an unbiased estimator of gradient (refer to Appendix
A.1).
Assumption 2 (iii,b).
When the gradients are in fixed-point format i.e. ˆ g ( w k , ξ k ) , there exist scalars
M ⩾ 0 , M V ⩾ 0 , M q ⩾ 0 and M qV ⩾ 0 such that for all iterations of SGD
[Equation]
If ˜ M := M + M q and ˜ M V :=
M V + M qV , then Assumption 2 (iii,b) takes the exact form of
Assumption 2 (iii,a) i.e. V ξ k { g ( w k , ξ k ) } ⩽ ˜ M + ˜ M V ||∇L ( w k ) || 22 .
However, here we separated M q and M qV to emphasize the effect of fixed-point mapping on the true gradients.
Remark 2.
If Assumption 2 (iii,b) holds true, inequality (9) can be transformed to its fixed-point version
[Equation]
Inequality (10) shows the effect of added representation mapping variance with bounds, M q and M qG , on each iterate of SGD.
This observation shows that fixed-point mapping degrades the convergence of SGD unless its variance bounds are relatively small, or controlled by the learning rate.
As an example, refer to Appendix A.2 for analytical derivations of M q and M qG for the back-propagation of a linear layer involving a fixed-point inner product.
4.2 Strongly convex and locally convex loss
Assumption 3 (Strong convexity).
The loss function L ( w ) is differentiable and strongly convex.
We recall that strong convexity for differentiable functions is equivalent to the following inequality with some constant c >[-
-]
[Equation]
A strongly convex function has a unique minimum point at w ∗ with the loss value L ∗ =
L ( w ∗ ) .
Theorem 1.
Suppose Assumptions 1, 2(i), 2(ii), 2 (iii,b), 3 are all true, then a SGD method running with fixed-point gradients i.e. ˆ g ( w k , ξ k ) and a fixed learning rate [-
-]0 < ¯ α ⩽ 1 L ( M G + M qG ) satisfies the following bound for its optimality gap with the minimum loss L ∗ at the k th iteration
[Equation]
Proof.
See Appendix A.3.
Remark 3.
Note that when k −→ ∞ , ¯ αLM 2 c is the original optimality gap (i.e. E {L ( w k ) − L ∗ } ) with real gradients, see Bottou et al.
[14, Theorem 4.6], and then, the optimality gap is also increased by ¯ αLM q 2 c due to fixed-point representation.
Here we argue that by keeping the variance bound M q relatively small via choosing the correct number format, we can theoretically achieve the original performance.
On the other hand, optimality gap is related to ¯ α , which means smaller learning rates leads to smaller optimality gap.
Remark 4 (Local convexity).
Strongly convex loss is not a realistic assumption in large deep learning models.
However, local convexity is a more realistic assumption i.e. L is convex around the minimum point w ∗ .
Hence, inequality (12) still holds around the minimum point w ∗ of a locally convex loss.
We essentially train our ImageNet ResNet18 classification in the same way: first we train the network with a fixed learning rate until reaching a certain optimality gap; then the learning rate is reduced to a smaller fixed value in order to shrink the optimality gap.
Empirical evidence: Figure 3(a) demonstrates the locally convex loss landscape of ResNet18 training on CIFAR10 dataset using floating-point computation.
Figure 3(b) shows the same loss landscape in the fixed-point int8 format.
We perturbed the weights around the w ∗ in x and y axes using Gaussian noise and evaluated the loss L in the z axis to acquire Figures 3(a) and 3(b).
Comparing these two figures pronounces our assumption of local convexity in both integer and floating-point tests.
Figure 3(c) shows a comparison of the loss trajectory for floating-point and integer training.
The fixed-point gradients are unbiased estimators of the true gradients, so the trajectory of the integer training closely follows the trajectory of its floating-point counterpart.
Remark 5.
We implemented an integer weight update, hence, the computations of equation (6) is also performed in integer arithmetic.
It is shown in Appendix A.4 that integer weight update with stochastic rounding is an unbiased estimator of the true weight update.
[Figure]
Experimental results
Image classification: Table 1 reports the experimental results of our proposed integer training method on the conventional vision classification models.
In this set of models, we used int8 linear layer, int8 convolutional layer, int8 batch-norm layer, and int16 SGD to form a fully integer training pipeline.
Note that all forward and back-propagation computations of layers are performed in integer format.
Moreover, we have chosen smaller vision models such as ResNet18 and MobileNetV knowing that smaller models are harder to train.
Our experimental results show that our proposed integer training method is as good as floating-point with negligible loss of accuracy ( ⩽ 0 .
5% on
ImageNet).
Remark 6.
We compare our integer training results with official Pytorch floating-point training.
For instance, in conventional vision models (Table 1), we compare the results directly with the accuracy of torchvision models that are reported on the official Pytorch website 1 .
We further emphasize that there is no change in hyper-parameters of the training.
The full set of hyper-parameters of our experiments are reported in Appendix A.5.
Vision transformer:
We validated the applicability of our proposed integer training on the original vision transformer model, notably ViT-B-16-224 [15].
We took the floating point checkpoint pretrained on ImageNet21K 2 .
We used Huggingface [16] to fine-tune the model on CIFAR10.
In this experiment, we used int8 linear layer, int8 matrix multiplication, int8 convolutional layer, and int8 layer-norm for our integer training pipeline.
The result is reported in Table 1 demonstrating negligible loss of accuracy ( ⩽ 0 .
Note that for this experiment, the computation of softmax in attention mechanism is in floating point.
Semantic segmentation: To validate our proposed integer training on semantic segmentation task, we used DeepLabV1/V2 3 with ResNet-101 as the backbone.
We trained the models on PASCAL
VOC-2012 [17] and MS COCO 10K [18].
For the PASCAL dataset, as suggested by Chen et al. [19], the models were initialized with MS COCO checkpoint and data augmentation was used during training.
Likewise, for COCO dataset we used PASCAL checkpoint for initialization.
The batch-norm layers are frozen in our experiments as suggested in [19].
We used linear and convolutional layers in int8 format.
Table 2 shows that the mean intersect of union (mIOU) of our method closely matches the floating-point baseline.
Object detection: We used Faster R-CNN[20] with ResNet50 as its backbone and Single Shot
Detector (SSD)[21] which relies on VGG-16 for its backbone.
In these experiments, we used int linear and convolutional layers, while batch-norm layers are frozen.
We integrated our integer training method with MMDetection[22] toolbox and performed our experiments on MS COCO 10K[18],
PASCAL VOC-2007[23], VOC-2012[17], and Cityscapes[24] datasets.
As shown in Table 3, the mean average precision (mAP) of our proposed integer training method is very close to floating-point baseline. Refer : https: Refer to : https:
[Table] [Table]
[Table] [Table]
[Table]
with state of the art: Table 4 provides a comparison between our training method and state of the art across different experiments.
There are some important differences between our proposed method and other works: (i) our proposed integer training method uses a fixed-point batchnorm layer where both forward and back propagation is computed using integer arithmetic, (ii) our proposed training method uses an integer-only SGD, (iii) in our training method, no hyper-parameter is changed while other methods have changed hyper-parameters or used gradient clipping.
Low-bit integer training: Table 5 provides an ablation study of how lowering integer bit-width can affect the training accuracy.
Our experiments shows that training has a significant drop of accuracy with int5 and diverges using int4 number formats.
Conclusion
We proposed a hardware-friendly integer training method based on coupling a linear fixed-point mapping with a non-linear inverse mapping.
This method performs the representation mapping directly on the floating-point number format.
Furthermore, our method uses stochastic rounding and produces unbiased estimators of gradients in the back-propagation.
Thus, benefiting from unbiased gradients and effective representation mapping, the training loss trajectory closely follows its floatingpoint version.
Moreover, there is no need to tune hyper-parameters or perform gradient clipping to correct the inappropriate gradient and weight updates.
Using our proposed technique, we designed the integer version of the most vital components of deep learning such as linear layer, convolutional layer, batch-norm, layer-norm, and stochastic gradient descent (SGD).
Our experimental results show the effectiveness of the proposed method, where the accuracy loss is negligible in a wide variety of the training tasks.
Furthermore, in our classification tests all training components are in integer arithmetic.
We theoretically studied the effect of our proposed method on SGD and demonstrated why the optimality gap of convergence is shifted by a negligible amount.
Acknowledgments and Disclosure of Funding
The authors would like to thank Richard Wu and Vanessa Courville for their constructive comments.
The authors would like to thank Sara Elkerdawy for initiating the object detection and semantic segmentation experiments during her internship at Huawei.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Theorem 1 and Remark 3 in Section 4.2 reflecting the effect of representation mapping noise.
Moreover, the accuracy deviation from baseline floating-point training experiments are reported in Table 1, Table 2, and Table(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The work is purely methodological and related to number format.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.1, Appendix A.2, Appendix A.3, and Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code is proprietary, however we will provide them upon publication.
We ran the experiments on publicly available datasets.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] .
It was too costly to train multiple times, we just ran the experiments once.
Also note the type of training experiments we reported are resilient to random initialization changes.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A.1 Stochastic Rounding
In this paper, stochastic rounding for a variable x where x 1 < x < x 2 is defined as
[Equation]
13) it is easy to see that ˆ x is an unbiased estimator of x :
[Equation]
(14) and if we relate x and ˆ x using an error term δ (i.e. ˆ x = x + δ ), then E { δ } = 0 .
[Figure]
A realization of the stochastic rounding is shown in Figure 4.
Here, a 24-bit single floating-point mantissa (including implicit hidden bit) is rounded stochastically to a 7-bit value.
In this figure, the direction of rounding is determined by comparing a random number that is generated on-the-fly with the lower 17-bit of the mantissa.
A.2 Representation mapping increases the gradients variance: Linear layer example
A linear layer is essentially a matrix multiplication.
Let us denote X as inputs, W as weights and Y as the output of a linear layer where Y = XW .
Following the notation of this paper, we also denote our fixed-point version of this layer as ˆ Y =
ˆ X ˆ W .
In our proposed integer back-propagation [Equation] the layer receives the upstream gradient ˆ G :
=
∂ ˆ L∂ ˆ Y .
Using the chain-rule, the gradient with respect to weights is
[Equation]
Note that in inequality (16), σ 2 G = max( V { δ Gi,j } ) knowing that error terms δ Gi,j have essentially similar distributions.
Likewise, σ 2 X = max( V { δ Xi,j } ) .
Also note that X and G are matrices of random variables in the back-propagation and || X ⊤ i,
|| 22 denotes the norm-2 of the i th row of X ⊤ and || G ,j || 22 denotes the norm-2 of the j th column of G .
Also, ˆ X ⊤ ik denotes ik th element of the matrix ˆ X ⊤ .
By having the following definitions
[Equation] [Equation]
we can re-organize the inequality (16) as
[Equation]
Remark.
Inequality (18) supports our Assumption 2 (iii,b) i.e.
[Equation]
for considering the effect of our representation mapping method on gradients variance.
A.3 Proof of Theorem
The proof goes along the proof of Bottou et al.
[14, Theorem 4.6] in the case that the gradient variance bound increased as stated in Assumption 2 (iii, b) .
A convex function satisfying the inequality (11) given ¯ w, w ∈ R d represents a quadratic model
[Equation]
and has a unique minimizer at w ∗
[Equation]
Also remember that the fixed learning rate in our integer back-propagation has the following constraint
[Equation]
Starting from inequality (10) and using inequalities (20) and (21) we can write
[Equation]
[Equation]
By subtracting L ∗ , rearrange, and taking total expectation from both sides of inequality (22) we have
[Equation]
Then by subtracting ¯ αL ( M + M q ) 2 c from both sides E
[Equation]
Thus, Theorem 1 can be proven by applying inequality (24) repeatedly for k ∈ N .
Also note that using inequality (21) it is easy to derive that 0 < (1 − ¯ αc ) < 1 because
[Equation]
hence, in Theorem 1 , if k −→ ∞ , then (1 − ¯ αc ) k −→ 0 and the optimality gap of integer training algorithm using inequality (12) is
[Equation]
A.4 Integer weight update
In the integer weight update, the equation (6) transforms to its fixed-point version with integer-only arithmetic
[Equation]
By expanding the error terms for the representation mapping and taking expatiation on both sides we can see the weights are on the average updated equivalent to the true wights.
[Equation]
[Equation]
We used stochastic rounding for our weight update operation, thus E { δ } = 0 and using our proposed method, ˆ g is unbiased estimator of g .
A.5 Experimental setup
Computing resources: We ran our experiments using an in-house developed integer emulator to avoid common floating-point quantization techniques.
In our emulator, we perform the representation mapping within the GPU memory.
We further developed the integer deep learning modules using Pytorch autograd functionality on top of our integer emulator.
Experimental results of this paper are run using the following number of GPUs.
• ResNet18 on ImageNet requires 4 × V100 GPUs when batch size is 512 and 2 × V100 GPUs when batch size is 256.
• MobileNetV2 on ImageNet requires 8 × V100 GPUs when batch size is 512. • ResNet18 on CIFAR10 runs on 1 × V100 GPUs when batch size is 128.
Semantic segmentation experiments require 2 × V100 GPUs.
• Object detection experiments require 8 × V100 GPUs.
• ViT-B fine-tuning experiment requires 8 × V100 GPUs.
Classification: The hyper-parameters of our classification experiments are reported in Table 6.
We used SGD with momentum of 0.9 for conventional classification experiments and AdamW for ViT fine-tuning.
[Table]
Semantic segmentation:
The hyper-parameters for the semantic segmentation experiments are provided in Table7.
For DeepLabV1 and DeepLabV2 one-scale and multi-scale loss was used respectively.
Conditional random field post-processing was used at the network’s output as proposed in Chen et al.
In all the experiments we use SGD with momentum of 0.9 and weight decay of 5 × 10 − 4 .
[Table]
Object detection: The hyper-parameters are provided in Table 8.
All experiments use an SGD optimizer with a momentum of 0.9 and a weight decay parameter of 10 − 5 .
The experiments indicated with LR Warmup use a linear warm-up function with a ratio of 10 − 3 for the first 500 iterations.
For the rest of training, the learning rate is reduced by 0.1 at the epochs indicated in the LR Reduction Epochs column.
[Table]
A.6 Background on quantization methods
Symmetric uniform quantization is normally used in the literature of quantized back-propagation (see [2, 4, 3] ) to convert the gradients to 8-bit integers.
Suppose scale s = max( | x | ) , then
[Equation]
with the clipping function clamp( x, s ) defined as:
[Equation]
Furthermore, given the scale factor s , the de-quantization is performed as:
[Equation]
We further emphasize here that our method of representation mapping differs from this quantization method, where we directly manipulate the floating point number format as mentioned in Section 3.1.