L OGARITHMIC U NBIASED Q UANTIZATION : P RACTICAL 4- BIT T RAINING IN D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training.
Current methods enable 4-bit quantization of the forward phase.
However, this constitutes only a third of the training process.
Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers.
In this work, we examine the importance of having unbiased quantization in quantized neural network training, where to maintain it, and how.
Based on this, we suggest a logarithmic unbiased quantization (LUQ) method to quantize both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training.
For example, in ResNet50 on ImageNet, we achieved a degradation of 1.18%; we further improve this to degradation of only 0.64% after a single epoch of high precision ﬁne-tuning combined with a variance reduction method.
Finally, we suggest a method that exploits the low precision format by avoiding multiplications during two-thirds of the training process, thus reducing by 5x the area used by the multiplier.
A reference implementation is supplied in the supplementary material.
I NTRODUCTION
Deep neural networks (DNNs) are a powerful tool that has shown superior performance in various tasks spanning computer vision, natural language processing, autonomous cars, and more.
Unfortunately, their vast demand for computational resources, especially during the training process, is one of the main bottlenecks in the evolution of these models.
Training of DNNs consists of three main general-matrix-multiply (GEMM) phases: the forward phase, backward phase, and update phase.
Quantization has become one of the main methods to compress
DNNs and reduce the GEMM computational resources.
There has been a signiﬁcant advance in the quantization of the forward phase.
There, it is possible to quantize the weights and activations tobits while preserving model accuracy (Banner et al., 2019; Nahshan et al., 2019; Bhalgat et al., 2020;
Choi et al., 2018b).
Despite these advances, they only apply to a third of the training process while the backward phase and update phase are still computed with higher precision.
Recently, Sun et al.
(2020) was able, for the ﬁrst time, to train a DNN while reducing the numerical precision of most of its parts to 4 bits with some degradation (e.g., 2.49% error in ResNet50).
To do so, Sun et al.
(2020) suggested a non-standard radix-4 ﬂoating-point format, combined with double quantization of the neural gradients (called two-phase rounding).
This was an impressive step forward in the ability to quantize all the training processes.
However, since a radix-4 format is not aligned with conventional radix-2, any numerical conversion between the two requires an explicit multiplication to modify both the exponent and mantissa and may require additional hardware (Kupriianova et al., 2013).
Thus, their non-standard quantization requires speciﬁc hardware support that can reduce the beneﬁt of quantization to low bits.
The main challenge in reducing the numerical precision of the entire training process is quantizing the neural gradients, i.e. the backpropagated error.
Speciﬁcally, Chmiel et al. (2021) showed the neural gradients have a heavy tailed near-lognormal distribution, and therefore they should be logarithmically quantized at low precision levels.
For example, for FP4 the optimal format was [sign,exponent,mantissa] =
[1,3,0], i.e. without mantissa bits.
In contrast, that weights and activations
, and therefore are better approximated using uniform quantization (e.g., INT4).
In this work, in order to reduce the computational resources bottleneck, we dive deeper into understanding neural gradients’ quantization.
Based on the ﬁndings of Chmiel et al.
(2021), we focus on the [1,3,0] FP4 format for the neural gradients.
We analyze different types of quantization stochastic rounding schemes to explain the importance of having an unbiased rounding scheme for the neural gradients with such a logarithmic quantization format.
We compare it with forward phase quantization, where the bias is not a critical property.
Building on this analysis, we suggest a method called logarithmic unbiased quantization (LUQ) for unbiased quantization of the neural gradients to the standard FP4 format of [1,3,0].
This, together with quantization of the forward phase to INT4, achieves state-of-the-art results in full training inbits (e.g., 1.18% error in ResNet50), with no overhead.
Moreover, we suggest two additional methods to further reduce the degradation, with some overhead: the ﬁrst method reduces the quantization variance of the neural gradients, while the second is a simple method of ﬁne-tuning in high precision.
Combining LUQ with these two proposed methods we achieve, for the ﬁrst time, only 0.64% error in 4-bit training of ResNet50.
The overhead of our additional methods is no more than the previously presented in Sun et al.
Finally, we exploit the speciﬁc FP4 format mantissa used for the neural gradients ([1,3,0]) and suggest replacing the multiplication blocks with a proposed multiplication free backpropagation (MF-BPROP) block.
This way we completely avoid multiplication in the backward phase and the update phase (which constitute two thirds of the training) and reduce by 5x the area which was previously used for the multiplications in these phases.
The main contributions of this paper:
• A comparison of different rounding schemes.
• We suggest a simple and hardware friendly logarithmically unbiased quantization for the neural gradients called LUQ.
• We demonstrate that two simple methods can further improve the accuracy in 4-bit training: (1) variance reduction using re-sampling and (2) high precision ﬁne-tuning for one epoch.
• We design a modern hardware block that exploits LUQ quantization to avoid multiplication in two thirds of the training process, thus reducing by 5x the multiplier logical area.
R OUNDING SCHEMES C OMPARISON
In this section, we study the effects of unbiased rounding for the forward and backward passes.
We show that rounding-to-nearest (RDN) should be applied for the forward phase while stochastic rounding (SR) is more suitable for the backward phase.
Speciﬁcally, We show that although SR is unbiased, it generically has worse mean-square-error compared to RDN.
Given that we want to quantize x in a bin with a lower limit l ( x ) and an upper limit u ( x ) , stochastic rounding can be stated as follows:
[Equation]
The expected rounding value is given by
[Equation]
where here and below the expectation is over the randomness of SR (i.e., x is a deterministic constant).
Therefore, stochastic rounding is an unbiased approximation of x , since it has zero bias:
[Equation]
However, stochastic rounding has variance, given by
[Equation]
where the last transition follows from substituting the terms E [SR( x )]) , and p ( x ) into Eq.
We turn to consider the round-to-nearest method (RDN).
The bias of RDN is given by
Bias[RDN(x)]
= min ( x −
[Equation] [Equation]
Since RDN is a deterministic method, it is evident that the variance is 0 i.e.,
[Equation]
Finally for every value x and a rounding method R ( x ) , the mean-square-error (MSE) can be written as the sum of the rounding variance and the squared rounding bias,
[Equation]
Therefore, we have the following MSE distortion when using round-to-nearest and stochastic rounding:
[Equation]
Note that since min( a, b ) 2 ≤ a · b for every a, b , we have that
[Equation]
In Fig.
1a we plot the mean-square-error for x ∈ [0 , 1] , l ( x ) = 0 , and u ( x )
= 1 .
While round-tonearest has a lower MSE than stochastic rounding, the former is a biased estimator.
[Figure]
In (a) we present the MSE of a distributed tensor
Quantization to 4 bits of the forward phase (b) and backward phase (c) of ResNet18 - Cifar dataset with SR and RDN.
Notice that while MSE is important in the forward phase, unbiasness achieved with SR is crucial for the backward phase.
The bwd and fwd in (b) and (c) respectively, are in full precision to focus on the effect of the rounding scheme only in one pass of the network in each experiment.
2. B ACKGROUND : UNBIASED GRADIENT ESTIMATES
To prove convergence, textbook analyses of SGD typically assume the expectation of the (mini-batch) weight gradients is sufﬁciently close to the true (full-batch) gradient (e.g., assumption 4.3 in (Bottou et al., 2018)).
This assumption is satisﬁed when the weight gradients are unbiased.
Next, we show that this condition is met when the neural gradients are quantized without bias.
Denote W l as the weights between layer l − 1 and l , C the cost function, and f l as the activation function at layer l .
Given an input–output pair ( x, y ) , the loss is:
[Equation]
Let z l be the weighted input (pre-activation) of layer l and denote the output (activation) of layer l by a l .
The derivative of the loss in terms of the inputs is given by the chain rule:
[Equation]
In its quantized version, Eq.
(11) gets the following form.
[Equation]
Assuming Q ( x ) is an unbiased stochastic quantizer with E [ Q ( x )]
= x , the quantized backpropogation δ lq is an unbiased approximation of backpropogation δ l
[Equation]
In Eq.
(13) we used the linearity of back-propagation to express the expected product as a product of expectations.
Finally, since the gradient of the weights in layer l is ∇ W l C = δ l · a l − 1 and in its quantized form it becomes ∇ W l C q = δ lq · a l − 1 , the update ∇ W l C q is an unbiased estimator of ∇
W l C :
[Equation]
The forward pass is different from the backward pass in that unbiasedness at the tensor level is not necessarily a guarantee of unbiasedness at the model level since the activation functions and loss function are not linear.
Therefore, even after stochastic quantization, the forward phase remains biased.
Conclusions.
It was previously proved that unbiased neural gradients quantization leads to an unbiased estimate of the weight gradients (e.g., Chen et al.
(2020a)), which enables proper convergence of SGD (Bottou et al., 2018).
Thus, bias in the gradients can hurt the performance and should be avoided, even at the cost of increasing the MSE.
Therefore, neural gradients, following should be quantized using SR, following subsection 2.1.
However, the forward phase should be quantized deterministically (using RDN) since stochastic rounding will not make the loss estimate unbiased (due to the non-linearity of the loss and activation functions) while unnecessarily increasing the MSE (as shown in Eq.
There are cases where adding limited noise, such as dropout, increases MSE but improves generalization.
However, this is typically not the case, especially if the noise is large.
Figs.
1b and 1c show that these theoretical observations are consistent with empirical observations favoring RDN for the forward pass and SR for the backward pass.
LUQ - A LOGARITHMIC UNBIASED QUANTIZER
A recent work (Chmiel et al., 2021) showed that the neural gradients can be approximated with the lognormal distribution.
This distribution has many values concentrated around the mean but is also heavy-tailed, making the extreme values orders of magnitudes larger than the small values sampled from this distribution.
They exploit this fact and showed that the neural gradients can be pruned to a high pruning ratio without accuracy degradation (e.g., 85% in ResNet18 ImageNet dataset), using an unbiased pruning method.
We build on top of this pruning method, and combine it with an unbiased logarithmic quantizer, as described below.
Unbiased stochastic pruning Given an underﬂow threshold α we deﬁne a stochastic pruning operator, which prunes a given value x , as
[Equation]
Unbiased FP quantizer Given an underﬂow threshold α , let Q α ( X ) be a FP round-to-nearest b -bits quantizer with bins { α, 2 α, ..., 2 b − 1 α } .
Assume, without loss of generality, 2 n − 1 α < x < 2 n α ( n ∈ { 0 , 1 ..., b − 1 } ) .
We will use the following unbiased quantizer, which is a special case of SR (Eq.
[Equation]
It is unbiased since
[Equation]
and as a special case of Eq.
Underﬂow threshold In order to create an unbiased quantizer, the largest quantization value 2 b − 1 α should avoid clipping any values of x , otherwise this will create a bias.
Therefore, the underﬂow threshold α is chosen as the optimal unbiased value, i.e
[Equation]
where b = 3 for FP4.
Logarithmic rounding
Traditionally, stochastic rounding as in eq.
16 is implemented by adding a uniform random noise (cid:15) ∼ U [ − 2 n − 1 α
[Equation]
n [Equation] 1 α] to x and then use a round-to-nearest operation.
In order to implement round-to-nearest directly on the exponent, we need to correct an inherent bias since α · 2 (cid:98) log ( | x | α ) (cid:101) (cid:54) =
α ·
[Equation]
log
[Equation]
α
[Equation]
For a bin [2 n − 1 , 2 n ] , the midpoint x m is
[Equation] [Equation]
Therefore, we can apply round-to-nearest-power (RDNP) directly on the exponent x of any value 2 n − 1 ≤ 2 x ≤ 2 n as follows:
[Equation]
Logarithmic unbiased quantization (LUQ)
In the following, we suggest LUQ, a 4-bit unbiased estimation of neural gradients that apply stochastic pruning (Eq.
(15)) to the 4-bit ﬂoating point
quantizer Q α ( x )
[Equation]
Since T α and Q α are unbiased, it follows by the law of total expectation that X q is an unbiased estimator for x :
[Equation]
where the expectation is over the randomness of T α and Q α .
In Fig.
2 we show an illustration of LUQ.
The ﬁrst step includes stochastic pruning for the values below the pruning threshold ( | x | < α ).
The second step includes the logarithmic quantization with format FP4 of the values above the pruning threshold ( | x | > α ).
In Fig.
3a we show an ablation study of the effect of the different parts of LUQ on ResNet50 ImageNet dataset - while standard FP4 diverges, adding stochastic-pruning or round-tonearest-power allow to converge with signiﬁcant degradation.
Combining both methods improves the results and ﬁnally the suggested LUQ which includes additionally the suggested underﬂow threshold choice as the optimal unbiased value gets the best results. [Figure]
3. SMP: R EDUCING THE VARIANCE WHILE KEEPING IT UNBIASED
In the previous section, we presented an unbiased method for logarithmic quantization of the neural gradients called LUQ.
Following the bias-variance decomposition, if the gradients are now unbiased, then the only remaining issue should be their variance.
Therefore, we suggest a method to reduce the quantization variance by repeatedly sampling from the stochastic quantizers in LUQ, and averaging the resulting samples of the ﬁnal weight gradients.
The different samples can be calculated in parallel, so the only overhead on the network throughput will be the averaging operation.
The power overhead will be ∼ 13 of the number of additional samples since it affects only in the update GEMM (Eq.
For N different samples, the proposed method will reduce the variance by a factor of 1 N , without affecting the bias (Gilli et al., 2019).
In Fig.
3b we show the effect of the different number of samples (SMP) on 2-bit quantization of ResNet18 Cifar100 dataset.
There, we achieve with 16 samples accuracy similar to a full-precision network.
This demonstrates that the variance is the only remaining issue in neural gradient quantization using LUQ, and that the proposed averaging method can erase this variance gap, with some overhead.
[Figure]
3. FNT: F INE - TUNING IN HIGH PRECISION FOR ONE EPOCH
We suggest running one additional epoch in which we increase all the network parts to full-precision, except the weights which remain in low precision.
We notice that with this scheme we get the best accuracy for the ﬁne-tuned network.
In inference time the activations and weights are quantized to lower precision.
In Table 1 we can see the effect of the proposed ﬁne-tuning scheme, improving the accuracy of the models by ∼ 0 .
E XPERIMENTS
In this section, we evaluate the proposed LUQ for 4-bit training on various DNN models.
For all models, we use their default architecture, hyper-parameters, and optimizers combined with a custom-modiﬁed Pytorch framework that implemented all the low precision schemes.
Additional experimental details appear in Appendix A.1.
Main results In Table 1 we show the top-1 accuracy achieved in 4-bit training using LUQ to quantizing the neural gradients to FP4 and combined with a previously suggested method, SAWB (Choi et al., 2018a), to quantize the weights and activations to INT4.
We compare our method with
Ultra-low (Sun et al., 2020) showing better results in all the models, achieving SOTA in 4-bit training.
Moreover, we improve the results with the two proposed schemes: neural gradients sampling (SMPSection 3.1) and ﬁne-tune in high precision (FNT - Section 3.2) achieving for the ﬁrst time in 4-bit training 0.64% error in ResNet-50 ImageNet dataset, with our simple and hardware friendly methods.
In Table 2 we apply the proposed LUQ on NLP models, achieving less than 0 .
4% BLUE score degradation in Transformer-base model on the WMT En-De task.
Overhead of SMP and FNT We limit our experiments with the proposed SMP method to only two samples.
This is to achieve a similar computational overhead as Ultra-low, with their suggested two-phase-rounding (TPR) which also generates a duplication for the neural gradient quantization.
The FNT method is limited to only 1 epoch to reduce the overhead in comparison to Ultra-low, which keeps the 1x1 convolutions in 8-bit.
Speciﬁcally, the throughput of a 4-bit training network is 16x in comparison to high precision training (Sun et al., 2020).
This means that doing one additional epoch in high precision reduces the throughput of ResNet-50 training by ∼ 16% .
In comparison,
Ultra-low (Sun et al., 2020) does full-training with all the 1x1 convolutions in 8-bits, which reduces the throughput by
[Table] [Table]
Forward-backward ablations In Table
we show the top-1 accuracy in ResNet50 with different quantization schemes.
The forward phase (activations + weights) is quantized to INT4 with SAWB (Choi et al., 2018a) and the backward phase (neural gradients) to FP4 with LUQ.
As expected, the network is more sensitive to the quantization of the backward phase
[Table]
MF-BPROP: MULTIPLICATION FREE BACKPROPAGATION
The main problem of using different datatypes for the forward and backward phases is the need to cast them to a common data type before the multiplication during the backward and update phases.
In our case, the weights ( W ) and pre-activations ( a ) are quantized to INT4 while the neural gradients ( ∂C∂a ) to FP4, where C represent the loss function, φ a non-linear function and v the post-activation.
During the backward and update phases, in each layer l there are two GEMMs between different datatypes:
[Equation]
Regularly, to calculate these GEMMs there is a need to cast both data types to a common data type (in our case, FP7 [1,4,2]), then do the GEMM and ﬁnally, the results are usually accumulated in a wide accumulator (Fig. 4a).
This casting cost is not negligible.
For example, casting INT4 to FP consumes ∼ 15% of the area of an FP7 multiplier.
In our case, we are dealing with a special case where we do a GEMM between a number without mantissa (neural gradient) and a number without exponent (weights and activations), since INT4 is almost equivalent to FP4 with format [1,0,3].
We suggest transforming the standard GEMM block (Fig. 4a) to Multiplication Free BackPROP (MF-BPROP) block which contains only a transformation to standard FP7 format (see Fig. 4b) and a simple XOR operation.
More details on this transformation appear in Appendix A.3.
In our analysis (Appendix A.4) we show the MF-BPROP block reduces the area of the standard GEMM block by 5x.
Since the FP32 accumulator is still the most expensive block when training with a few bits, we reduce the total area in our experiments by ∼ 8% .
However, as previously showed (Wang et al., 2018) 16-bits accumulators work well with 8-bit training, so it is reasonable to think, it should work also with 4-bit training.
In this case, the analysis (Appendix A.4) shows that the suggested MF-BPROP block reduces the total area by ∼ 22% .
[Figure]
R ELATED WORKS
Neural networks Quantization has been extensively investigated in the last few years.
Most of the quantization research has focused on reducing the numerical precision of the weights and activations for inference (e.g., Courbariaux et al.
; Rastegari et al.
; Banner et al.
; Nahshan et al.
; Choi et al.
(2018b)
; Bhalgat et al.
; Choi et al.
(2018a)
; Liang et al.
In standard ImageNet models, the best performing methods can achieve quantization to 4 bits with small or no degradation Choi et al.
(2018a).
These methods can be used to reduce the computational resources in approximately a third of the training.
However, without quantizing the neural gradients, we cannot reduce the computational resources in the remaining two thirds of the training process.
An orthogonal approach of quantization is low precision for the gradients of the weights in distributed training (Alistarh et al., 2016;
Bernstein et al., 2018) in order to reduce the bandwidth and not the training computational resources.
Sakr & Shanbhag (2019) suggest a systematic approach to design a full training using ﬁxed point quantization which includes mixed-precision quantization.
Banner et al.
ﬁrst showed that it is possible to use INT8 quantization for the weights, activations, and neural gradients, thus reducing the computational footprint of most parts of the training process.
Concurrently, Wang et al.
(2018) was the ﬁrst work to achieve full training in FP8 format.
Additionally, they suggested a method to reduce the accumulator precision from 32bit to 16 bits, by using chunk-based accumulation and ﬂoating point stochastic rounding.
Later, Wiedemann et al.
(2020) showed full training in INT8 with improved convergence, by applying a stochastic quantization scheme to the neural gradients called non-subtractive-dithering (NSD) that induce sparsity followed by stochastic quantization.
Also, Sun et al.
(2019) presented a novel hybrid format for full training in FP8, while the weights and activations are quantized to [1,4,3] format, the neural gradients are quantized to [1,5,2] format to catch a wider dynamic range.
Fournarakis & Nagel (2021) suggest a method to reduce the data trafﬁc during the calculation of the quantization range.
While it appears that it is possible to quantize to 8-bits all computational elements in the training process, 4-bits quantization of the neural gradients is still challenging.
Chmiel et al.
(2021) suggested that this difﬁculty stems from the heavy-tailed distribution of the neural gradients, which can be approximated with a lognormal distribution.
This distribution is more challenging to quantize in comparison to the normal distribution which is usually used to approximate the weights or activations (Banner et al., 2019).
Sun et al.
(2020) was the ﬁrst work that presented a method to reduce the numerical precision to 4-bits for the vast majority of the computations needed during DNNs training.
They use known methods to quantize the forward phase to INT4 (SAWB (Choi et al., 2018a) for the weights and PACT (Choi et al., 2018b) for the activations) and suggested to quantize the neural gradients twice (one for the update and another for the next layer neural gradient) with a non-standard radix-4 FP4 format.
The use of the radix-4, instead of the commonly used radix-2 format, allows covering a wider dynamic range.
The main problem of their method is the speciﬁc hardware support for their suggested radixdatatype, which may limit the practicality of implementing their suggested data type.
Chen et al.
(2020b) suggested reducing the variance in neural gradients quantization by dividing them into several blocks and quantizing each to INT4 separately.
Their method requires each iteration to sort all the neural gradients and divide them into blocks, a costly operation that will affect the network throughput.
Additionally, they suggested another method to quantize each sample separately.
The multiple scales per layer in both methods do not allow the use of an efﬁcient GEMM operation.
C ONCLUSIONS
In this work, we analyze the difference between two rounding schemes: round-to-nearest and stochastic-rounding.
We showed that, while the former has lower MSE and works better for the quantization of the forward phase (weights and activations), the latter is an unbiased approximation of the original data and works better for the quantization of the backward phase (speciﬁcally, the neural gradients).
Based on these conclusions, we propose a logarithmic unbiased quantizer (LUQ) to quantize the neural gradients to format FP4 [1,3,0].
Combined with a known method for quantizing the weights and activations to INT4 we achieved, without overhead, state-of-the-art in 4-bit training in all the models we examined, e.g., 1.18 % error in ResNet50 vs 2.49 % for the previous known SOTA (Sun et al.
Moreover, we suggest two more methods to improve the results, with overhead comparable to Sun et al.
The ﬁrst reduces the quantization variance, without affecting the unbiasedness of LUQ, by averaging several samples of stochastic neural gradients quantization.
The second is a simple method for ﬁne-tuning in high precision for one epoch.
Combining all these methods, we were able for the ﬁrst time to achieve 0.64 % error in 4-bit training of ResNet
ImageNet dataset.
Finally, we exploit the special formats used for quantization (INT4 in the forward phase and FP4 format [1,3,0] in the backward phase) and suggest a block called MF-BPROP that avoids multiplication during two thirds of the training, thus reducing by 5x the area previously used by the multiplier.
R EPRODUCIBILITY
A source code of the experiments appears in the supplementary material.
Additionally, in Appendix A.1 we added the details of the experiments, including the hyper-parameters used.
A A PPENDIX A. E XPERIMENTS DETAILS
In all our experiments we use the most common approach (Banner et al., 2018; Choi et al., 2018b) for quantization where a high precision of the weights are kept and quantized on-the-ﬂy.
The updates are done in full precision.
ResNet / ResNext We run the models ResNet-18, ResNet-50 and ResNext-50 from torchvision.
We use the standard pre-processing of ImageNet ILSVRC2012 dataset.
We train for 90 epochs, use an initial learning rate of 0.1 with a 0.1 decay at epochs 30,60,80.
We use standard SGD with momentum of 0.9 and weight decay of 1e-4.
The minibatch size used is 256.
Following the DNNs quantization conventions (Banner et al., 2018; Nahshan et al., 2019; Choi et al., 2018b) we kept the ﬁrst and last layer (FC) at higher precision.
Additionally, similar to Sun et al.
we adopt the full precision at the shortcut which constitutes only a small amount of the computations ( ∼ 1% ).
The ”underﬂow threshold” in LUQ is updated in every bwd pass as part of the quantization of the neural gradients.
In all experiments, the BN is calculated in high-precision.
MobileNet V
We run Mobilenet V2 model from torchvision.
We use the standard pre-processing of ImageNet ILSVRC2012 dataset.
We train for 150 epochs, use an initial learning rate of 0.05 with a cosine learning scheduler.
We use standard SGD with momentum of 0.9 and weight decay of 4e-5.
The minibatch size used is 256.
Following the DNNs quantization conventions (Banner et al., 2018;
Nahshan et al., 2019; Choi et al., 2018b)
we kept the ﬁrst and last layer (FC) at higher precision.
Additionally, similar to Sun et al.
we adopt the full precision at the depthwise layer which constitutes only a small amount of the computations ( ∼ 3% ).
The ”underﬂow threshold” in LUQ is updated in every bwd pass as part of the quantization of the neural gradients.
In all experiments, the
BN is calculated in high-precision.
Transformer We run the Transformer-base model based on the Fairseq implementation on the
WMT 14 En-De translation task.
We use the standard hyperparameters of Fairseq including Adam optimizer.
We implement LUQ over all attention and feed forwards layers.
The forward phase is quantized to INT4 format with SAWB (Choi et
) while the backward phase is quantized with the proposed LUQ.
As expected, the quantization of the backward phase makes more degradation to the network accuracy.
[Table]
A. A DDITIONAL EXPERIMENTS
LUQ requires the measurement of the maximum of the neural gradient in order to get the underﬂow threshold (Section 3).
In Fig.
5a we compare the proposed LUQ with the method proposed in
Fournarakis & Nagel (2021) which suggests reducing the data movement overhead that occurs in the calculation on-the-ﬂy of the quantization ranges by using a running average of the previous iterations statistics.
As we can notice, the limited dynamic range in 4-bit quantization requires an exact statistics measurement of the tensor since the proposed approximation induces signiﬁcant accuracy degradation.
The SMP method (Section 3.1) has a power overhead of ∼ 13 of the number of additional samples since it inﬂuences only the update GEMM.
In Fig.
5b we compare LUQ with one additional sample which has ∼ 33% power overhead with regular LUQ with additional ∼ 33% epochs.
The lr scheduler was expanded respectively.
We can notice that even both methods have similar overhead the variance reduction achieved with SMP is more important for the network accuracy than increasing the training time.
[Figure]
A. T RANSFORM TO STANDARD FP
We suggest a method to avoid the use of an expensive GEMM block between the INT4 (activation or weights) and FP4 (neural gradient).
It includes 2 main elements: The ﬁrst is a simple xor operation between the sign of the two numbers and the second is a transform block to standard FP7 format.
In
Fig.
6 we present an illustration of the proposed method.
The transformation can be explained with a simple example: for simplicity, we avoid the sign which requires only xor operation.
The input arguments are 3 (011 bits representation in INT4 format) and 4 (011 bits representation in FP4 1-3format).
The concatenation brings to the bits 011 011.
Then looking at the table in the input column where the M=3 (since the INT4 argument = 3) and get the results in FP7 format of 0100 10 ( = E+ 2) which is 12 in FP7 (1-4-2) as the expected multiplication result.
In the next section, we analyze the area of the suggested block in comparison to the standard GEMM block, showing a 5x area reduction.
[Figure]
A. B ACKPROPAGATION WITHOUT MULTIPLCATION ANALYSIS
In this section, we show a rough estimation of the logical area of the proposed MF-BPROP block which avoids multiplication and compares it with the standard multiplier.
In hardware design, the logical area can be a good proxy for power consumption (Iman & Pedram, 1997).
Our estimation doesn’t include synthesis optimization.
In Table 4 we show the estimation of the number of gates of a standard multiplier, getting 264 logical gates while the proposed MF-BPROP block has an estimation of 49 gates (Table 5) achieving a ∼ 5 x area reduction.
For fair comparison we remark that in the proposed scheme the FP32 accumulator is the most expensive block with an estimation of 2453 gates, however we believe it can be reduced to a narrow accumulator such as FP16 (As previously shown in
Wang et al.
(2018) which have an estimated area of 731 gates.
In that case, we reduce the total are by ∼ 22%
[Table]
[Table] [Table]