M ONTE
C ARLO
D EEP
N EURAL
N ETWORK
A RITHMETIC
Anonymous authors
Paper under double-blind review
A BSTRACT
Quantization is a crucial technique for achieving low-power, low latency and high throughput hardware implementations of Deep Neural Networks.
Quantized ﬂoating point representations have received recent interest due to their hardware efﬁciency beneﬁts and ability to represent a higher dynamic range than ﬁxed point representations, leading to improvements in accuracy.
We present a novel technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), for determining the sensitivity of Deep Neural Networks to quantization in ﬂoating point arithmetic.
We do this by applying Monte Carlo Arithmetic to the inference computation and analyzing the relative standard deviation of the neural network loss.
The method makes no assumptions regarding the underlying parameter distributions.
We evaluate our method on pre-trained image classiﬁcation models on the
CIFAR-10 and ImageNet datasets.
For the same network topology and dataset, we demonstrate the ability to gain the equivalent of bits of precision by simply choosing weight parameter sets which demonstrate a lower loss of signiﬁcance from the Monte Carlo trials.
Additionally, we can apply MCDA to compare the sensitivity of different network topologies to quantization effects.
I NTRODUCTION
Deep Neural Networks have achieved state-of-the-art performances in many machine learning tasks such as such as speech recognition (Collobert et al., 2011), machine translation (Bahdanau et al., 2014), object detection (Ren et al., 2015) and image classiﬁcation (Krizhevsky et al., 2012).
However, excellent performance comes at the cost of signiﬁcantly high computational and memory complexity, typically requiring teraops of computation during inference and Gigabytes of storage.
To overcome these complexities, compression methods have been utilized, aiming to exploit the inherent resilience of DNNs to noise.
These engender representations which maintain algorithm performance but signiﬁcantly improve latency, throughput and power consumption of hardware implementations.
In particular, exploiting reduced numerical precision for data representations through quantization has been emphatically promising, whereby on customizable hardware, efﬁciency scales quadratically with each bit of precision.
Quantization of ﬁxed-point arithmetic (Q-FX) for DNN inference has been extensively studied, and more recently there has been increasing interest in quantized ﬂoating point (Q-FP) arithmetic for both DNN inference and training (Wang et al., 2018).
Q-FP has the advantage of higher dynamic range compared to equivalent Q-FX representations and reduced hardware cost over single-precision ﬂoating point (FP).
This has inﬂuenced application speciﬁc integrated circuits (ASICs) such as Google’s tensor processing unit (TPU), which supports 16-bit ﬂoating point (16-FP) and soft processors such as Microsoft’s Project Brainwave which utilizes 8-FP.
To illustrate these hardware beneﬁts, we synthesized arithmetic logic units (ALUs) in different formats and different precision on an FPGA and present performance estimates in operations per second (OPs) and area estimates in Look-up Tables (LUTs) per operation (LUTs/Op) in Figure 1.
As shown, 8-bit ﬁxed point (8-FX) achieves improved performance and area over 8-FP. However, 7-FP is a signiﬁcant improvement over 8 or 12-FX and 8 or 9-FP. These examples demonstrate substantial performance and area beneﬁts from reducing FP precision by only 1 to 2-bits.
Thus, if we can design networks which not only achieve high accuracy but are robust to quantization, higher performing hardware solutions are possible.
Since in Q-FP, we are trying to represent the inﬁnite set of real numbers using a ﬁnite number of bits, quantization and rounding artefacts will be introduced, with inaccuracies being cascaded along the computation graph (IEEE, 1985; Goldberg, 1991; Higham, 2002).
This paper proposes Monte
Carlo Deep Neural Network Arithmetic (MCDA), a novel way to apply Monte Carlo Arithmetic (MCA) (Parker et al., 2000) for determining the sensitivity of Deep Neural Networks to Q-FP representations.
It allows hardware-software designers to quantify the impact of quantization, enabling more efﬁcient systems to be discovered.
We do this by exploiting Monte Carlo simulations under which rounding effects are randomized.
This, in turn, allows one to infer the sensitivity of executing a computation graph to quantization effects.
Our MCDA technique is highly sensitive, allowing very small differences in quantization behaviour to be detected.
The technique makes no assumptions regarding data distributions and directly measures the effects of quantization on the problem under study.
This allows us to provide insights into the precision requirements of any inference network for any given dataset.
Additionally we can use the technique to select weight parameters which are more robust to ﬂoating point rounding.
The theoretical and practical contributions of this work can be summarized as follows:
• We introduce a novel and rigorous analysis technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), which measures the sensitivity of Deep Neural Network inference computation to ﬂoating point rounding error.
• When applied to neural network inference, we show MCDA can determine the precision requirements of different networks, rank them, and detect small differences between different neural network topologies and weight sets.
• We demonstrate that while a network with the same topology but different weights may have the same loss and validation accuracy, their sensitivity to quantization can be vastly different.
Using the CIFAR-10 and ImageNet datasets, we introduce a method to choose weights which are more robust to rounding error, resulting in a greatly improved accuracyarea tradeoff over state-of-the-art methods.
It is worth noting that although we consider convolutional neural networks for image classiﬁcation, this method could be applied for any neural network model architectures and applications.
Moreover, while the experiments in this paper are limited to inference, it may be possible to apply the same idea to analyze training algorithms.
R ELATED W ORK
Low-precision representations of deep learning have been extensively studied.
Many training methods have been developed to design representations for ﬁxed point inference (Jacob et al., 2018;
Faraone et al., 2018;
Zhou et al., 2016) and training (Wu et al., 2018b; Yang et al., 2019; Gupta et al., 2015; Sakr & Shanbhag, 2019).
Other methods have also utilized Q-FP arithmetic for inference and training whilst maintaining single-precision accuracy.
(Micikevicius et al., 2018) implemented 16FP arithmetic training whilst storing a 32-FP master copy for the weight updates.
Additionally,
(Wang et al., 2018; Mellempudi et al., 2019) with 8-FP arithmetic whilst using a 16-FP copy of the weights and 16/32-bits for the accumulator.
Techniques for determining per-layer sensitivity to quantization have also been studied (Choi et al., 2016; Sakr & Shanbhag, 2018).
Further, other studies have successfully determined the minimum ﬁxed point precision requirements for a given DNN accuracy threshold (Sakr et al., 2017).
The accuracy and stability of various numerical algorithms in ﬁnite precision arithmetic has been studied in (Higham, 2002; Wilkinson, 1994).
This has led to techniques for tracking information lost from ﬁnite precision arithmetic using random perturbation such as Monte Carlo Arithmetic (Parker et al., 2000; Frechtling & Leong, 2015).
Monte Carlo methods have also been used in Bayesian Neural Networks (Buchholz et al., 2018; Blier & Ollivier, 2018). In particular, (Achterhold et al., 2018) introduced a quantizing prior to learn weights which are either close to a quantized representation or have high variance. (Louizos et al., 2017) used hierarchical priors to prune nodes and posterior uncertainties to determine the optimal ﬁxed point precision. (Blundell et al., 2015) use a Monte Carlo approach to learn a probability distribution on the weights of a neural network.
To the best of our knowledge, our work is the ﬁrst to present a technique for directly determining the sensitivity of DNNs to ﬂoating point rounding and to explicitly compute precision bounds of a trained network.
These ideas can be very usefully applied to extending the limits of low-precision representations in deep learning applications.
B ACKGROUND
In this section, we describe background theory upon which our technique for determining the sensitivity of DNNs to ﬂoating point rounding is based.
3. F LOATING P OINT A RITHMETIC
The IEEE-754 binary ﬂoating point format (IEEE, 1985) represents most real numbers x by a subset in normal form as:
[Equation]
where s x ∈ { 0 , 1 } is the sign bit, e x is an integer representing the exponent of ˆ x and m x is the mantissa of ˆ x . Such number formats can be described as a ( s x , e x , m x ) tuple.
In binary form the representation is ( b s , b e 1 , b e 2 , ..., b eB ex , b
m 1 , b m 2 ..., b mB mx ) ∈ { 0 , 1 } B , with B e x and B m x being the number of exponent and mantissa bits, respectively.
The inﬁnite set of real numbers R is represented in a computer with B = 1+ B e x + B m x bits, and we deﬁne the ﬁnite set of real numbers representable in ﬂoating point format as exact values , F ⊂ R .
Real numbers which aren’t representable are rounded to their nearest exact value. We call this set of numbers inexact values , I , where I ∪ F = R .
The approximation ˆ x = F ( x ) = x (1 + δ ) , given x ∈ I , introduces rounding error into the computation.
The value of δ = (cid:13)(cid:13) x − ˆ xx (cid:13)(cid:13) , represents the relative error which is a function of the machine hardware precision, p , as δ ≤ (cid:15) , where (cid:15)
= 2 − p (IEEE, 1985; Goldberg, 1991; Higham, 2002).
In general, inexactness can be caused by ﬁnite representations or errors propagating from earlier parts of the computation.
Often the primary cause of error in ﬂoating point arithmetic is catastrophic cancellation which causes numerical inaccuracy.
Catastrophic cancellation occurs when for example, two near equal FP numbers, sharing k signiﬁcant digits, are subtracted from one another as shown in (2) (Higham, 2002).
[Equation]
In normalized form, the leading zeros are removed by shifting the result to the left and adjusting the exponent accordingly.
The result is 0 .h
k +1 ...
h t
i 1 ...i k which has only ( t − k ) accurate digits and digits i which are unknown.
Additionally, the remaining accurate digits h are most likely affected by rounding error in previous computations.
This can signiﬁcantly magnify errors, especially in computing large computational graphs such as that of state-of-the-art DNNs.
If either operand in (2) is inexact, then the digits h are no more signiﬁcant than any other sequence of digits.
Yet, FP arithmetic has no mechanism of recording this loss of signiﬁcance.
By padding both our operands with random digits r and ˆ r in (3), the resulting digits i are randomized.
If k digits are lost in the result, then k random digits will be in the normalized result and when computed over many random trials, the results will disagree on the trailing k digits.
In this case, we are able to detect catastrophic cancellation because the randomization over many trials provides a statistical simulation of round-off errors.
We can use techniques from numerical analysis such as Monte Carlo methods to appropriately insert precision-dependant randomization in this way.
3. M ONTE C ARLO A RITHMETIC
Monte Carlo methods can be used to analyze rounding by representing inexact values as random variables (Parker et al., 2000; Frechtling & Leong, 2015).
The real value x , as represented in (1), can be modelled to t digits, using:
[Equation]
where δ ∈ U ( − 12 , 12 ) is a uniformly distributed random variable and t is a positive integer representing the virtual precision of concern.
For the same input ˆ x in (4), we can run many Monte Carlo trials which will yield different values on each trial, where 0 < t < p so that the MCA can be run accurately on a computer with machine precision p .
The ability to vary t is useful because it allows us to then evaluate the hardware precision requirements of a given system or computational graph for a given DNN.
MCA is a method to model the effect of rounding on a computational graph by randomizing all arithmetic operations.
The randomization is applied for both generating inexact operands and also in rounding.
In each operation using MCA, ideally both catastrophic cancellation and rounding error can be detected.
An operation using MCA is deﬁned as:
[Equation]
where ◦ ∈ (cid:0) + , − , × , ÷ ) .
By applying the inexact function to both operators we make it possible to detect catastrophic cancellation.
Furthermore, applying the inexact function to the operation output and then rounding this value implements random rounding and hence is used to detect rounding error (Parker & Langley, 1997).
Hence, for the same input into the system, each trial will yield different operands and output.
After modifying the inexact and rounding operations as described, we use random sampling to simulate Monte Carlo trials.
For each trial, we collect data on the resulting output of the system and compute summary statistics to quantify its behaviour (Parker et al., 2000).
With sufﬁciently large number of Monte Carlo trials and virtual precision t , the expected value of the output from these trials will equal the value from using real arithmetic.
As explained in the next section, we can determine the total number of digits lost to rounding error and the minimum precision required to avoid a total loss of signiﬁcance.
3. A NALYSIS
The relative error is bounded by δ ≤ 2 − p from the design of IEEE FP arithmetic (Wilkinson, 1994; Goldberg, 1991).
With this inequality, we can determine the expected number of signiﬁcant binary digits available from a p-digit FP system as p ≤ − log 2 ( δ ) .
These deﬁnitions can be adapted for MCA by replacing the precision of the FP system, p , by the virtual precision, t , of an MCA operation.
Thus, the relative error of an MCA operation, for virtual precision t , is δ ≤ 2 − t and the expected number of signiﬁcant binary digits in a t -digit MCA operation is at most t .
Using this deﬁnition and the proof provided in (Parker & Langley, 1997), the total signiﬁcant binary digits in a set of M trials is s (cid:48)
= log 2 µσ where µ is the mean and σ the standard deviation.
The output of the system should be some scalar value so that we can perform such analysis.
For experimentation, M trials are run for all of t ∈ { 1 , 2 , 3 , ..., t max } .
The total number of base-2 signiﬁcant digits lost in a set of M trials is K t , in (6):
[Equation]
where Θ = σµ , for ( µ (cid:54) = 0 ), is the relative standard deviation (RSD) of the MCA results.
The virtual precision t controls the perturbation strength applied by the inexact function.
For a given K t , as we reduce t , the RSD should increase according to equation 6.
At some point, an unexpected loss of signiﬁcance (Frechtling & Leong, 2015) is encountered due to the nonlinear effects of quantization.
The value at which this occurs is deﬁned as t min .
The number of signiﬁcant digits lost for the system being analyzed is then computed by averaging all K t whereby t > t min as shown in (7):
[Equation] where t max > K [Equation]
where t max =(7)
For DNN inference, we propose to use K as a sensitivity measure for the network to FP rounding.
The method for implementing this is discussed in the next section.
We now describe MCDA, a methodology for applying MCA techniques to DNN computation, allowing us to understand the sensitivity of a given network and its weight representation to FP rounding.
4. N ETWORK M ODEL
We consider a generalized non-linear L -layer neural network with an output vector y L , input data vector x and learnable weight parameter tensor w = (cid:83) w l ( l = 1 , . . . , L ), whereby y L = f
( x ; w ) .
To compute y , several layers consisting of general matrix multiplication (GEMM) operations (such as convolutional and fully-connected layers) between the layer input x l and weight parameters w l , followed by a non-linear activation function, h , producing intermediate layer outputs x l , i.e. y l = h ( x l ⊗ w l ) .
The output of a given layer becomes the input to the subsequent {+layer, i.e. x l +
 =+}
y l , with x 1 = x . A loss function is the objective function to minimize updating w via an optimizer such as stochastic gradient descent.
For a given set of input data X , the total network loss during inference is calculated by applying a loss function loss ( f ( x ; w ) , ˆ y ( x )) where ˆ y ( x ) is the target ground truth output for x .
The total loss for X is then a scalar output, such that:
[Equation]
Updates of w are usually done in small batches over subsets of X .
Naively applying MCA to each operation (ﬁne-grained MCA) as described in equation (5) poses signiﬁcant computational difﬁculties for DNN models.
We observe two primary issues with employing ﬁne-grained MCA to a DNN computational graph:
• Firstly, the number of required trials for Monte Carlo experiments to generate robust results can typically be in the hundreds or thousands.
As DNN inference of state-of-the-art networks typically consist of billions of operations, the computational requirements of applying MCA after each operation will be very large, making the technique impractical.
• Using the accuracy as the system output for MCA experiments is problematic because it is a discrete value.
For high values of t , Monte Carlo results across different t then become indistinguishable and the standard deviation for a given t is potentially zero.
4. M ONTE C ARLO N ETWORK I NFERENCE
To reduce the computational cost of Monte Carlo experiments, we employ MCDA, which is a coarsegrained approach to MCA for GEMM operations.
Conveniently, these can be naturally implemented in modern machine learning frameworks such as PyTorch.
Furthermore, to ensure the system output is a continuous value, the loss function output is used, rather than the accuracy.
In this case, small perturbations in layer operands are more likely to produce observable changes in the output.
MCDA applies a vector version of (5) to the DNN inference computational problem in (8).
Our operands in this case are vectors and ◦ represents a neural network layer operation.
For example, the output from performing a GEMM operation can thus be represented by:
[Equation]
Since the inexact function is applied to the inputs and outputs of a GEMM operation, an optimized implementation can be used. This is in contrast to full MCA which requires the application of (5) to every individual scalar operation.
The vector form in (9) is applied to each edge of neural network computational graph where multiply (division) and/or add (subtract) operations are performed.
Hence, it is not applied to operations such as MaxPool and ReLu.
As an example, in Figure 2 we show where the inexact function is applied for a residual block with folded batch normalization, which is a repeating sequence of layers found in ResNet models (He et al., 2015).
At the ﬁnal output of the network, the loss is computed with (8) and the inexact function is applied to the outputs y and also the loss output scalar value L .
From analyzing the behavior of the loss, we infer the sensitivity of the accuracy of the system to FP rounding.
By using MCDA for the GEMM operations, we will not be able to detect all instances of catastrophic cancellation.
However, we signiﬁcantly reduce execution time over ﬁne-grained MCA and show in the next section that we can still retrieve valuable information about our system.
In fact, for one trial with one batch of 32 images on ImageNet running on a Nvidia Titan Xp GPU, the speed up of regular inference without MCDA is only 1.05 × (for a single Monte Carlo trial).
We also note that ﬁne-grained MCA could be applied with a simple modiﬁcation and would be possible given appropriate customized hardware support for parallel Monte Carlo computations (Yeung et al., 2011).
E XPERIMENTAL R ESULTS
In this section, we present experimental results for applying MCDA to exemplary convolutional neural networks.
We use the CIFAR-10 and ImageNet image classiﬁcation datasets to compare
MobileNet-v2 (Sandler et al., 2018), EfﬁcientNet (Tan & Le, 2019), AlexNet, ResNet (He et al., 2015), SqueezeNet (Iandola et al., 2016) and MnasNet (Tan et al., 2018).
For CIFAR-10, we use a batch size of 128, whilst we use a batch size of 32 for ImageNet experiments.
Cross-entropy is used as the loss function for both datasets.
For a given network, dataset, weight representation and t , we apply MCDA with T = 1000 trials.
The resulting loss from each trial is computed with the same single batch of images and hence additional data is not required. We compute Θ t from our results for all t ∈ { 1 , 2 , 3 , ..., 16 } . Following this, we run linear regression analysis using the MCALIB(Frechtling & Leong, 2015) R library, on our Θ t values, to determine t min and K .
t min is deﬁned as the point of lowest t where the difference between the regression line and the equivalent Θ t is less than half a binary digit, i.e. log 10 (2 0 . 5 ) . Further detail on the calculation of K and t min from
MCALIB can be found in Appendix A.1.
We report Q-FP validation accuracy using the quantization function from (Wang et al., 2018) with stochastic rounding 3 (See Appendix A.2).
5. D ISTINGUISHING W EIGHT P ARAMETER R EPRESENTATIONS
As discussed in Section 3.1, the inexactness in FP arithmetic largely depends on the numerical value of operands.
Two instances of the same network and dataset, with the same validation accuracy, but vastly different weight representations, will likely produce differing sensitivities to FP rounding.
We ﬁrst train 8 instances of EfﬁcientNet-b0 and MobileNet-V2 on CIFAR-10 from scratch with random initialization from (Glorot & Bengio, 2010), all achieving within 1% validation accuracy of one another. Using MCDA, we calculate the K values for each model (See Appendix A.3). [Figure]
[Table] We then test their percentage validation accuracy decrease from using post-training quantization (i [Table] e. no ﬁnetuning) with varying Q-FP precisions.
In Figure [Table] we see that the models with higher
K values typically experience a larger drop in Q-FP accuracy, indicating they are more sensitive to ﬂoating point rounding error.
Notably, the model with lowest K for 7-bit MobileNet-v2 experiences a lower percentage validation accuracy drop than three of the 8-bit models.
In this case, MCDA model selection enables the saving of a bit of precision while achieving smaller accuracy decrease than some of the trained 8-FP models
[Table]
5. C OMPARISON T O P REVIOUS W ORK
One practical use case from the insights gained by MCDA is model selection for quantization.
Typically when quantizing a given model trained on a given dataset, the model with highest validation accuracy is chosen and the sensitivity to quantization is assumed to be the same across models.
As discussed, for Q-FP representations this is not necessarily the case.
We can use K from MCDA to predict which models will be more robust to quantization.
To demonstrate this, in Table 1 we compare post-training quantization results for model selection based on K from MCDA, against a baseline model chosen based on the highest single-precision validation accuracy.
Evidently, even though the single-precision accuracy is initially as much as 0.9% higher, after quantizing the network to 8-5 bits, the accuracy of the network chosen by smallest K is always signiﬁcantly higher.
5. N ETWORK C OMPARISON
Modern DNNs consist of convolutional blocks with highly varying computational graphs (Wu et al., 2018a; Howard et al., 2017).
Using MCDA we can also compute and compare their sensitivites to ﬂoating point rounding error to determine which networks will be robust to Q-FP representations.
In Figure 4 we show the Θ t of pre-trained models, trained on the ImageNet dataset from PyTorch 4, for differing values of t and run linear regression analysis over our data points.
From here we can then assign a K value to each network and compare their loss of signiﬁcance.
At each t , the distance from the regression lines to the ideal line represents the values of K t , as described in (6).
From the MCDA results, AlexNet is the least sensitive to rounding and ResNet-50 is the most, with various models in between these two.
Additionally we compare EfﬁcientNet at two different model scales and evidently the larger model has much larger sensitivity.
We then also compare the validation accuracy percentage decrease of all models at 10, 9 and 8-FP post-training in Figure 5.
At 8-FP, besides MnasNet which experiences a large accuracy drop, K is able to predict validation accuracy degradation.
Thus, MCDA provides very valuable information about Q-FP model design.
C ONCLUSION
We present a novel, highly sensitive, technique to quantify rounding error in DNNs.
This is the ﬁrst method to successfully compare the sensitivity of networks to ﬂoating point rounding error.
Ultimately, this technique provides a tool for enabling the design of networks which perform higher when quantized.
We do this by applying concepts from Monte Carlo Arithmetic theory to DNN computation.
Furthermore, we show that by calculating the loss of signiﬁcance metric K from
MCDA, on the CIFAR-10 and ImageNet datasets, we can compare network sensitivities to ﬂoating point rounding error and gain valuable insights to potentially design better neural networks.
This is an important contribution due to the increasing interest in low-precision ﬂoating point arithmetic for efﬁcient DNN hardware systems.
The theoretical and practical contributions of this paper will likely translate well to analyzing ﬂoating point rounding in backpropagation in future work.
[Figure]
A A PPENDIX A. E XPERIMENTAL S ETUP
In all our experiments for a given network dataset weight representation and t we run 1000 Monte Carlo trials. The network loss output from each trial is computed with the same single batch of images from the training dataset.
We then independently compute the Θ t of the network loss for t ∈ { 1 , 2 , 3 , ..., t max } where t max = 16 . Following this we then run linear regression and calculate K and t min using MCALIB 6 . To compute the linear regression, MCALIB uses a log transformed variable, with log(Θ) as the dependant variable and t as the exploratory variable (Frechtling & Leong, 2015). [Equation]
[Equation]
Initialize: Pre-train a single-precision DNN model.
Set t max .
Set number of trials M .
Inputs: Batch of inputs & targets ( X, ˆ Y ) , loss function loss ( f ( x ; w ) , ˆ y ( x )) , current weights w
Outputs: t min and K
Monte Carlo Trials: for t = 1 to t max do for trials=1 to M do
L ( X ; w ) =
ForwardPath ( X, ˆ Y , w , t ) using (9) end for
Compute µ and σ of all trials
Compute Θ t using (6) end for
Calculate K and t min : for t = 1 to t max do
Compute P t using (10) - (14)
Compute P t − Θ t if P t − Θ t < log 10 (2 0 .
5 ) then t min = t break elsecontinueendif end for
Compute K using (6) and (7) where m = − log 10 (2) = − 0 .
30103 is the slope and c is the intercept such that K = log 2 (10 c ) .
Given these inputs, the intercept c is calculated by minimizing the following objective function using
Brents method (Brent, 1973) for single variable optimization:
[Equation]
(13) where e i =
Θ i − ( mt i + c ) is the residual error, c ∈ [(Θ t max − mt max ) ± 2 m ] is the initial search space for the intercept, γ = 0 .
75 and ρ H ( e ) is the Huber loss function (Huber, 1964): ρ H ( e ) = (cid:26) 12 efor | e | ≤ k k | e | − 1 2 kfor | e | > k (14)
where k
= 1 .
345 σ e and σ e is the standard deviation of the residual error set, e .
After determining the linear regression model, P t = mt + c , we determine whether each Θ t is an outlier.
If a value for Θ t differs to the equivalent P t by more than half a binary digit, then it is classed an outlier.
t min is then deﬁned as the lowest t where P t − Θ t < log 10 (2 0 . 5 ) .
To compute K for a given network, we then average the values for K t whereby t > t min .
This removes the outliers from the computation of K .
We have summarized how the experiments were simulated in Algorithm 1.
A. Q UANTIZED F LOATING P OINT W ITH S TOCHASTIC R OUNDING
To quantize our pre-trained networks to Q-FP representations, we used stochastic rounding as described in (Wang et al., 2018) and implemented in QPytorch 7 .
Two common forms of rounding for
FP arithmetic are round-to-nearest and stochastic rounding.
The former discards information in the least signiﬁcant bit (LSB) which is rounded off.
This information loss can be signiﬁcant, especially when quantizing to a small number of bits.
Stochastic rounding provides a method to capture this information loss from rounding off the LSB.
Given x = ( − 1) s x
(1 + m x )2 e x as described in (1).
Assume that m x is in ﬁxed precision with z (cid:48) bits which needs to be rounded to z bits, then stochastic rounding is as follows:
[Equation]
where (cid:98) m x (cid:99) is the truncation of z (cid:48)
− z LSBs of m , (cid:15) =
2 − z .
For each Q-FP accuracy reported in our experiments, we tested all possible combinations of the number of bits allowed for the exponent and mantissa which satisﬁed the desired precision.
We then chose the combination which produced the highest accuracy.
A. CIFAR-10 R EGRESSION A NALYSIS
In Figure 6, we display the linear regression analysis using MCALIB for the 8 MobileNet-v2 and
EfﬁcientNet-b0 models trained on CIFAR-10.
As shown, Monte Carlo trials were run for each of these models for t ∈ { 1 , 2 , 3 , ..., 16 } .
The corresponding t min and K values were then calculated using methods discussed in Appendix A.1.
Thus, in Figure 3, it is these K values which are plotted against 8 and 7-bit Q-FP validation accuracy.
Also, the models with lowest K values are chosen for
MCDA model selection in Table 1.