Original: A B LOCK M INIFLOAT R EPRESENTATION FOR T RAINING D EEP N EURAL N ETWORKS
Revised: A B LOCK M INIFLOAT R EPRESENTATION FOR T RAINING D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training Deep Neural Networks (DNN) with high efﬁciency can be difﬁcult to achieve with native ﬂoating point representations and commercially available hardware.
Revised: Training Deep Neural Networks (DNN) with high efﬁciency can be difﬁcult to achieve with native ﬂoating point representations and commercially available hardware.

Original: Specialized arithmetic with custom acceleration offers perhaps the most promising alternative.
Revised: Specialized arithmetic with custom acceleration offers perhaps the most promising alternative.

Original: Ongoing research is trending towards narrow ﬂoating point representations, called miniﬂoats, that pack more operations for a given silicon area and consume less power.
Revised: Ongoing research is trending towards narrow ﬂoating point representations, called miniﬂoats, that pack more operations for a given silicon area and consume less power.

Original: In this paper, we introduce Block Miniﬂoat (BM), a new spectrum of miniﬂoat formats capable of training DNNs end-to-end with only 4-8 bit weight, activation and gradient tensors.
Revised: In this paper, we introduce Block Miniﬂoat (BM), a new spectrum of miniﬂoat formats capable of training DNNs end-to-end with only 4-8 bit weight, activation and gradient tensors.

Original: While standard ﬂoating point representations have two degrees of freedom, via the exponent and mantissa, BM exposes the exponent bias as an additional ﬁeld for optimization.
Revised: While standard ﬂoating point representations have two degrees of freedom, via the exponent and mantissa, BM exposes the exponent bias as an additional ﬁeld for optimization.

Original: Crucially, this enables training with fewer exponent bits, yielding dense integer-like hardware for fused multiply-add (FMA) operations.
Revised: Crucially, this enables training with fewer exponent bits, yielding dense integer-like hardware for fused multiply-add (FMA) operations.

Original: For ResNet trained on ImageNet, 6-bit
Revised: For ResNet trained on ImageNet, 6-bit

Original: BM achieves almost no degradation in ﬂoating point accuracy with FMA units that are 4 .
Revised: BM achieves almost no degradation in ﬂoating point accuracy with FMA units that are 4 .

Original: 1 × (23 . 9 × ) smaller and consume 2 .
Revised: 1 × (23 . 9 × ) smaller and consume 2 .

Original: 3 × (16 . 1 × ) less energy than FP (FP32).
Revised: 3 × (16 . 1 × ) less energy than FP (FP32).

Original: Furthermore, our 8-bit BM format matches ﬂoating-point accuracy while delivering a higher computational density and faster expected training times.
Revised: Furthermore, our 8-bit BM format matches ﬂoating-point accuracy while delivering a higher computational density and faster expected training times.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The energy consumption and execution time associated with training Deep Neural Networks (DNNs) is directly related to the precision of the underlying numerical representation.
Revised: The energy consumption and execution time associated with training Deep Neural Networks (DNNs) is directly related to the precision of the underlying numerical representation.

Original: Most commercial accelerators, such as NVIDIA Graphics Processing Units (GPUs), employ conventional ﬂoating point representations due to their standard of use and wide dynamic range.
Revised: Most commercial accelerators, such as NVIDIA Graphics Processing Units (GPUs), employ conventional ﬂoating point representations due to their standard of use and wide dynamic range.

Original: However, double-precision (FP64) and single-precision (FP32) formats have relatively high memory bandwidth requirements and incur signiﬁcant hardware overhead for general matrix multiplication (GEMM).
Revised: However, double-precision (FP64) and single-precision (FP32) formats have relatively high memory bandwidth requirements and incur signiﬁcant hardware overhead for general matrix multiplication (GEMM).

Original: To reduce these costs and deliver training at increased speed and scale, representations have moved to 16-bit formats, with NVIDIA and Google providing FP16 (IEEE-754, 2019) and Bﬂoat16 (Kalamkar et al., 2019) respectively.
Revised: To reduce these costs and deliver training at increased speed and scale, representations have moved to 16-bit formats, with NVIDIA and Google providing FP16 (IEEE-754, 2019) and Bﬂoat16 (Kalamkar et al., 2019) respectively.

Original: With computational requirements for DNNs likely to increase, further performance gains are necessary in both datacenter and edge devices, where there are stricter physical constraints.
Revised: With computational requirements for DNNs likely to increase, further performance gains are necessary in both datacenter and edge devices, where there are stricter physical constraints.

Original: New number representations must be easy to use and lead to high accuracy results.
Revised: New number representations must be easy to use and lead to high accuracy results.

Original: Recent 8-bit ﬂoating-point representations have shown particular promise, achieving equivalent FP32 accuracy over different tasks and datasets (Wang et al., 2018; Sun et al., 2019).
Revised: Recent 8-bit ﬂoating-point representations have shown particular promise, achieving equivalent FP32 accuracy over different tasks and datasets (Wang et al., 2018; Sun et al., 2019).

Original: We refer to such representations as miniﬂoats in this paper.
Revised: We refer to such representations as miniﬂoats in this paper.

Original: Miniﬂoats are ideal candidates for optimization.
Revised: Miniﬂoats are ideal candidates for optimization.

Original: By varying the number of exponent and mantissa bits, many formats can be explored for different trade-offs of dynamic range and precision.
Revised: By varying the number of exponent and mantissa bits, many formats can be explored for different trade-offs of dynamic range and precision.

Original: These include logarithmic and ﬁxed point representations which provide substantial gains in speed and hardware density compared to their ﬂoating-point counterparts.
Revised: These include logarithmic and ﬁxed point representations which provide substantial gains in speed and hardware density compared to their ﬂoating-point counterparts.

Original: For instance, 32-bit integer adders are approximately 10 × smaller and 4 × more energy efﬁcient than comparative
Revised: For instance, 32-bit integer adders are approximately 10 × smaller and 4 × more energy efﬁcient than comparative

Original: FP16 units (Dally, 2015).
Revised: FP16 units (Dally, 2015).

Original: That said, ﬁxed point representations still lack the dynamic range necessary to represent small gradients for backpropagation, and must be combined with other techniques for training convergence.
Revised: That said, ﬁxed point representations still lack the dynamic range necessary to represent small gradients for backpropagation, and must be combined with other techniques for training convergence.

Original: Block ﬂoating point (BFP) in (Yang et al., 2019; Drumond et al., 2018) share exponents across blocks of 8-bit integer numbers, and provide a type of coarse-grained dynamic range for training.
Revised: Block ﬂoating point (BFP) in (Yang et al., 2019; Drumond et al., 2018) share exponents across blocks of 8-bit integer numbers, and provide a type of coarse-grained dynamic range for training.

Original: This approach will typically incur some accuracy loss on more challenging datasets, however all dotproducts within the block can be computed with dense ﬁxed point logic.
Revised: This approach will typically incur some accuracy loss on more challenging datasets, however all dotproducts within the block can be computed with dense ﬁxed point logic.

Original: In comparison, HFP8 (Sun et al., 2019) miniﬂoats require larger ﬂoating-point units (expensive FP16 adders in particular) but have at least 5 exponent bits dedicated to each gradient and suffer zero degradation in accuracy.
Revised: In comparison, HFP8 (Sun et al., 2019) miniﬂoats require larger ﬂoating-point units (expensive FP16 adders in particular) but have at least 5 exponent bits dedicated to each gradient and suffer zero degradation in accuracy.

Original: It would seem that an ideal representation for training with both high efﬁciency and accuracy should bridge the gap between each of these approaches.
Revised: It would seem that an ideal representation for training with both high efﬁciency and accuracy should bridge the gap between each of these approaches.

Original: Our work achieves this for 8-bit and sub 8-bit precision schemes, overcoming two key challenges in the process.
Revised: Our work achieves this for 8-bit and sub 8-bit precision schemes, overcoming two key challenges in the process.

Original: These are listed below and discussed with related works.
Revised: These are listed below and discussed with related works.

Original: 1. C HALLENGES AND R ELATED W ORK
Revised: 1. C HALLENGES AND R ELATED W ORK

Original: Minimising data loss with fewer bits: While several works have demonstrated training with fewer than 8 bits of precision, they typically lead to loss of accuracy on more complex problems and have performance bottlenecks when parts of the algorithm are left in high precision (Hubara et al., 2017;
Revised: Minimising data loss with fewer bits: While several works have demonstrated training with fewer than 8 bits of precision, they typically lead to loss of accuracy on more complex problems and have performance bottlenecks when parts of the algorithm are left in high precision (Hubara et al., 2017;

Original: Zhou et al., 2016; Miyashita et al., 2016).
Revised: Zhou et al., 2016; Miyashita et al., 2016).

Original: The ability to train with persistent reduced precision representations without losing accuracy remains a key challenge for end-to-end training.
Revised: The ability to train with persistent reduced precision representations without losing accuracy remains a key challenge for end-to-end training.

Original: In this regard, 8 bit tensors with 16-bit updates can be trained effectively (Banner et al., 2018).
Revised: In this regard, 8 bit tensors with 16-bit updates can be trained effectively (Banner et al., 2018).

Original: Data loss arises when formats do not have enough range to capture variations in tensor distributions during training.
Revised: Data loss arises when formats do not have enough range to capture variations in tensor distributions during training.

Original: BFloat (Kalamkar et al., 2019) adds two extra exponent bits to their custom 16-bit representation, and the Apex library is used in (Micikevicius et al., 2017;
Revised: BFloat (Kalamkar et al., 2019) adds two extra exponent bits to their custom 16-bit representation, and the Apex library is used in (Micikevicius et al., 2017;

Original: Wang et al., 2018; Sun et al., 2019) for scaling the loss function into a numerically representable range.
Revised: Wang et al., 2018; Sun et al., 2019) for scaling the loss function into a numerically representable range.

Original: Block ﬂoating point and other variants apply similar functionality for ﬁxed point numbers, but at a ﬁner granularity.
Revised: Block ﬂoating point and other variants apply similar functionality for ﬁxed point numbers, but at a ﬁner granularity.

Original: WAGE (Wu et al., 2018) uses layer-wise scaling factors, SWALP (Yang et al., 2019)
Revised: WAGE (Wu et al., 2018) uses layer-wise scaling factors, SWALP (Yang et al., 2019)

Original: shares exponents across feature maps or convolution channels, and HBFP (Drumond et al., 2018) does the same for dot-products, though their implementation requires caching of intermediate activations in FP32 and wide weight storage for better accuracy.
Revised: shares exponents across feature maps or convolution channels, and HBFP (Drumond et al., 2018) does the same for dot-products, though their implementation requires caching of intermediate activations in FP32 and wide weight storage for better accuracy.

Original: S2FP8 (Cambier et al., 2020) replaces loss-scaling in FP8 (Wang et al., 2018) with squeeze and shift factors that center 8-bit miniﬂoats over the mean exponent of the value distribution.
Revised: S2FP8 (Cambier et al., 2020) replaces loss-scaling in FP8 (Wang et al., 2018) with squeeze and shift factors that center 8-bit miniﬂoats over the mean exponent of the value distribution.

Original: Shift factors work in a similar fashion to BFP shared exponents, however squeeze factors canpotentially cause precision to be diverted away from high value regions leading to errors in dot-product calculations.
Revised: Shift factors work in a similar fashion to BFP shared exponents, however squeeze factors can cause precision to be diverted away from high value regions leading to errors in dot-product calculations.

Original: We provide some empirical evidence of this effect in Section 4.5.
Revised: We provide some empirical evidence of this effect in Section 4.5.

Original: Finally, HFP8 (Sun et al., 2019) deﬁnes two miniﬂoat formats that are optimized for range and precision requirements of forward and backward paths separately.
Revised: Finally, HFP8 (Sun et al., 2019) deﬁnes two miniﬂoat formats that are optimized for range and precision requirements of forward and backward paths separately.

Original: Ideally, we seek miniﬂoat formats that have also been optimized for arithmetic density.
Revised: In this work, we seek miniﬂoat formats that have also been optimized for arithmetic density.

Original: Increasing the performance density of ﬂoating-point: Most DNN training frameworks are developed with GEMM accumulation in FP32.
Revised: Increasing the performance density of ﬂoating-point: Most DNN training frameworks are developed with GEMM accumulation in FP32.

Original: The authors in (Wang et al., 2018) reduced the accumulation width to FP16 with chunk-based computations and stochastic rounding.
Revised: The authors in (Wang et al., 2018) reduced the accumulation width to FP16 with chunk-based computations and stochastic rounding.

Original: However, training miniﬂoats with even denser dot-products has not been demonstrated.
Revised: However, training miniﬂoats with even denser dot-products has not been demonstrated.

Original: For DNN inference, ELMA (Johnson, 2018) and posit number systems (Gustafson & Yonemoto, 2017) describe arithmetic that accumulate miniﬂoat-like numbers as integers.
Revised: For DNN inference, ELMA (Johnson, 2018) and posit number systems (Gustafson & Yonemoto, 2017) describe arithmetic that accumulate miniﬂoat-like numbers as integers.

Original: Such work is applicable when the number of exponent bits is small, however training under such regimes may lead to data loss due to limited dynamic range.
Revised: Such work is applicable when the number of exponent bits is small, however training under such regimes can lead to data loss due to limited dynamic range.

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: In this paper, we present the Block Miniﬂoat (BM) representation which addresses both of the aforementioned challenges.
Revised: In this paper, we present the Block Miniﬂoat (BM) representation which addresses both of the aforementioned challenges.

Original: BM is a modiﬁcation of block ﬂoating point that replaces the ﬁxed point values with miniﬂoats, whilst maintaining shared exponents across blocks of numbers.
Revised: BM is a modiﬁcation of block ﬂoating point that replaces the ﬁxed point values with miniﬂoats, whilst maintaining shared exponents across blocks of numbers.

Original: BM formats generalise a far wider spectrum of reduced precision representations and produce better outcomes than previous 8-bit regimes.
Revised: BM formats generalise a far wider spectrum of reduced precision representations and produce better outcomes than previous 8-bit regimes.

Original: Speciﬁc contributions of our work include:
Revised: Speciﬁc contributions of our work include:

Original: • Block Miniﬂoat (BM), an efﬁcient alternative to INT8 and FP8 for end-to-end DNN training with comparable accuracy.
Revised: • Block Miniﬂoat (BM), an efﬁcient alternative to INT8 and FP8 for end-to-end DNN training with comparable accuracy.

Original: Shared exponent biases provide dynamic range and accuracy, while small exponent encodings provide ﬁne-grained dynamic range and reduce the hardware cost ofFP8/FP16 GEMM accumulation.
Revised: Shared exponent biases provide dynamic range and accuracy, while small exponent encodings provide ﬁne-grained dynamic range and reduce the hardware cost of GEMM accumulation.

Original: • A new 8-bit ﬂoating-point format that uses no more than3 exponent bits for forward propagation and 4 exponent bits for backward propagation, achieving equivalent accuracy to ﬂoating-point with denser hardware via efﬁcient Kulisch accumulation.
Revised: • A new 8-bit ﬂoating-point format that uses no more than 4 exponent bits , achieving equivalent accuracy to ﬂoating-point with denser hardware via efﬁcient Kulisch accumulation.

Original: • An exploration of the BM design space showing high accuracy DNN training with sub 8-bit representations for all weights, activations and gradients.
Revised: • An exploration of the BM design space showing high accuracy DNN training with sub 8-bit representations for all weights, activations and gradients.

Original: This includes two techniques for minimising data loss of a practical implementation, namely gradual underﬂow and cost-aware block designs, and results inthe best known 6-bit and 5-bit formats for training.[Figure] xxx [Figure]
Revised: This includes two techniques for minimising data loss of a practical implementation, namely gradual underﬂow and cost-aware block designs, and results in 6-bit and 5-bit formats for training.

Original: Exp

Original: [Figure]
Revised: [Figure]

Original: 5|X| minifloat (3,2) log 2 |a| |a| value distribution
Revised: 5|X| minifloat (3,2) log 2 |a| |a| value distribution

Original: Align with max.
Revised: Align with max.

Original: exponent ß ­2 ­
Revised: exponent ß ­2 ­

Original:  B LOCK M INIFLOAT R EPRESENTATION 2. M INIFLOAT N UMBER F ORMAT
Revised:  B LOCK M INIFLOAT R EPRESENTATION 2. M INIFLOAT N UMBER F ORMAT

Original: Equation (1) computes the real value of a miniﬂoat number, where ( e, m ) denote the number of exponent and mantissa bits in the representation, and s is the sign bit.
Revised: Equation (1) computes the real value of a miniﬂoat number, where ( e, m ) denote the number of exponent and mantissa bits in the representation.

Original: [Equation]
Revised: [Equation]

Original: The decimal expansions of the exponent and mantissa are both unsigned integers, given by E and F respectively, while β = 2 e − 1 is therequired exponent bias for the binary-offset encoding scheme.
Revised: The decimal expansions of the exponent and mantissa are both unsigned integers, given by E and F respectively, s refers to the sign bit and β = 2 e − 1 is the exponent bias for the binaryoffset encoding scheme.

Original: This is consistent with IEEE-754 ﬂoating point standards, except that our miniﬂoats are considerably smaller (4-8 bits only), can generalise to multiple ( e, m ) conﬁgurations, and do not handle nan/inﬁnity bit patterns.
Revised: This is consistent with IEEE-754 ﬂoating point standards, except that our miniﬂoats are considerably smaller (4-8 bits only), can generalise to multiple ( e, m ) conﬁgurations, and do not handle nan/inﬁnity bit patterns.

Original: Instead, arithmetic is allowed to saturate at the limits of the representable range [ X + min , X + max ] .
Revised: Instead, arithmetic is allowed to saturate at the limits of the representable range [ X + min , X + max ] .

Original: For example, a miniﬂoat representation with X (cid:10) 4 , 3 (cid:11) have exponent and mantissas that range between [0 , 15] and [0 , 7] respectively.
Revised: For example, a miniﬂoat representation with X (cid:10) 4 , 3 (cid:11) have exponent and mantissas that range between [0 , 15] and [0 , 7] respectively.

Original: Therefore, the largest normal and smallest denormal positive numbers are X + max = 240 and X + min = 2 − 10 .
Revised: Therefore, the largest normal and smallest denormal positive numbers are X + max = 240 and X + min = 2 − 10 .

Original: This corresponds to a dynamic range ofdB shown in Table 5 in Appendix A.1.
Revised: This corresponds to a dynamic range of 108 dB shown in Table 7 in Appendix A.1.

Original: 2. S HARED E XPONENT B IAS
Revised: 2. S HARED E XPONENT B IAS

Original: The main difference between miniﬂoat and block miniﬂoat (BM) representations are highlighted in Figure 1.
Revised: The main difference between miniﬂoat and block miniﬂoat (BM) representations are highlighted in Figure 1.

Original: Miniﬂoats have one exponent per element, but that exponent must be wide enough to tolerate changes in DNN tensor distributions during training (i.e. 5 bits for gradients in FP8 (Wang et al., 2018)).
Revised: Miniﬂoats have one exponent per element, but that exponent must be wide enough to tolerate changes in DNN tensor distributions during training (i.e. 5 bits for gradients in FP8 (Wang et al., 2018)).

Original: In contrast, BM share exponent biases across blocks of N miniﬂoat numbers.
Revised: In contrast, BM share exponent biases across blocks of N miniﬂoat numbers.

Original: This provides equivalent dynamic range with respect to the block, but with fewer exponent bits than the original miniﬂoat format.
Revised: This provides equivalent dynamic range with respect to the block, but with fewer exponent bits than the original miniﬂoat format.

Original: Block Floating Point (BFP) operates similarly, but all numbers within the block are integers (Drumond et al., 2018).
Revised: Block Floating Point (BFP) operates similarly, but all numbers within the block are integers (Drumond et al., 2018).

Original: BM can generalise for this case, i.e. when e = 0 .
Revised: BM can generalise for this case, i.e. when e = 0 .

Original: The real value of an element from BM tensor a is given in Equation (2).
Revised: The real value of the i th element from BM tensor a is given in Equation (2), where X is an unbiased miniﬂoat tensor, represented by ( e , m ) exponent and mantissa bits, and β a is the shared exponent bias.

Original: [Equation]
Revised: [Equation]

Original: In this example, a  can only be represented accurately when the  exponent bias β a (calculated for the entire tensor) and the miniﬂoat format Xa jointly capture theunderlying value distribution.
Revised: In this example, a i can only be represented accurately when the shared exponent bias β a (calculated for the entire tensor) and the distribution of X jointly captures the value distribution of a .

Original: For example, large and small values in a could saturate or be lost altogether if β a is too large or too small.
Revised: For example, large and small values in a could saturate or be lost altogether if β a is too large or too small.

Original: However, some leeway does exist when exponents are shared across dot products.
Revised: However, some leeway exists when exponents are shared across dot products.

Original: This is because dot products are reduce operations, meaning their sum is dominated by the largest values in the inputs.
Revised: This is because dot products are reduce operations, meaning their sum is dominated by the largest values in the inputs.

Original: For this reason, we calculate β  to speciﬁcally guard against overﬂow, and unlike (Cambier et al., 2020)
Revised: For this reason, we calculate β a to speciﬁcally guard against overﬂow, and unlike (Cambier et al., 2020)

Original: we don’t apply any scaling which could divert precision away from larger value regions.
Revised: we don’t apply any scaling which could divert precision away from larger value regions.

Original: Figure 2 shows an illustrative example of how βis updated during training.
Revised: Our method of updating β during training is illustrated in Figure 2 and formalized in Equation 3 below.

Revised: [Equation]

Revised: The ﬁrst term denotes the maximum exponent for the tensor a , which changes and must be updated during training, while the second term is ﬁxed and refers to the maximum exponent of X .

Original: In terms of hardware, shared biases ensure that all dot products can be computed with denser miniﬂoat arithmetic.
Revised: In terms of hardware, shared biases ensure that all dot products can be computed with denser miniﬂoat arithmetic.

Original: This is shown in Equation 3 for BM tensors a and b , each with N elements.
Revised: This is shown in Equation 4 for BM tensors a and b , each with N elements.

Original: [Equation]
Revised: [Equation]

Original: The dot product, X a · X b ,involves the miniﬂoats that have  smaller exponents, while the cost of  storing and aligning the exponent biases  is amortized over the length of the dot-product.
Revised: The dot product, X a · X b , have miniﬂoat formats with smaller exponents, while the cost of calculating, storing and aligning the exponent biases during training is amortized over the length of the dot-product.

Original: Next we show how miniﬂoat formats with fewer exponent bits lead to faster and more compact hardware.
Revised: Next we show how miniﬂoat formats with fewer exponent bits lead to faster and more compact hardware.

Original: 2. K ULISCH A CCUMULATION
Revised: 2. K ULISCH A CCUMULATION

Original:  A Kulisch accumulator (Kulisch & Miranker, 2014) is a ﬁxed point accumulator that is wide enough to compute an error free sum of scalar ﬂoating-point products, over the entire range of possible values.
Revised: [Table] A Kulisch accumulator (Kulisch & Miranker, 2014) is a ﬁxed point accumulator that is wide enough to compute an error free sum of scalar ﬂoating-point products, over the entire range of possible values.

Original: Kulisch accumulators operate by shifting the mantissa of the ﬂoating point product into an internal register according to the exponent of the product.
Revised: Kulisch accumulators operate by shifting the mantissa of the ﬂoating point product into an internal register according to the exponent of the product.

Original: The sum proceeds as integer addition which is 4 − 10 × more efﬁcient in terms of area and power compared to FP16 (Dally, 2015).
Revised: The sum proceeds as integer addition which is 4 − 10 × more efﬁcient in terms of area and power compared to FP16 (Dally, 2015).

Original: The number of bits required for the internal register (i.e. the addend) and shifter, scale the size and complexity of the accumulator and are provided as formulas in Equation (4) for BM operands a = ( e a , m a ) and b = ( e b , m b ) .
Revised: The number of bits required for the internal register (i.e. the addend) and shifter, scale the size and complexity of the accumulator and are provided as formulas in Equation (5) for BM operands a = ( e a , m a ) and b = ( e b , m b ) .

Original: [Equation]
Revised: [Equation]

Original: In the above equations, kadd calculates the number of bits required for the largest product of two numbers, plus one extra bit for the addition, and kshift is the mantissa product subtracted from kadd since the mantissa product is already aligned with the addend.
Revised: In the above equations, kadd calculates the number of bits required for the largest product of two numbers, plus one extra bit for the addition, and kshift determines the maximum number of bits the mantissa product must be shifted to align with the addend.

Original: By considering the size of kadd and kshift , BM formats can be designed to trade-off ﬁne-grained dynamic range (i.e. exponent bits) for more precision and smaller hardware.
Revised: Crucially, by considering the size of kadd and kshift , BM formats can be designed to trade-off ﬁne-grained dynamic range (i.e. exponent bits) for more precision and smaller hardware.

Revised: In fact, formats with exponents up to 4 bits may yield kadd of approximately the same size as INT8/INT32 arithmetic units, while kadd becomes prohibitively wider and more expensive for larger exponents.

Original: We discover that exponents up to 4  bits offer distinct advantages over competitive representations, and is a key contribution of this work.
Revised: This is clearly shown via example in Table 1 above, but more importantly, it is supported by hardware synthesis results given in Sectionand Appendix A.4. For example, an 8-bit miniﬂoat format having 4 exponent bits achieves a 1 . 6 × area reduction compared to HFP8 (Sun et al., 2019) with 5 exponent bits. Furthermore, through an extensive set of experiments we discover that such representations can also achieve high training accuracy which forms a key contribution of our work.

Original:  T RAINING WITH B LOCK M INIFLOAT 3. M INIMIZING D ATA L OSSBM
Revised:  T RAINING WITH B LOCK M INIFLOAT 3. M INIMIZING D ATA L OSS

Original:  arithmetic will incur data loss when the value distribution is too wide or requires more precision than can be captured by the underlying miniﬂoat representation within a block.
Revised: BM arithmetic will incur data loss when the value distribution is too wide or requires more precision than can be captured by the underlying miniﬂoat representation within a block.

Original: Below, we describe steps taken to mitigate this problem without substantially increasing implementation overheads.
Revised: Below, we describe steps taken to mitigate this problem without substantially increasing implementation overheads.

Original: Gradual underﬂow: Our miniﬂoats support denormal numbers as deﬁned in Equation (1).
Revised: Gradual underﬂow: Our miniﬂoats support denormal numbers as deﬁned in Equation (1).

Original: Denormal numbers have precision close to zero, and ensure that consecutively smaller quantized numbers approach zero gradually.
Revised: Denormal numbers have precision close to zero, and ensure that consecutively smaller quantized numbers approach zero gradually.

Original: The alternative is ﬂush-to-zero which discards the mantissa bits when
Revised: The alternative is ﬂush-to-zero which discards the mantissa bits when

Original: E = 0 .
Revised: E = 0 .

Original: This equates to approximately 12 .
Revised: This equates to approximately 12 .

Original: 5% of the exponent encoding when e = 3 ; this is highly inefﬁcient.
Revised: 5% of the exponent encoding when e = 3 ; this is highly inefﬁcient.

Original: Overhead for denormal numbers in hardware is minimal, and only requires detection of
Revised: Overhead for denormal numbers in hardware is minimal, and only requires detection of

Original: E = 0 and a single bit ﬂip in the multiplier.
Revised: E = 0 and a single bit ﬂip in the multiplier.

Original: Our experiments show that gradual underﬂow is crucial for BM formats with less than four exponent bits.
Revised: Our experiments show that gradual underﬂow is crucial for BM formats with less than four exponent bits.

Original: Block Size: Matrix multiplication is computed by dividing tensors into N × N blocks that bound the number of exponent biases and reduce data loss.

Original: Square blocks are chosen so that biases are contiguous in memory regardless of whether the block is operating in the forward path or after transposition in the backward path.

Original: As such, BM can be stored with a persistent data structure, that doesn’t require recasting or extra memory transfers during training.

Original: This makes BM easy to use at the software level, but does mean that biases are shared across N independent dot products.

Original: In terms of hardware, a ﬂoating point (FP) unit is required to convert, align and accumulate the ﬁxed point dot product result.

Original: Furthermore, Kulisch adders should be wider to prevent overﬂow, and additional memory is required to store 8 bit biases.

Original: These costs are formalized in Equation (6), where α refer to resource costs for each overhead.

Original: [Equation]

Original: For large block sizes, the extra silicon area is negligible compared to the GEMM but data loss from sharing biases can still be signiﬁcant.

Original: In Section (4), we determined that a block size of N =offers a good balance for both objectives and is used for the rest of this paper.

Original: Hybrid representation: Different miniﬂoat representations for forward and backward paths have been shown to produce better accuracy for FP8 training (Sun et al., 2019).

Original: We apply the same idea, and ﬁnd the best balance of precision and range for both paths separately.

Original: Forward and backward conﬁgurations are given by ( e f , m f ) /

Original: ( e b , m b ) notation in our experiments.

Original: x l
Revised: x l

Original: BM+Kul.
Revised: BM+Kul.

Original: GEMM
Revised: GEMM

Original: RF -/+
Revised: RF -/+

Original: FP x l+1 w t
Revised: FP x l+1 w t

Original: BM
Revised: BM

Original: BM (a) Fwd activation e l+
Revised: BM (a) Fwd activation e l+

Original: BM+Kul.
Revised: BM+Kul.

Original: GEMM
Revised: GEMM

Original: RF -/+
Revised: RF -/+

Original: FP e l w t
Revised: FP e l w t

Original: BM
Revised: BM

Original: BM (b) Bwd activation grad.
Revised: BM (b) Bwd activation grad.

Original: e l+
Revised: e l+

Original: BM+Kul.
Revised: BM+Kul.

Original: GEMM w -/+
Revised: GEMM w -/+

Original: FP x l ∆ w t w t+
Revised: FP x l ∆ w t w t+

Original: BM BM (c) Bwd weight grad.
Revised: BM BM (c) Bwd weight grad.

Original: and update
Revised: and update

Revised: Block Size: Matrix multiplication with BM is computed by dividing tensors into N × N blocks that bound the number of exponent biases and reduce data loss, since each block shares one exponent bias.

Revised: Square blocks are chosen so that biases are in memory regardless of whether the block is operating in the path or after transposition in the path.

Revised: As such can be stored with a or

Revised: This makes to at the but are

Original: 3. T RAINING D ETAILS AND GPU S IMULATION
Revised: N In terms of hardware cost Equation 7 formalizes the relationship between the size of N and three overheads, where α refer to relative area costs for each overhead. cost =

Revised: Kulisch

Revised: [Equation]

Revised: For the ﬁrst term, the width of the Kulisch accumulator must increase by log 2 N bits to prevent overﬂow in the BM dot-products.

Revised: In the second term, ﬂoating-point hardware (including quantization and conversion modules) are required to accumulate, align and convert BM partial results, but the cost is amortized over N ﬁxed-point operations.

Revised: Finally, in the last term, additional memory is required to compute and store one 8-bit bias every N × N values.

Revised: For large block sizes, the extra silicon area from Equation 7 is negligible compared to the GEMM but data loss from sharing biases can still be signiﬁcant.

Revised: In Section (4.5), we used Equation 7 and determined that a block size of N = 48 offers a good balance for both objectives and is used for the rest of this paper.

Revised: Hybrid representation: Different miniﬂoat representations for forward and backward paths have been shown to produce better accuracy for FP8 training (Sun et al., 2019).

Revised: We apply the same idea, and ﬁnd the best balance of precision and range for both paths separately.

Revised: Full details for all BM formats are provided in Table 7 (in Appendix A.1) where forward and backward conﬁgurations are given by ( e f , m f ) /

Revised: ( e b , m b ) notation.

Revised: Our formats cover each precision level between 4 andbits, and are denoted by BM4, BM5, BM6, BM7 and BM8 in our experiments.

Revised: For example, BM (2 , 3) / (3 , 2) refers to 6-bit BM training with weight and activation tensors represented by (2 , 3) and activation gradient tensors represented by (3 , 2) miniﬂoat formats.

Revised: 3. T RAINING D ETAILS AND GPU S IMULATION

Original: BM offers an alternative to standard FP32 for the computationally intensive parts of training, which is typically mapped to general matrix multiplication (GEMM).
Revised: BM offers an alternative to standard FP32 for the computationally intensive parts of training, which is typically mapped to general matrix multiplication (GEMM).

Original: However, specialised hardware is required to realise its potential gains in speed and energy efﬁciency.
Revised: However, specialised hardware is required to realise its potential gains in speed and energy efﬁciency.

Original: For the purposes of this paper, we simulate the behaviour of BM hardware using GPUs and PyTorch.
Revised: For the purposes of this paper, we simulate the behaviour of BM hardware using GPUs and PyTorch.

Original: Given that dot products are computed exactly via Kulisch accumulators, existing CUDA libraries for GEMM can be used without modiﬁcation, and all data loss is attributed to quantization only.
Revised: Given that dot products are computed exactly via Kulisch accumulators, existing CUDA libraries for GEMM can be used without modiﬁcation, and all data loss is attributed to quantization only.

Original: Figure 3 provides an illustration of the setup for each GEMM in forward and backward paths.
Revised: Figure 3 provides an illustration of the setup for each GEMM in forward and backward paths.

Original: In a practical implementation, BM does not require any costly movement or storage of high precision tensors.
Revised: In a practical implementation, BM does not require any costly movement or storage of high precision tensors.

Original: This is enabled by scalar processors after the GEMM (for FP32 operations, Kulisch to ﬂoating-point conversion, block miniﬂoat alignments, quantization etc.) and a weight update scheme that can compute and cache high precision gradients on-chip (Sun et al., 2019).
Revised: This is enabled by scalar processors after the GEMM (for FP32 operations, Kulisch to ﬂoating-point conversion, block miniﬂoat alignments, quantization etc.) and a weight update scheme that can compute and cache high precision gradients on-chip (Sun et al., 2019).

Original: Weight, activation and gradient tensors are quantized to BM numbers with stochastic rounding as described in (Wang et al., 2018).
Revised: Weight, activation and gradient tensors are quantized to BM numbers with stochastic rounding as described in (Wang et al., 2018).

Original: For the software simulation, quantization is applied before each GEMM in forward and backward paths and contributes signiﬁcant performance overhead compared to standard PyTorch layers.
Revised: For the software simulation, quantization is applied before each GEMM in forward and backward paths and contributes signiﬁcant performance overhead compared to standard PyTorch layers.

Original: An approximate 5 × slow-down is realised on most networks and datasets, with support for denormal numbers the main implementation bottleneck.
Revised: An approximate 5 × slow-down is realised on most networks and datasets, with support for denormal numbers the main implementation bottleneck.

Original: The realisation of the same function is comparatively cheap in custom hardware however, and can be fully-pipelined for fast training times
Revised: The realisation of the same function is comparatively cheap in custom hardware however, and can be fully-pipelined for fast training times

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluated the training accuracy of BM on a subset of image, language and object detection modelling tasks.
Revised: We evaluated the training accuracy of BM on a subset of image, language and object detection modelling tasks.

Original: The entire spectrum of representations were explored on ImageNet (Deng et al., 2009) and CIFAR (Krizhevsky et al., 2009) image recognition benchmarks, with results compared against well-calibrated INT8, FP8 and FP32 baselines.
Revised: The entire spectrum of representations were explored on ImageNet (Deng et al., 2009) and CIFAR (Krizhevsky et al., 2009) image recognition benchmarks, with results compared against well-calibrated INT8, FP8 and FP32 baselines.

Original: On other tasks, BM8 is compared with an
Revised: On other tasks, BM8 is compared with an

Original: FP32 baseline.
Revised: FP32 baseline.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: with base √

Original: 4. CIFAR-10 AND CIFAR- 
Revised: 4. CIFAR-10 AND CIFAR-

Original: We ran CIFAR experiments using SGD with momentum of 0.9 for 200 epochs in batches ofimages and initial learning rate of 0.1 which is decayed by a factor of 5 at the 60th, 120th and 160th epochs.
Revised: We ran CIFAR experiments using SGD with momentum of 0.9 for 200 epochs in batches ofimages and initial learning rate of 0.1 which is decayed by a factor of 5 at the 60th, 120th and 160th epochs.

Original: Table 1 presents results for training [-ResNet-
-] (He et al., 2016) with only small  BM  representations.
Revised: Table 2 presents results for training [-ResNet-
-]ResNet-18 (He et al., 2016) with only small BM6, BM and BM4 representations.

Original: These offer the highest reduction in memory usage while still reaching very close to the FP32 baseline.
Revised: These offer the highest reduction in memory usage while still reaching very close to the FP32 baseline.

Original: For example, 6-bit BM training only records a 0.3% loss in accuracy compared to FP32 on CIFAR-100 while theoretically saving 25% of memory read and write overheads compared to FP8.
Revised: For example, 6-bit BM training only records a 0.3% loss in accuracy compared to FP32 on CIFAR-100 while theoretically saving 25% of memory read and write overheads compared to FP8.

Original: We also tested logarithmic BM formats on CIFAR-10 and VGG16 network.
Revised: We also tested logarithmic BM formats on CIFAR-10 and VGG16 network.

Original: Log representations arise when m = 0 , and require only adds and shifts for multiply-add arithmetic.
Revised: Log representations arise when m = 0 , and require only adds and shifts for multiply-add arithmetic.

Original: Our results use the same training parameters as before and are shown in Table 2.
Revised: Our results use the same training parameters as before and are shown in Table 3.

Original: We compare against the only previously known result for log training, i [Table] e. log-5b (Miyashita et al., 2016) and achieve similar results with respect to FP32 for 5 bit and 4 bit.
Revised: We compare against the only previously known result for log training, i.e. log-5b (Miyashita et al., 2016) and achieve similar results with respect to FP32 for 5 bit and 4 bit.

Original: BM representations have exponent biases that shifts tensor distributions dynamically during training, whereas log-5b deﬁne offset parameters at each layer that are ﬁxed.
Revised: BM representations have exponent biases that shifts tensor distributions dynamically during training, whereas log-5b deﬁne offset parameters at each layer that are ﬁxed.

Original: Allowing biases to vary during training gives BM an advantage, and results in similar validation accuracy with only 4 bit words.
Revised: Allowing biases to vary during training gives BM an advantage, and results in similar validation accuracy with only 4 bit words.

Original: This corresponds to approximately half the cost for multiplication in the linear domain (by exponent add and Kulisch shift)
Revised: This corresponds to approximately half the cost for multiplication in the linear domain (by exponent add and Kulisch shift).

Original: [Table]
Revised: [Table]

Revised: S2FP8 (

Original: [Table]
Revised: [Table]

Revised: et al.,

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: HFP8 (Sun et al., 2019) (4 , 3) (4 , 3) (5 , 10) (5 , 2) (5 , 10) 69.

Revised: BM8 (2,5)/(4,3) (2 , 5) (2 , 5) (6 , 9) (4 , 3) 3169.

Revised: BM7 (2,4)/(4,2) (2 , 4) (2 , 4) (6 , 9) (4 , 2) 2969.

Revised: BM6 (2,3)/(3,2) (2 , 3) (2 , 3) (6 , 9) (3 , 2) 2069.

Revised: BM5 (2,2)/(3,1) (2 , 2) (2 , 2) (6 , 9) (3 , 1) 1869.

Revised: 1 Fixed point

Original: 4. I MAGE N ET
Revised: 4. I MAGE N ET

Original: The ImageNet dataset has 1000 class labels, and consists of 256x256 images split into a training set with 1.28 million images and validation set with 50,000 images.
Revised: The ImageNet dataset has 1000 class labels, and consists of 256x256 images split into a training set with 1.28 million images and validation set with 50,000 images.

Original: We use ResNet-18 (He et al., 2016) and AlexNet (Krizhevsky et al., 2012)
Revised: We use ResNet-18 (He et al., 2016) and AlexNet (Krizhevsky et al., 2012)

Original: architectures from the ofﬁcial PyTorch implementation 1 , and train on one GPU with standard settings; SGD with momentum of 0.9, batches of 256 images, and an initial learning rate of 0.1 (0.01 for AlexNet) which is decayed by a factor of 10 at epoch 30 and 60.
Revised: architectures from the ofﬁcial PyTorch implementation 1 , and train on one GPU with standard settings; SGD with momentum of 0.9, batches of 256 images, and an initial learning rate of 0.1 (0.01 for AlexNet) which is decayed by a factor of 10 at epoch 30 and 60.

Original: ResNet-18 has been widely tested upon in previous work, and offers the most suitable benchmark for exploring the full spectrum of BM representations, especially given the size of the network as well as the cost of BM quantization on training times (approx.
Revised: ResNet-18 has been widely tested upon in previous work, and offers the most suitable benchmark for exploring the full spectrum of BM representations, especially given the size of the network as well as the cost of BM quantization on training times (approx.

Original: 5 × slow-down).
Revised: 5 × slow-down).

Original: Results are presented in Table 3.
Revised: Results are presented in Table 4 where columns w, x, dw, dx and acc refer to the numerical representation for weight, activation, weight gradient, activation gradient and on-chip GEMM accumulator.

Original: We achieve FP32 equivalent accuracy for BM8 and BM7, slight degradation for BM6, while our BM5 exceeds the reported accuracy for 8 bit SWALP (Yang et al., 2019).
Revised: We achieve FP32 equivalent accuracy for BM8 and BM7, slight degradation for BM6, while our BM5 exceeds the reported accuracy for 8 bit SWALP (Yang et al., 2019).

Original: Compared to

Original:  S2FP8 (Cambier et al., 2020), our BM6 representation reaches similar levels of relative accuracy, but with two fewer bits and without a high precision master copy of the weights.
Revised: Compared to S2FP8 (Cambier et al., 2020), our BM6 representation reaches similar levels of relative accuracy, but with two fewer bits and without a high precision master copy of the weights.

Original: We provide some insights into possible reasons for this in Section 4.5 by considering the possibility of diminishing returns in accuracy from scaling miniﬂoat representations.
Revised: We provide some insights into possible reasons for this in Section 4.5 by considering the possibility of diminishing returns in accuracy from scaling miniﬂoat representations.

Original: Compared with HFP8 (Sun et al., 2019), which offersthe best and most robust 8-bit training results, BM8 produces the same accuracy on ImageNet while improving upon HFP8 in hardware density and performance.
Revised: Compared with HFP8 (Sun et al., 2019), which offers robust 8-bit training results, BM8 produces the same accuracy on ImageNet while improving upon HFP8 in hardware density and performance.

Original: BM8 tensors can be represented with fewer exponent bits, and thus perform dot products via Kulisch accumulators that are smaller and faster than FP16 units.
Revised: BM8 tensors can be represented with fewer exponent bits, and thus perform dot products via Kulisch accumulators that are smaller and faster than FP16 units.

Original: Furthermore, BM offers tradeoffs for even denser arithmetic and lower memory usage.
Revised: Furthermore, BM offers tradeoffs for even denser arithmetic and lower memory usage.

Original: In these regimes, BM hardware is more comparable to SWALP (Yang et al., 2019) which performs the GEMM in ﬁxed point.
Revised: In these regimes, BM hardware is more comparable to SWALP (Yang et al., 2019) which performs the GEMM in ﬁxed point.

Original: These design points are explored more inFigure 6, Section 5. 4. L ANGUAGE M ODELLING WITH LSTM
Revised: Proof of BM design efﬁciencies are provided with RTL synthesis results in Section 5, Figure 6.

Original: We compared 8-bit formats for language modeling on the Penn Treebank dataset Marcus et al.

Original: We adapted the 2-layer Long Short Term Memory (LSTM) network from PyTorch Examples 2 and perform all GEMM operations with BM8 arithmetic.

Original: The batch size is 20, initial learning rate iswith 0.25 decay, the embedding and hidden dimensions are 650 and sequence length is 35.

Original: Results in Figure 4 show BM8 with (2 , 5) / (4 , 3) hybrid conﬁguration achieving better accuracy than BFP and HFP8 variants.

Original: The proposed BM8 representation has more ﬁne-grained dynamic range and fewer mantissa bits than BFP8, and more precision and fewer exponent bits than HFP8 formats.

Original: Model (Dataset)
Revised: Model (Dataset)

Original: [Metric]
Revised: [Metric]

Original: [Table]
Revised: [Figure]

Original: AlexNet (ImageNet)
Revised: AlexNet (ImageNet)

Original: [Table]
Revised: [Figure]

Original: EfﬁcientNet-b [Table] small ImageNet)
Revised: EfﬁcientNet-b0 (small ImageNet)

Original: [Table]
Revised: [Figure]

Original: LSTM (PTB)[Val ppl.]
Revised: LSTM (PTB)[Val ppl.]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Transformer-base 
Revised: Transformer-base (IWSLT)

Original: [Table]
Revised: [Table]

Revised: BLEU]

Original: [Table]
Revised: [Table]

Original: SSD
Revised: SSD-Lite

Original: [Table]
Revised: [Table]

Original: Language
Revised: [Table] [Figure] 4. L ANGUAGE M ODELLING WITH LSTM We compared 8-bit formats for language modeling on the Penn Treebank dataset Marcus et al.

Original: [Figure]
Revised: We adapted the 2-layer Long Short Term Memory (LSTM) network from PyTorch Examples 2 and perform all GEMM operations with BM8 arithmetic. The batch size is 20, initial learning rate iswith 0.25 decay, the embedding and hidden dimensions are 650 and sequence length is 35. Results in Figure 4 show BM8 with (2 , 5) / (4 , 3) hybrid conﬁguration achieving better accuracy than BFP and HFP8 variants. The proposed BM8 representation has more ﬁne-grained dynamic range and fewer mantissa bits than BFP8, and more precision and fewer exponent bits than HFP8 formats.

Original: This design point achieves better outcomes in terms of accuracy and hardware density than either representation separately (see Figure 6, Section 5).
Revised: This design point achieves better outcomes in terms of accuracy and hardware density than either representation separately (see Figure 6, Section 5).

Original: Validation perplexity of 87.33 is also comparable to 84.70 obtained with full-precision ﬂoating-point.
Revised: Validation perplexity of 87.33 is also comparable to 84.70 obtained with full-precision ﬂoating-point.

Original: 4. A DDITIONAL E XPERIMENTS
Revised: 4. A DDITIONAL E XPERIMENTS

Original: To demonstrate wider applicability of the BM number representation, we tested BM8 on several additional networks and modelling tasks.
Revised: To demonstrate wider applicability of the BM number representation, we tested BM8 on several additional networks and modelling tasks.

Original: Results are summarized in Table 4 with full details of each experiment provided in Appendix A.3.
Revised: Results are summarized in Table 5 with full details of each experiment provided in Appendix A.3.

Original: Crucially, every network tested achieved comparable accuracy with baseline FP32.
Revised: Crucially, every network tested achieved comparable accuracy with baseline FP32.

Original: This includes EfﬁcientNet-b0 (Tan & Le, 2019) image classiﬁcation and SSD-lite (Liu et al., 2016) with MobileNet-V2 object detection models, both of which represent the type of network and application well suited to resource constrained hardware devices.
Revised: This includes EfﬁcientNet-b0 (Tan & Le, 2019) image classiﬁcation and SSDlite (Liu et al., 2016) with MobileNet-V2 object detection models, both of which represent the type of network and application well suited to resource constrained hardware devices.

Original: Furthermore, we also trained a small Transformer network for translation on the IWSLT German to English dataset (Cettolo et al., 2014).
Revised: Furthermore, we also trained a small Transformer network for translation on the IWSLT German to English dataset (Cettolo et al., 2014).

Original: In future work, we plan to scale our implementation and demonstrate training with BM representations on larger networks and datasets.
Revised: In future work, we plan to scale our implementation and demonstrate training with BM representations on larger networks and datasets.

Original: Network design with BM is another interesting research direction, since the network architecture (including topology, layer type, shape and connectivity) of each of the tested networks may have some bias since they were initially designed to optimize anobjective with FP32 arithmetic. [Figure] [Figure]
Revised: Network design with BM is another interesting research direction, since the majority of network architectures have been designed and optimized while assuming an FP32 arithmetic scheme.

Original: [Figure]
Revised: [Figure]

Original:  E MPIRICAL A NALYSIS
Revised: 4. E MPIRICAL A NALYSIS

Original: Effect of Denormal Numbers: To study the effect that denormal numbers have on training convergence in sub 8-bit networks, we trained ResNet-18 on ImageNet for BM6 with denormals (ours) and without denormals, using QPyTorch library (Zhang et al., 2019).
Revised: Effect of Denormal Numbers: To study the effect that denormal numbers have on training convergence in sub 8-bit networks, we trained ResNet-18 on ImageNet for BM6 with denormals (ours) and without denormals, using QPyTorch library (Zhang et al., 2019).

Original: Results are plotted against ﬂoating point accuracy in Figure 5a.
Revised: Results are plotted against ﬂoating point accuracy in Figure 5a.

Original: Without denormals, small numbers are ﬂushed-to-zero and training stagnates immediately.
Revised: Without denormals, small numbers are ﬂushed-to-zero and training stagnates immediately.

Original: Although not shown here, 8-bit representations with more than e =bits do not suffer similar accuracy degradation without denormals.
Revised: Although not shown here, 8-bit representations with more than e =bits do not suffer similar accuracy degradation without denormals.

Original: This investigation conﬁrms the importance of denormal numbers for training BM formats with fewer exponent bits, and differentiates our software and hardware experiments substantially from previous 8-bit regimes.
Revised: This investigation conﬁrms the importance of denormal numbers for training BM formats with fewer exponent bits, and differentiates our software and hardware experiments substantially from previous 8-bit regimes.

Original: Selecting the Block Size: We conducted experiments on CIFAR100 to determine suitable block sizes - those which simultaneously increase dynamic range and have low hardware overhead.
Revised: Selecting the Block Size: We conducted experiments on CIFAR100 to determine suitable block sizes - those which simultaneously increase dynamic range and have low hardware overhead.

Original: Results are shown in Figure 5b.
Revised: Results are shown in Figure 5b.

Original: We took the average of the largest range observed in gradient tensors at different block settings, over the entire duration of training.
Revised: We took the average of the largest range observed in gradient tensors at different block settings, over the entire duration of training.

Original: Estimates of area come from Equationwith parameters; α 1 = 1 , α 2 = 10 (relative area of ﬁxed-point and ﬂoating-point respectively), kadd = 21 and α
Revised: Estimates of area come from Equationwith parameters; α 1 = 1 , α 2 = 10 (relative area of ﬁxed-point and ﬂoating-point respectively), kadd = 21 and α

Original: 3 = 0 .
Revised: 3 = 0 .

Original: We saturate the area cost at N = 256 , which is consistent with the length of dot-products supported by the GEMM architecture in TPU hardware (Jouppi et al., 2017).
Revised: We saturate the area cost at N = 256 , which is consistent with the length of dot-products supported by the GEMM architecture in TPU hardware (Jouppi et al., 2017).

Original: Finally, N = 48 emerged as a good selection, corresponding to one ﬂoating-point unit everymultiply-accumulate operations and one 8-bit exponent bias every 2304 miniﬂoat numbers.
Revised: Finally, N = 48 emerged as a good selection, corresponding to one ﬂoating-point unit everymultiply-accumulate operations and one 8-bit exponent bias every 2304 miniﬂoat numbers.

Original: Scaling the Miniﬂoat Representation:
Revised: Scaling the Miniﬂoat Representation:

Original: In Figure 2, which was discussed previously, miniﬂoats have exponent biases that shift the representation to align with the maximum of the underlying value distribution.
Revised: In Figure 2, which was discussed previously, miniﬂoats have exponent biases that shift the representation to align with the maximum of the underlying value distribution.

Original: Additionally, the miniﬂoat representation could be scaled (or stretched) over a wider or narrower part of the value distribution.
Revised: Additionally, the miniﬂoat representation could be scaled (or stretched) over a wider or narrower part of the value distribution.

Original: We investigate this effect by varying the base of the exponent, and inspecting the underﬂow and root mean square error (rmse) of dot-products after quantization; results are shown in Figure 5c.
Revised: We investigate this effect by varying the base of the exponent, and inspecting the underﬂow and root mean square error (rmse) of dot-products after quantization; results are shown in Figure 5c.

Original: The tensor under test is a gradient tensor with maximum exponent of -17 and mean exponent of -21.
Revised: The tensor under test is a gradient tensor with maximum exponent of -17 and mean exponent of -21.

Original: Mean scaling was proposed in S2FP8 (Cambier et al., 2020) for 8-bit training and works by centering the miniﬂoat over the mean of the exponent value distribution.
Revised: Mean scaling was proposed in S2FP8 (Cambier et al., 2020) for 8-bit training and works by centering the miniﬂoat over the mean of the exponent value distribution.

Original: For the (3,2) format, mean scaling requires a base of 2.52, calculated as b = 2 − 17+217 − 4 .
Revised: For the (3,2) format, mean scaling requires a base of 2.52, calculated as b = 2 − 17+217 − 4 .

Original: This is akin to redirecting precision from high value regions into smaller underﬂow regions, the result of which observably leads to increased error in the tested 6-bit regime.
Revised: This is akin to redirecting precision from high value regions into smaller underﬂow regions, the result of which observably leads to increased error in the tested 6-bit regime.

Original: Better approaches could be designed to detect underﬂow and use higher precision arithmetic where necessary. 
Revised: Better approaches could be designed to detect underﬂow and use higher precision arithmetic where necessary. [Table] [Table]

Original:  H ARDWARE E VALUATION
Revised:  H ARDWARE E VALUATION

Revised: In this section

Original: [Figure]
Revised: [Table]

Original: In this section, we evaluate the proposed block miniﬂoat representation in hardware and compare against competitive integer and ﬂoating-point arithmetic.
Revised: we evaluate the proposed block miniﬂoat representation in hardware and compare against competitive integer and ﬂoating-point arithmetic.

Original: Figure 6 summarizes our results with a plot of computational density (measured as operations per unit silicon area) and ResNet18 training accuracy on ImageNet.
Revised: Figure 6 summarizes our results with a plot of computational density (measured as operations per unit silicon area) and ResNet-18 training accuracy on ImageNet.

Original: Computational density was obtained from an RTL design of single-cycle fused multiply-add (FMA) units and 4x4 systolic array multipliers.
Revised: Computational density was obtained from an RTL design of single-cycle fused multiply-add (FMA) units and 4x4 systolic array multipliers.

Original: We performed synthesis at 750MHz for 28nm silicon technology and recorded area and power measurements for each number representation.
Revised: We performed synthesis at 750MHz for 28nm silicon technology and recorded area and power measurements for each number representation.

Original: Full coverage of our results are tabulated in Appendix A.4.
Revised: Table 6 provides a subset of these results, with coverage of all BM formats supplied in Appendix A. [Table]

Original: In summary, BM8 and BM6 arithmetic units are 2 .
Revised: In summary, BM8 and BM6 arithmetic units are 2 .

Original: 1 × (12 . 2 × ) and 4 .
Revised: 1 × (12 . 2 × ) and 4 .

Original: 1 × (23 . 9 × ) smaller and consume 1 .
Revised: 1 × (23 . 9 × ) smaller and consume 1 .

Original: 25 × (8 . 8 × ) and 2 .
Revised: 25 × (8 . 8 × ) and 2 .

Original: 3 × (16 . 1 × ) less power than competitive FP8/(FP32) representations.
Revised: 3 × (16 . 1 × ) less power than competitive FP8/(FP32) representations.

Original: Such arithmetic, which has similar hardware complexity to INT8, may be especially useful in embedded applications where there are stricter area and power constraints but training still needs to achieve normal levels of accuracy and relatively high performance.
Revised: Such arithmetic, which has similar hardware complexity to INT8, may be especially useful in embedded applications where there are stricter area and power constraints but training still needs to achieve normal levels of accuracy and relatively high performance.

Original: With high computational density, BM arithmetic can achieve higher training throughput on compute intensive problems, while sub 8-bit BM formats have lower bandwidth requirements leading to faster training times in memory bound applications.
Revised: With high computational density, BM arithmetic can achieve higher training throughput on compute intensive problems, while sub 8-bit BM formats have lower bandwidth requirements leading to faster training times in memory bound applications.

Original: Finally,  conversion fromthe ﬁxed point Kulisch representation into ﬂoating point requires very little  area and does not incur any performance overhead.
Revised: Finally, overheads related to conversion from Kulisch to ﬂoating point and BM quantization are expected to contribute little logic area relative to GEMM.

Revised: This includes modules for leading-one detection, barrel shifter, maximum exponent calculation, pre-quantization buffering and stochastic rounding, each of which have an efﬁcient implementation.

Revised: Further support of these claims and other system-level effects are the subject of future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: A new representation called Block Miniﬂoat (BM) was presented for training DNNs effectively with reduced precision.
Revised: A new representation called Block Miniﬂoat (BM) was presented for training DNNs effectively with reduced precision.

Original: Our representation allows the implicit exponent bias within IEEEﬂoating-point speciﬁcations to vary for a block of numbers, and can be trained with high accuracy using narrow exponent encodings.
Revised: Our representation allows the implicit exponent bias within IEEEﬂoating-point speciﬁcations to vary for a block of numbers, and can be trained with high accuracy using narrow exponent encodings.

Original: We describe how few exponent bits lead to signiﬁcantly smaller hardware, while smaller representations reduce memory bandwidth requirements, leading to faster training than previous 8-bit approaches.
Revised: We describe how few exponent bits lead to signiﬁcantly smaller hardware, while smaller representations reduce memory bandwidth requirements, leading to faster training than previous 8-bit approaches.

Original:  LOCK 
Revised: B LOCK M INIFLOAT N UMBER F ORMATS

Original: M INIFLOAT N UMBER F ORMATS

Original: The full spectrum of Block Miniﬂoat (BM) formats are presented in Table 5.
Revised: The full spectrum of Block Miniﬂoat (BM) formats are presented in Table 7.

Original: BM formats are designed with consideration for the hardware cost of the Kulisch accumulator and also dynamic range and precision requirements.
Revised: BM formats are designed with consideration for the hardware cost of the Kulisch accumulator and also dynamic range and precision requirements.

Original: Integer-like hardware is realised by limiting the value of kadd to approximately 32 bits (to match INT32).
Revised: Based on hardware synthesis results (in Table 11) integer-like hardware is realised by limiting the value of kadd to approximately 32 bits (to match INT32).

Original: As observed in Table
Revised: As observed in Table

Original: [Table]
Revised: [Table]

Original: this translates into BM formats consisting of no more than 4 exponent bits.
Revised: this translates into BM formats consisting of up to 4 exponent bits.

Original: Training with such representations, that have comparatively low dynamic range, is a key contribution of this work
Revised: Training with such representations, that have comparatively low dynamic range, is a key contribution of this work

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. S OFTWARE I MPLEMENTATION D ETAILS
Revised: A. S OFTWARE I MPLEMENTATION D ETAILS

Original: Block Miniﬂoat (BM) arithmetic requires custom hardware to achieve gains in speed and energy efﬁciency.
Revised: Block Miniﬂoat (BM) arithmetic requires custom hardware to achieve gains in speed and energy efﬁciency.

Original: We use QPyTorch (Zhang et al., 2019), an open source framework for low-precision training, to simulate the behaviour of BM hardware with existing PyTorch and CUDA libraries.
Revised: We use QPyTorch (Zhang et al., 2019), an open source framework for low-precision training, to simulate the behaviour of BM hardware with existing PyTorch and CUDA libraries.

Original: QPyTorch provides a simple interface for applying quantization in the forward path (weight and activation tensors) and backward path (error, gradient and momentum tensors), ensuring that all numbers have a low-precision representation, while the actual GEMM and AXPY operations are computed in single-precision ﬂoating-point (FP32).
Revised: QPyTorch provides a simple interface for applying quantization in the forward path (weight and activation tensors) and backward path (error, gradient and momentum tensors), ensuring that all numbers have a low-precision representation [Table] while the actual GEMM and AXPY operations are computed in single-precision ﬂoating-point (FP32).

Original: This last point means that QPyTorch can not ordinarliy be used to research low-precision accumulation strategies.
Revised: This last point means that QPyTorch can not ordinarliy be used to research low-precision accumulation strategies.

Original: Our work is different.
Revised: Our work is different.

Original: Kulisch accumulators (as described in Section 2) compute exact dot-products, and therefore our
Revised: Kulisch accumulators (as described in Section 2) compute exact dot-products [Table] and therefore our

Original: GEMMs are adequately approximated by FP32 arithmetic (which is close to exact).
Revised: GEMMs are adequately approximated by FP32 arithmetic (which is close to exact).

Original: QPyTorch is avalaible online 3 and supports ﬂoating-point (without denormals), ﬁxed-point and block ﬂoating point number formats.
Revised: QPyTorch is avalaible online 3 and supports ﬂoating-point (without denormals), ﬁxed-point and block ﬂoating point number formats.

Original: Our code implementation extends this package with the addition of a parameterised
Revised: Our code implementation extends this package with the addition of a parameterised

Original: BlockMinifloat class, which supports denormals and allows the user to specify any conﬁguration for exponent, mantissa and  size.
Revised: BlockMinifloat class, which supports denormals and allows the user to specify any conﬁguration for exponent, mantissa and block size.

Original: A. M ODEL D ETAILS AND A DDITIONAL E XPERIMENTS
Revised: A. M ODEL D ETAILS AND A DDITIONAL E XPERIMENTS

Original: A.3.
Revised: A.3.

Original: B LOCK M INIFLOAT C ONVERGENCE C URVES ON
Revised: B LOCK M INIFLOAT C ONVERGENCE C URVES ON

Original: I MAGE N
Revised: I MAGE N

Original: ET
Revised: ET

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C OMPARISON WITH

Original: As described in Section 1, block miniﬂoats bridge the gap between narrow ﬂoating-point and block ﬂoating point (BFP) representations.
Revised: As described in Section 1, block miniﬂoats bridge the gap between narrow ﬂoating-point and block ﬂoating point (BFP) representations.

Original: The main idea is that better outcomes in terms of accuracy and hardware efﬁciency can be achieved by exploring the spectrum between the two representations.
Revised: The main idea is that better outcomes in terms of accuracy and hardware efﬁciency can be achieved by exploring the spectrum between the two representations.

Original: While BFP ensures that the majority of computation involves dense integer arithmetic, the lack of ﬁne-grained dynamic range typically leads to accuracy loss on larger models and more complex datasets.
Revised: While BFP ensures that the majority of computation involves dense integer arithmetic, the lack of ﬁne-grained dynamic range typically leads to accuracy loss on larger models and more complex datasets.

Original: In Table 6 (below), we show that BM recovers accuracy loss for 6-bit and 8-bit formats on
Revised: In Table 8 (below), we show that BM recovers accuracy loss for 6-bit and 8-bit formats on

Original: ImageNet while maintaining the same advantages in hardware.
Revised: ImageNet while maintaining the same advantages in hardware.

Original: Scheme

Original: [Table]
Revised: [Table]

Original: -bit 69.
Revised: -bit 69.

Original: +0.
Revised: +0.

Original: A.3. T RANSFORMER M ODEL (WMT)
Revised: A.3. T RANSFORMER M ODEL (WMT)

Original: We trained the Transformer Base model from the FairSeq 4 repository on the IWSLT’14 German to English translation task.
Revised: We trained the Transformer Base model from the FairSeq 4 repository on the IWSLT’14 German to English translation task.

Original: We used Adam optimizer and modiﬁed the FairSeq implementation with BM8 quantization.
Revised: We used Adam optimizer and modiﬁed the FairSeq implementation with BM8 quantization.

Original: We used default training parameters found in the repository and trained for 25 epochs.
Revised: We used default training parameters found in the repository and trained for 25 epochs.

Original: BLEU scores were calculated using the script from the repository and show similar convergence between BM8 and FP32 models.
Revised: BLEU scores were calculated using the script from the repository and show similar convergence between BM8 and FP32 models.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We adapted a PyTorch implementation of SSD-lite from an online repository 5 .
Revised: We adapted a PyTorch implementation of SSD-lite from an online repository 5 .

Original: The base network is MobileNet-V2 (Sandler et al., 2018) which was pretrained on ImageNet.
Revised: The base network is MobileNet-V2 (Sandler et al., 2018) which was pretrained on ImageNet.

Original: The enitre network is trained on VOC2012 and VOC2007 trainval datasets and evaluated on VOC2007 validation dataset.
Revised: The enitre network is trained on VOC2012 and VOC2007 trainval datasets and evaluated on VOC2007 validation dataset.

Original: We apply BM8 quantization to all weights, activations and gradients before GEMM computations in the forward and backward paths.
Revised: We apply BM8 quantization to all weights, activations and gradients before GEMM computations in the forward and backward paths.

Original: The network was trained with default parameter settings provided in the repository as follows:
Revised: The network was trained with default parameter settings provided in the repository as follows:

Original: SGD with momentum of 0.9, weight decay factor 0.0005, batches ofimages, and cosine annealing ( t max = 200 ) with an initial learning rate of 0.01.
Revised: SGD with momentum of 0.9, weight decay factor 0.0005, batches ofimages, and cosine annealing ( t max = 200 ) with an initial learning rate of 0.01.

Original: After 200 epochs,
Revised: After 200 epochs,

Original: BM8 achieves a mAP of 68.0 which is sufﬁciently close to the reported accuracy of 68.6.
Revised: BM8 achieves a mAP of 68.0 which is sufﬁciently close to the reported accuracy of 68.6.

Original: A.3. E FFICIENT N ET - B 6 (I MAGE N ET )
Revised: A.3. E FFICIENT N ET - B 6 (I MAGE N ET )

Original: We adapted a PyTorch implementation of EfﬁcientNet (Tan & Le, 2019) from an online repository 6 .
Revised: We adapted a PyTorch implementation of EfﬁcientNet (Tan & Le, 2019) from an online repository 6 .

Original: We trained the smallest EfﬁcientNet-b0 network on a reduced sized ImageNet dataset, where the images are resized from 256x256 to 128x128.
Revised: We trained the smallest EfﬁcientNet-b0 network on a reduced sized ImageNet dataset, where the images are resized from 256x256 to 128x128.

Original: This choice was made to accelerate the training time, which is slowed down by 5 × with our BM8 quantization function.
Revised: This choice was made to accelerate the training time, which is slowed down by 5 × with our BM8 quantization function.

Original: The network is trained on one
Revised: The network is trained on one

Original: GPU for only 60 epochs using batch size 256 and an initial learning rate of 0.1 which is decayed exponentially with gamma of 0.90387.
Revised: GPU for only 60 epochs using batch size 256 and an initial learning rate of 0.1 which is decayed exponentially with gamma of 0.90387.

Original: Figure 9 shows convergence of BM8 with an FP32 baseline.
Revised: Figure 9 shows convergence of BM8 with an FP32 baseline.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. H ARDWARE S YNTHESIS
Revised: A. H ARDWARE S YNTHESIS

Original: Fused multply-add (FMA) units were designed in RTL for ﬂoating-point and block miniﬂoat representations.
Revised: Fused multply-add (FMA) units were designed in RTL for ﬂoating-point and block miniﬂoat representations.

Original: We modiﬁed code from the Deepﬂoat 7 repository for FP32, FP16 and FP8 units.
Revised: We modiﬁed code from the Deepﬂoat 7 repository for FP32, FP16 and FP8 units.

Original: The
Revised: The

Original: BM units with Kulisch accumulation were hand written in Verilog following the block design given in Figure 10.
Revised: BM units with Kulisch accumulation were hand written in Verilog following the block design given in Figure 10.

Original: All designs were synthesised at 750Mhz using Cadence RTL compiler 14.11 and a commercial 28nm standard cell library.
Revised: All designs were synthesised at 750Mhz using Cadence RTL compiler 14.11 and a commercial 28nm standard cell library.

Original: Since GEMM hardware is typically designed from tiles of smaller computational units, we also provide synthesis results for small 4x4 systolic array multipliers.
Revised: Since GEMM hardware is typically designed from tiles of smaller computational units, we also provide synthesis results for small 4x4 systolic array multipliers.

Original: Full coverage of our results are shown in Table 7 and Table 8.
Revised: Full coverage of our results are shown in Table 9 and Table 10.

Original: Table 9 and Table 10 are also provided, and show the hierarchical breakdowns of  BM8 and BM6 units.
Revised: Table 11 is also provided, and shows the component breakdown and scaling of Kulisch related costs in different 8-bit BM regimes. Given that BM relies on hybrid formats, the multiplier operands are both sized for the largest mantissa plus one bit for denormal support. Compared to (4,3)/(5,2), which is the format used in HFP8 (Sun et al., 2019), BM8 (2,5)/(4,3) is 1 .

Original:  This provides more insight into the scaling of  Kulisch related costs between 6-bit and 8-bit regimes.
Revised: 6 × smaller. This is because BM8 has narrower exponent encodings that reduces the width of the Kulisch accumulator.

Original: Finally, the overhead for converting from Kulisch to FP32 is very small.
Revised: Finally, the overhead for converting from Kulisch to FP32 is relatively small.

Original: We synthesised this to only contribute 264 µm 2 in area for 32-bit accumulators, the cost of which is amortized over the length of the dot product.
Revised: We synthesised this to only contribute 264 µm 2 in area for 32-bit accumulators, the cost of which is amortized over the length of the dot product.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table] [Table] [Figure]
Revised: AB

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]