Original: A D YNAMIC A PPROACH TO 
Revised: D YNAMICALLY A DAPTING F LOATING -P OINT P RECISION TO A

Original: A CCELERATE
Revised: CCELERATE

Original: D EEP
Revised: D EEP

Revised: N EURAL

Original: L EARNING T RAINING
Revised: N ETWORK T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Mixed-precision arithmetic combining both single- and half-precision operands on the same operation has been successfully applied to train deep neural networks.
Revised: Mixed-precision arithmetic combining both single- and half-precision operands on the same operation has been successfully applied to train deep neural networks.

Original: Despite the advantages of mixed-precision arithmetic in terms of reducing the need for key resources like memory bandwidth or register ﬁle size it has a limited capacity for diminishing computing costs and requires 32-bits to represent its output operands
Revised: Despite the advantages of mixed-precision arithmetic in terms of reducing the need for key resources like memory bandwidth or register ﬁle size, it has a limited capacity for diminishing computing costs and requires 32-bits to represent its output operands.

Original: This paper proposes two approaches to replace mixed precision for half-precision arithmetic during a large portion of the training
Revised: This paper proposes two approaches to replace mixed-precision for half-precision arithmetic during a large portion of the training.

Original: The ﬁrst ap- proach achieves accuracy ratios slightly slower than the state-of-the-art by using half-precision arithmetic during more than 99% of training.
Revised: The ﬁrst approach achieves accuracy ratios slightly slower than the state-of-the-art by using half-precision arithmetic during more than 99% of training.

Original: The second approach reaches the same accuracy as the state- of -the-art by dynamically switching be- tween half- and mixed precision arithmetic during training
Revised: The second approach reaches the same accuracy as the state-of-the-art by dynamically switching between half- and mixed-precision arithmetic during training.

Original: It uses half precision during more than 94% of the training process.
Revised: It uses half-precision during more than 94% of the training process.

Original: This paper is the ﬁrst in demonstrating that half-precision can be used for a very large portion of DNNs training and still reach state-of-the-art accuracy
Revised: This paper is the ﬁrst in demonstrating that half-precision can be used for a very large portion of DNNs training and still reach state-of-the-art accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The use of Deep Neural Networks (DNNs) is becoming ubiquitous in areas like computer vision (Krizhevsky et al., 2012; Szegedy et al., 2015), speech recognition (Hinton et al., 2012), or language translation (Wu et al., 2016).
Revised: The use of Deep Neural Networks (DNNs) is becoming ubiquitous in areas like computer vision (Krizhevsky et al., 2012; Szegedy et al., 2015), speech recognition (Hinton et al., 2012), or language translation (Wu et al., 2016).

Original: DNNs display very remarkable pattern detection capacities and, more speciﬁcally, Convolutional Neural Networks (CNNs) are able to accurately detect and classify objects over very large image sets (Krizhevsky et al., 2012).
Revised: DNNs display very remarkable pattern detection capacities and, more speciﬁcally, Convolutional Neural Networks (CNNs) are able to accurately detect and classify objects over very large image sets (Krizhevsky et al., 2012).

Original: Despite this success, a large amount of samples must be exposed to the model for tens or even hundreds of times during training until an acceptable accuracy threshold is reached, which drives up training costs in terms of resources like memory storage or computing time.
Revised: Despite this success, a large amount of samples must be exposed to the model for tens or even hundreds of times during training until an acceptable accuracy threshold is reached, which drives up training costs in terms of resources like memory storage or computing time.

Original: To mitigate these very large training costs, approaches based on data representation formats simpler than the Floating Point 32-bit (FP32) standard have been proposed (Courbariaux et al., 2014; Gupta et al., 2015).
Revised: To mitigate these very large training costs, approaches based on data representation formats simpler than the Floating Point 32-bit (FP32) standard have been proposed (Courbariaux et al., 2014; Gupta et al., 2015).

Original: These approaches successfully mitigate the enormous training costs of DNNs by using data representation formats that either reduce computing costs or diminish the requirements in terms of memory storage and bandwidth.
Revised: These approaches successfully mitigate the enormous training costs of DNNs by using data representation formats that either reduce computing costs or diminish the requirements in terms of memory storage and bandwidth.

Original: In particular, some of these proposals have shown the beneﬁts of combining half-precision and single-precision compute during training in terms of keeping model accuracy and reducing compute and memory costs (Micikevicius et al., 2017; Kalamkar et al., 2019).
Revised: In particular, some of these proposals have shown the beneﬁts of combining half-precision and single-precision compute during training in terms of keeping model accuracy and reducing compute and memory costs (Micikevicius et al., 2017; Kalamkar et al., 2019).

Original: These approaches accelerate linear algebra operations by accumulating half-precision input operands to generate 32-bit outputs.
Revised: These approaches accelerate linear algebra operations by accumulating half-precision input operands to generate 32-bit outputs.

Original: While this mixed-precision (MP) arithmetic can successfully reduce the use of resources like memory bandwidth or hardware components like register ﬁle size, it has a very limited capacity for diminishing computing costs and it is unable to reduce output data size.
Revised: While this mixed-precision (MP) arithmetic can successfully reduce the use of resources like memory bandwidth or hardware components like register ﬁle size, it has a very limited capacity for diminishing computing costs and it is unable to reduce output data size.

Original: In this paper we propose new training methodologies able to exclusively use half-precision for a large part of the training process, which constitutes a very signiﬁcant improvement over mixedprecision approaches in terms of compute and memory bandwidth requirements.
Revised: In this paper we propose new training methodologies able to exclusively use half-precision for a large part of the training process, which constitutes a very signiﬁcant improvement over mixedprecision approaches in terms of compute and memory bandwidth requirements.

Original: We propose two different approaches, the ﬁrst one statically assigns either the Brain Floating Point 16-bit (BF16) or the FP32 format to the model parameters involved in the training process, while the second dynamically switches between BF16 and MP during training depending on its progress.
Revised: We propose two different approaches, the ﬁrst one statically assigns either the Brain Floating Point 16-bit (BF16) or the FP32 format to the model parameters involved in the training process, while the second dynamically switches between BF16 and MP during training depending on its progress.

Original: Our approaches do not require mixed-precision arithmetic while computing linear algebra operations for a large portion of the training process, which enables them to deliver the same performance as if they were [Figure] [Figure] [Figure]
Revised: Our approaches do not require mixed-precision arithmetic while computing linear algebra operations for a large portion of the training process, which enables them to deliver the same performance as if they were operating with half-precision arithmetic during the whole training while providing the same model accuracy as if FP32 was used.

Original: C

Original: D (

Original: [Figure]

Original: FP

Original: [Figure]

Original: MP) (

Original: [Figure]

Original: [Figure]

Original: ) (FP32) (

Original: [Figure]

Original: C

Original: D

Original: [Figure]

Original: (MP

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: C

Original: [Figure]

Original: operating with half-precision arithmetic during the whole training while providing the same model accuracy as if FP32 was used.

Original: This paper is the ﬁrst in demonstrating that half-precision can be extensively used during DNNs training without the need for mixed-precision arithmetic.
Revised: This paper is the ﬁrst in demonstrating that half-precision can be extensively used during DNNs training without the need for mixed-precision arithmetic.

Original: We made our code available 1 .
Revised: We made our code available 1 .

Original:  B ACKGROUND ON M IXED -P RECISION A PPROACHES AND M OTIVATION
Revised:  B ACKGROUND ON M IXED -P RECISION A PPROACHES AND M OTIVATION

Original: Mixed-Precision training has been extensively explored in recent years.
Revised: Mixed-Precision training has been extensively explored in recent years.

Original: Approaches mixing Floating Point 16-bit (FP16) and FP32 datatypes have been proposed (Micikevicius et al., 2017).
Revised: Approaches mixing Floating Point 16-bit (FP16) and FP32 datatypes have been proposed (Micikevicius et al., 2017).

Original: In these approaches, multiplications of FP16 parameters are accumulated in FP32 registers to minimize data representation range and precision issues.
Revised: In these approaches, multiplications of FP16 parameters are accumulated in FP32 registers to minimize data representation range and precision issues.

Original: Importantly, relevant phases of the training process like computing weight updates (WU) or dealing with batch normalization (BN) layers entirely use FP32, which implies that a FP32 representation of network weights and biases is kept during the whole training.
Revised: Importantly, relevant phases of the training process like computing weight updates (WU) or dealing with batch normalization (BN) layers entirely use FP32, which implies that a FP32 representation of network weights and biases is kept during the whole training.

Original: This approach requires some additional computations to enforce that FP32 values are converted to FP16 without data representation range issues.
Revised: This approach requires some additional computations to enforce that FP32 values are converted to FP16 without data representation range issues.

Original: This approach is used by Nvidia Tesla V
Revised: This approach is used by Nvidia Tesla V

Original: GPUs via mixed-precision computing units called tensor cores, which are able to multiply FP16 parameters and store the results in FP32.
Revised: GPUs via mixed-precision computing units called tensor cores, which are able to multiply FP16 parameters and store the results in FP32.

Original: Figure 1a displays the most fundamental operation of this approach combining FP16 and FP32, the mixed-precision Fused Multiply-Add (FMA) instruction, which computes D =
Revised: Figure 1a displays the most fundamental operation of this approach combining FP16 and FP32, the mixed-precision Fused Multiply-Add (FMA) instruction, which computes D =

Original: A · B + C .
Revised: A · B + C .

Original: Input parameters A and B are represented in the FP16 format.
Revised: Input parameters A and B are represented in the FP16 format.

Original: The result of the A · B operation is kept in FP32 and added to the C parameter, which is represented in FP32 as well.
Revised: The result of the A · B operation is kept in FP32 and added to the C parameter, which is represented in FP32 as well.

Original: The ﬁnal output D is also represented in FP32.
Revised: The ﬁnal output D is also represented in FP32.

Original: FMA instructions constitute around 60% of the whole training workload for several relevant CNN models, as Section 3 shows.
Revised: FMA instructions constitute around 60% of the whole training workload for several relevant CNN models, as Section 3 shows.

Original: A more recent approach proposes mixed-precision arithmetic combining BF16 and FP32 (Kalamkar et al., 2019).
Revised: A more recent approach proposes mixed-precision arithmetic combining BF16 and FP32 (Kalamkar et al., 2019).

Original: It is very close to its FP16 counterpart with the exception of the full to half precision conversion.
Revised: It is very close to its FP16 counterpart with the exception of the full to half precision conversion.

Original: Since BF16 has the same data representation range as FP32, conversion from full to half precision is very simple in this case since it just requires applying the Round to Nearest Even (RNE) technique.
Revised: Since BF16 has the same data representation range as FP32, conversion from full to half precision is very simple in this case since it just requires applying the Round to Nearest Even (RNE) technique.

Original: This approach also processes WU and BN layers with FP32.
Revised: This approach also processes WU and BN layers with FP32.

Original: Figure 1b shows a representation of a mixed-precision FMA combining BF16 and FP32.
Revised: Figure 1b shows a representation of a mixed-precision FMA combining BF16 and FP32.

Original: It is very close to the previously described FP16-FP32 FMA with the only difference being the data representation format of input parameters A and B .
Revised: It is very close to the previously described FP16-FP32 FMA with the only difference being the data representation format of input parameters A and B .

Original: While mixed-precision FMA instructions bring signiﬁcant beneﬁts since they require less memory bandwidth and register storage than FP32 FMAs, there is still a large margin for improvement if an entirely BF16 FMA like the one represented in Figure 1c could be extensively used for training purposes.
Revised: While mixed-precision FMA instructions bring signiﬁcant beneﬁts since they require less memory bandwidth and register storage than FP32 FMAs, there is still a large margin for improvement if an entirely BF16 FMA like the one represented in Figure 1c could be extensively used for training purposes.

Original: First, since BF16 FMA requires exactly one half of the register storage of a FP32 FMA, it doubles its Single Instruction Multiple Data (SIMD) vectorization capacity and, therefore, it may signiﬁcantly increase its FMA instructions per cycle ratio.
Revised: First, since BF16 FMA requires exactly one half of the register storage of a FP32 FMA, it doubles its Single Instruction Multiple Data (SIMD) vectorization capacity and, therefore, it may signiﬁcantly increase its FMA instructions per cycle ratio.

Original: Extensions of Instruction Set Architectures (ISA) to allow SIMD parallelism are becoming a key element for ﬂoating-point performance, which has motivated major hardware vendors to include them in their products (Jeffers et al., 2016;
Revised: Extensions of Instruction Set Architectures (ISA) to allow SIMD parallelism are becoming a key element for ﬂoating-point performance, which has motivated major hardware vendors to include them in their products (Jeffers et al., 2016;

Original: Stephens et al., 2017).
Revised: Stephens et al., 2017).

Original: Finally, BF16 FMA instructions also bring signiﬁcant reductions in terms of memory bandwidth since they involve 50% and 25% less data than FP32 and MP FMAs, respectively.
Revised: Finally, BF16 FMA instructions also bring signiﬁcant reductions in terms of memory bandwidth since they involve 50% and 25% less data than FP32 and MP FMAs, respectively.

Original: While half-precision arithmetic has not been used to train DNNs due to its lack of training convergence, this paper describes [Figure] P e r ce n t ag e
Revised: While half-precision arithmetic has not been used to train DNNs due to its lack of training convergence, this paper describes two techniques to fully use it while keeping the same convergence properties as FP32.

Original: o f I n s tr u c t i o n s

Original: Other FP

Original: FMA

Original: No FP (a) Instruction breakdown       14 Epochs 0 .0 .0 .0 .0 .0 .0 .0 . A cc u r a c y Top1-FP32 Top1-MP Top1-BF (b) Static techniques on ResNet-50.
Revised: This paper analyzes in detail 3 relevant training workloads in Section 3, and applies these ﬁndings to build its two main contributions in Section 4.

Original:  W ORKLOAD A NALYSIS
Revised:  W ORKLOAD A NALYSIS

Original: We consider three CNN models: AlexNet, Inception V2, and ResNet-50.
Revised: We consider three CNN models: AlexNet, Inception V2, and ResNet-50.

Original: Section 5 describes the exact way we use these models and the methodology we follow to analyze their training workload.
Revised: Section 5 describes the exact way we use these models and the methodology we follow to analyze their training workload.

Original: Figure 2a shows an instruction breakdown of processing one batch on these networks.
Revised: Figure 2a shows an instruction breakdown of processing one batch on these networks.

Original: This ﬁgure shows how ﬂoating-point instructions constitute a large portion of these workloads.
Revised: This ﬁgure shows how ﬂoating-point instructions constitute a large portion of these workloads.

Original: For example, they represent 58.44% of the total in the case of AlexNet.
Revised: For example, they represent 58.44% of the total in the case of AlexNet.

Original: A very large portion of these ﬂoatingpoint instructions, 57.42% of the total, are FMA instructions.
Revised: A very large portion of these ﬂoatingpoint instructions, 57.42% of the total, are FMA instructions.

Original: In the cases of Inception and ResNet50, FMA instructions represent 60.93% and 62.95% of the total, respectively.
Revised: In the cases of Inception and ResNet- [Figure] [Figure] 50, FMA instructions represent 60.93% and 62.95% of the total, respectively.

Original: Therefore, FMA instructions constitute a large portion of the whole training workload, while other FP32 instructions represent a small instruction count that remain below 1.1% for these three CNNs.
Revised: Therefore, FMA instructions constitute a large portion of the whole training workload, while other FP32 instructions represent a small instruction count that remain below 1.1% for these three CNNs.

Original: This justiﬁes to focus on FMA instructions, as executing them in half-precision has a large potential for performance improvement.
Revised: This justiﬁes to focus on FMA instructions, as executing them in half-precision has a large potential for performance improvement.

Original: Prior research (Micikevicius et al., 2017; Kalamkar et al., 2019) describes the need for using 32bit arithmetic in Weight Updates (WU) and Batch Normalization (BN) layers when using training approaches based on mixed-precision arithmetic.
Revised: Prior research (Micikevicius et al., 2017; Kalamkar et al., 2019) describes the need for using 32bit arithmetic in Weight Updates (WU) and Batch Normalization (BN) layers when using training approaches based on mixed-precision arithmetic.

Original: We run an experimental campaign to conﬁrm this observation and to measure the number of instructions devoted to WU and BN.
Revised: We run an experimental campaign to conﬁrm this observation and to measure the number of instructions devoted to WU and BN.

Original: For the case of
Revised: For the case of

Original: ResNet-50, this instruction count is around 30 million instructions per batch, that is, just 0.04% of the FP instructions.
Revised: ResNet-50, this instruction count is around 30 million instructions per batch, that is, just 0.04% of the FP instructions.

Original: AlexNet and Inception V2 produce similar results.
Revised: AlexNet and Inception V2 produce similar results.

Original: In conclusion, reducing the cost of FMA instructions has a high potential for very signiﬁcant performance improvements even if WU and BN layers are computed using full precision arithmetic.
Revised: In conclusion, reducing the cost of FMA instructions has a high potential for very signiﬁcant performance improvements even if WU and BN layers are computed using full precision arithmetic.

Original: Processing one training batch for the cases of AlexNet, Inception and ResNet-50 requires running 53.3, 37.2, and 70.0 billion dynamic instructions per batch, respectively.
Revised: Processing one training batch for the cases of AlexNet, Inception and ResNet-50 requires running 53.3, 37.2, and 70.0 billion dynamic instructions per batch, respectively.

Original: The number of model parameters drives the size of these workloads.
Revised: The number of model parameters drives the size of these workloads.

Original: AlexNet was trained with a batch size of 256, while
Revised: AlexNet was trained with a batch size of 256, while

Original: Inception and ResNet use a batch size of 64.
Revised: Inception and ResNet use a batch size of 64.

Original:  P ROPOSAL
Revised:  P ROPOSAL

Original: We propose two training methodologies that rely exclusively on half-precision BF16 for a large portion of the training process, i.e., a large portion of FMA instructions.
Revised: We propose two training methodologies that rely exclusively on half-precision BF16 for a large portion of the training process, i.e., a large portion of FMA instructions.

Original: Prior mixed-precision approaches preclude large gains in computing costs as some of the data elements remain in FP32.
Revised: Prior mixed-precision approaches preclude large gains in computing costs as some of the data elements remain in FP32.

Original: However, an FMA entirely relying on BF16 can potentially double the SIMD vectorization throughput of current processors and alleviate memory bandwidth requirements.
Revised: However, an FMA entirely relying on BF16 can potentially double the SIMD vectorization throughput of current processors and alleviate memory bandwidth requirements.

Original: We ﬁrst propose a scheme that performs all FMA instructions in BF16 (see Figure 1c) except those involved in computing WU and processing BN layers, which are entirely performed in FP32.
Revised: We ﬁrst propose a scheme that performs all FMA instructions in BF16 (see Figure 1c) except those involved in computing WU and processing BN layers, which are entirely performed in FP32.

Original: While this method might not deliver the desired level of accuracy for all CNNs, Section 6 shows how it behaves remarkably well for the Inception V2 model, since it obtains the same level of accuracy as state-of-the training using MP and FP32.
Revised: While this method might not deliver the desired level of accuracy for all CNNs, Section 6 shows how it behaves remarkably well for the Inception V2 model, since it obtains the same level of accuracy as state-of-the training using MP and FP32.

Original: However, some CNNs cannot entirely rely on half-precision arithmetic during training.
Revised: However, some CNNs cannot entirely rely on half-precision arithmetic during training.

Original: For example, Figure 2b shows top1 accuracy achieved by three training techniques during 15 epochs for
Revised: For example, Figure 2b shows top1 accuracy achieved by three training techniques during 15 epochs for

Original: ResNet-50.
Revised: ResNet-50.

Original: The ﬁrst technique (referred as FP32 in Figure 2b) entirely relies in FP32 arithmetic, the second approach (referred as MP in Figure 2b) represents the state-of-the art Mixed-Precision training (Kalamkar et al., 2019), and the third approach (referred as BF16 in Figure 2b) performs all FMA instructions in BF16 except for WU and BN.
Revised: The ﬁrst technique (referred as FP32 in Figure 2b) entirely relies in FP32 arithmetic, the second approach (referred as MP in Figure 2b) represents the state-of-the art Mixed-Precision training (Kalamkar et al., 2019), and the third approach (referred as BF16 in Figure 2b) performs all FMA instructions in BF16 except for WU and BN.

Original: While the BF16 approach behaves relatively well, it displays lower accuracy than MP and FP32 for all the epochs, which indicates the need for an approach able to take advantage of BF16 arithmetic while delivering the same accuracy results as mixed- or full-precision approaches.
Revised: While the BF16 approach behaves relatively well, it displays lower accuracy than MP and FP32 for all the epochs, which indicates the need for an approach able to take advantage of BF16 arithmetic while delivering the same accuracy results as mixed- or full-precision approaches.

Original: The methodology we use to generate Figure 2b is described in Section 5.
Revised: The methodology we use to generate Figure 2b is described in Section 5.

Original: Our second contribution dynamically switches between MP and BF16 to deliver the same accuracy as MP while relying in BF16 FMAs during a large portion of the training process.
Revised: Our second contribution dynamically switches between MP and BF16 to deliver the same accuracy as MP while relying in BF16 FMAs during a large portion of the training process.

Original: Algorithm 1 displays a high level pseudo-code of our proposal.
Revised: Algorithm 1 displays a high level pseudo-code of our proposal.

Original: It starts the training process using the state-of-the art mixed-precision approach (Kalamkar et al., 2019) for several batches, deﬁned by numBatchesMP parameter.
Revised: It starts the training process using the state-of-the art mixed-precision approach (Kalamkar et al., 2019) for several batches, deﬁned by numBatchesMP parameter.

Original: Then, it computes the Exponential Moving Average (EMA) (Lawrance & Lewis, 1977) of the training loss and, if its reduction is larger than a certain threshold ( emaThreshold parameter), it computes the next numBatchesBF 16 using BF16 FMAs, except for WU and BN.
Revised: Then, it computes the Exponential Moving Average (EMA) (Lawrance & Lewis, 1977) of the training loss and, if its reduction is larger than a certain threshold ( emaThreshold parameter), it computes the next numBatchesBF 16 using BF16 FMAs, except for WU and BN.

Original: Once training has gone through these numBatchesBF 16 batches, our algorithm checks EMA and compares its reduction with the emaThreshold parameter.
Revised: Once training has gone through these numBatchesBF 16 batches, our algorithm checks EMA and compares its reduction with the emaThreshold parameter.

Original: If this reduction is not large enough, the algorithm switches back to MP arithmetic.
Revised: If this reduction is not large enough, the algorithm switches back to MP arithmetic.

Original: Otherwise, it keeps using BF16 arithmetic for numBatchesBFbatches before checking EMA again.
Revised: Otherwise, it keeps using BF16 arithmetic for numBatchesBFbatches before checking EMA again.

Original: (cid:46) Input parameter deﬁning number of batches processed using MP 2: numBatchesBF(cid:46)

Original: Input parameter deﬁning number of batches processed using BF 3: emaThreshold (cid:46) Input parameter deﬁning the EMA reduction threshold 4: i ←5: Mixed Precision (MP) : 6: Check EMA every numBatchesMP batches (cid:46) EMA is calculated over the training loss 7: if EMA reduction > emaThreshold then (cid:46) EMA reduction check 8: goto BF16 Precision 9: else 10: goto Mixed Precision 11: BF16 Precision (BF16) :

Original: 12: i ← i +13: if i < = numBatchesBF16 then (cid:46)

Original: We stay on BF16 for a speciﬁc number of batches 14: goto BF16 Precision 15: else 16: if EMA reduction >

Original: emaThreshold then (cid:46)

Original: If training loss improves, stay in BF 17: i ←18: goto BF16 Precision 19: else (cid:46)

Original: If training loss is not improving, return to MP 20: goto Mixed Precision

Original:  E XPERIMENTAL M ETHODOLOGY 5. E XPERIMENTAL SETUP
Revised:  E XPERIMENTAL M ETHODOLOGY 5. E XPERIMENTAL SETUP

Original: Our experiments are performed on Intel Xeon Platinum 8160 processors, which include the AVX
Revised: Our experiments are performed on Intel Xeon Platinum 8160 processors, which include the AVX

Original: ISA.
Revised: ISA.

Original: We use the Intel-Caffe (Intel, a) framework (version 1.1.6a).
Revised: We use the Intel-Caffe (Intel, a) framework (version 1.1.6a).

Original: We use the Intel MKLDNN (Intel, c) (version 0.18.0)
Revised: We use the Intel MKLDNN (Intel, c) (version 0.18.0)

Original: Deep Neural Network library and the Intel MKL library (Intel, b) (version 2019.0.3) to run numerical kernels since both libraries are optimized to perform well on our testing infrastructure.
Revised: Deep Neural Network library and the Intel MKL library (Intel, b) (version 2019.0.3) to run numerical kernels since both libraries are optimized to perform well on our testing infrastructure.

Original: Finally, to deﬁne and run the experiments we use the pyCaffe python interface, which takes care of loading the data and orchestrating the execution.
Revised: Finally, to deﬁne and run the experiments we use the pyCaffe python interface, which takes care of loading the data and orchestrating the execution.

Original: 5. E MULATION OF BF16 USING DYNAMIC BINARY INSTRUMENTATION
Revised: 5. E MULATION OF BF16 USING DYNAMIC BINARY INSTRUMENTATION

Original: Due to the lack of available hardware implementing the BF16 numerical format, we rely on an emulation technique to perform our experiments.
Revised: Due to the lack of available hardware implementing the BF16 numerical format, we rely on an emulation technique to perform our experiments.

Original: Several approaches have been used in the past to emulate the behaviour of reduced ﬂoating-point representations, most notably via libraries that perform transformations like truncation and rounding (Chatelain et al., 2019; Dawson & D¨uben, 2017;
Revised: Several approaches have been used in the past to emulate the behaviour of reduced ﬂoating-point representations, most notably via libraries that perform transformations like truncation and rounding (Chatelain et al., 2019; Dawson & D¨uben, 2017;

Original: Kalamkar et al., 2019).
Revised: Kalamkar et al., 2019).

Original: We develop a binary analysis tool based on PIN 3.7 (Luk et al., 2005).
Revised: We develop a binary analysis tool based on PIN 3.7 (Luk et al., 2005).

Original: Our tool captures and instruments dynamic instructions, which enables adaptating numerical operands to the targeted numerical data format.
Revised: Our tool captures and instruments dynamic instructions, which enables adaptating numerical operands to the targeted numerical data format.

Original: Our approach seamlessly works on complex frameworks like
Revised: Our approach seamlessly works on complex frameworks like

Original: PyTorch, Tensorﬂow, or Caffe, with interpreted languages, and is able to instrument instructions triggered by dynamically linked libraries.
Revised: PyTorch, Tensorﬂow, or Caffe, with interpreted languages, and is able to instrument instructions triggered by dynamically linked libraries.

Original: Our binary analysis tool performs the following steps:
Revised: Our binary analysis tool performs the following steps:

Original: • It checks the current operation mode, which can be FP32, MP, or BF16 (see Figure 1).
Revised: • It checks the current operation mode, which can be FP32, MP, or BF16 (see Figure 1).

Original: • It checks the current execution routine to determine if we are executing routines that belong to WU or BN layers.
Revised: • It checks the current execution routine to determine if we are executing routines that belong to WU or BN layers.

Original: If that is the case, computation proceeds with FP32.
Revised: If that is the case, computation proceeds with FP32.

Original: • The tool intercepts the dynamic instructions of the workload and detects all ﬂoating-point operations, including FMAs.
Revised: • The tool intercepts the dynamic instructions of the workload and detects all ﬂoating-point operations, including FMAs.

Original: For each FMA instruction, operands that need to be rounded to BF16, depending on the current operation mode, are rounded using the RNE algorithm. 
Revised: For each FMA instruction, operands that need to be rounded to BF16, depending on the current operation mode, are rounded using the RNE algorithm. Algorithm

Revised: 1: numBatchesMP ←(cid:46)

Revised: Number of consecutive MP batches 2: numBatchesBF 16 ←(cid:46)

Original: • The tool can dynamically change its operation mode anytime via a simple inter-process communication method that can be invoked from the python high-level interface.
Revised: Number of consecutive BF16 batches 3: emaThreshold ← 0 .(cid:46) Deﬁnes EMA reduction threshold 4: 5: precisionModeBF 16 ← False (cid:46) Indicates current precision mode, True means BF 6: countBatchesBF 16 ←(cid:46) Counts how many numBatchesBF16 have been executed 7: numBatchesTrain ← numBatchesMP (cid:46) Number of batches per training loop iteration 8: 9: for i = 0 to niter do (cid:46) Training Loop: niter value depends on the number of epochs 10: train.step(numBatchesTrain) (cid:46) Execute numBatchesTrain batches in precisionModeBF 11: trainingLoss [ i ] ← train.trainingLoss 12: if ( i = 5) then (cid:46)

Revised: Initial history to calculate EMA 13:

Revised: EMA ← average ( trainingLoss ) 14: if ( i > 5) then 15:

Revised: EMAprev ← EMA 16:

Revised: EMA ← emaCalculation ( trainingLoss, EMAprev ) (cid:46)

Revised: Each numBatchesMP 17: if ( precisionModeBF 16!

Revised: =

Revised: True ) then 18: if (( EMAprev − EMA ) >

Revised: emaThreshold ) then (cid:46)

Revised: If training loss goes down 19: precisionModeBF 16 ← True 20: changeToBF16() (cid:46) Switch precision to BF 21: else 22: countBatchesBF 16 ← countBatchesBF 16 + numBatchesTrain 23: if ( countBatchesBF 16 = numBatchesBF 16) then 24: if (( EMAprev − EMA ) >

Revised: emaThreshold ) then (cid:46)

Revised: If training loss goes down 25: countBatchesBF 16 ←(cid:46) Stay in BF16 precision 26: else (cid:46)

Revised: If training loss stagnates 27: precisionModeBF 16 ← False 28: changeToMP() (cid:46) Switch precision to MP 29: countBatchesBF 16 ←

Revised: • The tool can dynamically change its operation mode anytime via a simple inter-process communication method that can be invoked from the python high-level interface.

Original: To mitigate the overhead of our binary analysis tool, we implement two optimizations: First, we vectorize the truncation and rounding routines via AVX512 instructions.
Revised: To mitigate the overhead of our binary analysis tool, we implement two optimizations: First, we vectorize the truncation and rounding routines via AVX512 instructions.

Original: Second, we avoid redundant rounding and truncation operations by identifying instructions belonging to the same basic block sharing some input operands already stored in the register ﬁle.
Revised: Second, we avoid redundant rounding and truncation operations by identifying instructions belonging to the same basic block sharing some input operands already stored in the register ﬁle.

Original: These two optimizations reduce the overhead of the tool from 100 × to 25 × with respect to native runs of the binary on real hardware.
Revised: These two optimizations reduce the overhead of the tool from 100 × to 25 × with respect to native runs of the binary on real hardware.

Original: 5. D YNAMIC AND S TATIC T ECHNIQUES
Revised: 5. D YNAMIC AND S TATIC T ECHNIQUES

Original: This paper considers two different types of training techniques: static schemes and dynamic schemes .
Revised: This paper considers two different types of training techniques: static schemes and dynamic schemes .

Original: When using static schemes, the training procedure uses the same data representation form for a given parameter during its complete execution.
Revised: When using static schemes, the training procedure uses the same data representation form for a given parameter during its complete execution.

Original: For example, the three techniques displayed in
Revised: For example, the three techniques displayed in

Original: Figure 2b are static.
Revised: Figure 2b are static.

Original: We deﬁne the following schemes:
Revised: We deﬁne the following schemes:

Original: • MP: FMA instructions belonging to WU and BN layers always use FP32 precision.
Revised: • MP: FMA instructions belonging to WU and BN layers always use FP32 precision.

Original: The remaining FMA instructions use the mixed-precision approach represented in Figure 1b).
Revised: The remaining FMA instructions use the mixed-precision approach represented in Figure 1b).

Original: [Table]
Revised: This scheme replicates prior work on mixed-precision(Kalamkar et al., 2019).

Original: • BF16: FMA instructions belonging to WU and BN layers always use FP32 precision.
Revised: • BF16: FMA instructions belonging to WU and BN layers always use FP32 precision.

Original: The remaining FMA instructions use BF16 operands to multiply and to accumulate (Figure 1c).
Revised: The remaining FMA instructions use BF16 operands to multiply and to accumulate (Figure 1c).

Original: The BF16 method is the ﬁrst contribution of this paper.
Revised: The BF16 method is the ﬁrst contribution of this paper.

Original: It extensively uses half-precision arithmetic while displaying good convergence properties.
Revised: It extensively uses half-precision arithmetic while displaying good convergence properties.

Original: The Dynamic scheme we propose in this paper switches between the MP and BF16 static techniques during training, as explained in Section 4 and detailed in Algorithm 1.
Revised: The Dynamic scheme we propose in this paper switches between the MP and BF16 static techniques during training, as explained in Section 4 and detailed in Algorithm 1.

Original: This dynamic method improves the training convergence properties of BF16 while still relying in half-precision FMAs for a very large portion of the execution.
Revised: This dynamic method im- proves the training convergence properties of BF16 while still relying in half-precision FMAs for a very large portion of the execution.

Original: The EMA threshold ( emaTreshold ) is set at 4%.
Revised: The EMA threshold ( emaTreshold ) is set at 4%.

Original: This value is computed as the average EMA reduction when using FP32 computations.
Revised: This value is computed as the average EMA reduction when using FP32 computations.

Original: The minimum number of batches to be performed in BF16 , deﬁned by the numBatchesBF 16 parameter is set to 1,000, which precludes frequent unnecessary transitions between the two schemes.
Revised: The minimum number of batches to be performed in BF16 , deﬁned by the numBatchesBF 16 parameter is set to 1,000, which precludes frequent unnecessary transitions between the two schemes.

Original: We set the numBatchesMP parameter to 10, which keeps the number of batches using the MP regime low while keeping its beneﬁts in terms of convergence.
Revised: We set the numBatchesMP parameter to 10, which keeps the number of batches using the MP regime low while keeping its beneﬁts in terms of convergence.

Original: 5. C ONVOLUTIONAL N EURAL N ETWORK M ODELS
Revised: 5. C ONVOLUTIONAL N EURAL N ETWORK M ODELS

Original: To evaluate our proposals we consider the AlexNet (Krizhevsky et al., 2012), Inception V2 (Szegedy et al., 2015) and ResNet50 (He et al., 2015b)models.
Revised: To evaluate our proposals we consider the AlexNet (Krizhevsky et al., 2012), Inception V2 (Szegedy et al., 2015) and ResNet50 (He et al., 2015b) models.

Original: They are representative CNN state-of-the-art.
Revised: They are representative CNN state-of-the-art.

Original: We use the ImageNet database (Deng et al., 2009) as training input.
Revised: We use the ImageNet database (Deng et al., 2009) as training input.

Original: To keep execution times manageable when using our binary instrumentation tool, we run the experiments using a reduced ImageNet Database, similar to the Tiny ImageNet Visual Recognition challenge data set (Fei-Fei).
Revised: To keep execution times manageable when using our binary instrumentation tool, we run the experiments using a reduced ImageNet Database, similar to the Tiny ImageNet Visual Recognition challenge data set (Fei-Fei).

Original: Therefore, we use 256,000 images divided into 200 categories for training, and 10,000 images for validation.
Revised: Therefore, we use 256,000 images divided into 200 categories for training, and 10,000 images for validation.

Original: The images have no modiﬁcations in terms of the size.
Revised: The images have no modiﬁcations in terms of the size.

Original: All the evaluated CNN models remain unmodiﬁed, the only change is loading a reduced dataset.
Revised: All the evaluated CNN models remain unmodiﬁed, the only change is loading a reduced dataset.

Original: AlexNet is selected due to its simplicity in terms of structure and amount of required computations.
Revised: AlexNet is selected due to its simplicity in terms of structure and amount of required computations.

Original: To train AlexNet we consider a batch size of 256 and the base learning rate is 0.01, which is adjusted every 20 epochs taking into account a weight decay of 0.0005 and a momentum of 0.9.
Revised: To train AlexNet we consider a batch size of 256 and the base learning rate is 0.01, which is adjusted every 20 epochs taking into account a weight decay of 0.0005 and a momentum of 0.9.

Original: This model is trained for 32 epochs.
Revised: This model is trained for 32 epochs.

Original: We use Inception because it is a model conceived to reduce computational costs via cheap 1x convolutions.
Revised: We use Inception because it is a model conceived to reduce computational costs via cheap 1x convolutions.

Original: To train it we use a batch size of 64 and a base learning rate of 0.045, which is updated every 427 steps (0.11 epochs).
Revised: To train it we use a batch size of 64 and a base learning rate of 0.045, which is updated every 427 steps (0.11 epochs).

Original: The gamma, momentum and weight decay are set to 0.96, 0.9, and 0.0002, respectively.
Revised: The gamma, momentum and weight decay are set to 0.96, 0.9, and 0.0002, respectively.

Original: The training process is executed for 16 epochs.
Revised: The training process is executed for 16 epochs.

Original: Finally we use ResNet-50.
Revised: Finally we use ResNet-50.

Original: It is a network that delivers good accuracy and avoids the vanishing gradients issue by using residual blocks and the MSRA initializer (He et al., 2015a).
Revised: It is a network that delivers good accuracy and avoids the vanishing gradients issue by using residual blocks and the MSRA initializer (He et al., 2015a).

Original: We train it using a multi-step approach.
Revised: We train it using a multi-step approach.

Original: The batch size is 64 and the base learning rate is 0.05, which is updated every 30 epochs.
Revised: The batch size is 64 and the base learning rate is 0.05, which is updated every 30 epochs.

Original: The gamma hyperparameter, momentum value, and weight decay are set to 0.1, 0.9, and 0.0001, respectively.
Revised: The gamma hyperparameter, momentum value, and weight decay are set to 0.1, 0.9, and 0.0001, respectively.

Original: The training process runs for a total of 32 epochs.
Revised: The training process runs for a total of 32 epochs.

Original:  E VALUATION
Revised:  E VALUATION

Original: Figure 3 and Table 1 show results from our evaluation campaign.
Revised: Figure 3 and Table 1 show results from our evaluation campaign.

Original: The x-axis of the three plots belonging to Figure 3 represent the epochs of the training process while the y-axis represents the accuracy reached by the model over the validation set.
Revised: The x-axis of the three plots belonging to Figure 3 represent the epochs of the training process while the y-axis represents the accuracy reached by the model over the validation set.

Original: Table 1 shows the test accuracy we reach for the three network models when using the FP32 and MP baselines and our two contributions: BF and Dynamic .
Revised: Table 1 shows the test accuracy we reach for the three network models when using the FP32 and MP baselines and our two contributions: BF and Dynamic .

Original: The AlexNet model, due to its structure, shows a good response when lower precision numerical data types are used.
Revised: The AlexNet model, due to its structure, shows a good response when lower precision numerical data types are used.

Original: As can be seen in Figure 3a all techniques converge, although the BF16 approach shows the worse accuracy when compared to the Dynamic or the MP techniques.
Revised: As can be seen in Figure 3a all techniques converge, although the BF16 approach shows the worse accuracy when compared to the Dynamic or the MP techniques.

Original: Table 1 shows that
Revised: Table 1 shows that

Original: FP32 , MP , Dynamic , and BF16 reach top-5 accuracies of 84.50%, 84.43%, 84.02% and 82.56% for
Revised: FP32 , MP , Dynamic , and BF16 reach top-5 accuracies of 84.50%, 84.43%, 84.02% and 82.56% for

Original: AlexNet after 32 epochs.
Revised: AlexNet after 32 epochs.

Original: Importantly, Dynamic reaches the same accuracy as FP32 and MP while using the BF16 approach for 94.60% of the FMAs.
Revised: Importantly, Dynamic reaches the same accuracy as FP32 and MP while using the BF16 approach for 94.60% of the FMAs.

Original: In contrast, the BF16 static technique does99.93% of the FMAs in full BF16 precision (0.07% are in WU and BN layers), but the accuracy drops by almost 3% in Top-1 and 2% in Top-5. This drop in accuracy happens just by doing an additional 5% BF16 FMAs. This give us some space to improve the Dynamic approach by reducing the percentage of BF16 FMAs with the objective to increase the accuracy of the model. Figure 3b shows the validation accuracy during 16 epochs for the Inception V2 model.
Revised: In contrast, the BF16 static technique does

Original: It shows ﬂuctuation on the accuracy evaluation during training due to its structure and hyperparameters tuning.

Original: Dynamic responds in a robust way to these changes, which highlights its general applicabality.

Original: Table 1 shows that FP32 , MP , Dynamic , and BF16 reach top-5 accuracies of 93.36%, 92.67%, 92.02%, and 92.05% for Inception V2 after 16 epochs.

Original: Finally, the evaluation on ResNet-50 demonstrates that the Dynamic approach is effective when applied to deeper CNNs.

Original: In this case, the precision of the model reaches state-of-the-art levels while using half-precision for 96.4% of the FMA instructions.

Original: Figure 3c and Table 1 display the exact accuracy numbers we get from our evaluation after 32 epochs. In this experiment the Topaccuracy drops just 1.2% comparing the BF16 and dynamic approaches, however we could improve the dynamic technique relaxing the quantity of BF16 FMA executed to gain more accuracy.  R ELATED W ORK Prior work indicates that dynamic ﬁxed-point is effective to train deep neural networks with low precision multipliers (Courbariaux et al., 2014).
Revised: [Table]

Original: This approach obtains state-of-the-art results by uniformly applying the dynamic ﬁxed point format with different scaling factors, which are driven

Original: Epochs 0 .0 .0 .0 .0 .
Revised: Epochs 0 .0 .0 .0 .0 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Top1-FP
Revised: Top1-FP

Original: Top1-MP
Revised: Top1-MP

Original: Top1-BF
Revised: Top1-BF

Original: Top1-Dyn
Revised: Top1-Dyn

Original: Top5-FP
Revised: Top5-FP

Original: Top5-MP
Revised: Top5-MP

Original: Top5-BF
Revised: Top5-BF

Original: Top5-Dyn (a) AlexNet
Revised: Top5-Dyn (a) AlexNet

Original: Epochs 0 .0 .0 .0 .0 .
Revised: Epochs 0 .0 .0 .0 .0 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Top1-FP
Revised: Top1-FP

Original: Top1-MP
Revised: Top1-MP

Original: Top1-BF
Revised: Top1-BF

Original: Top1-Dyn
Revised: Top1-Dyn

Original: Top5-FP
Revised: Top5-FP

Original: Top5-MP
Revised: Top5-MP

Original: Top5-BF
Revised: Top5-BF

Original: Top5-Dyn (b) Inception
Revised: Top5-Dyn (b) Inception

Original: Epochs 0 .0 .0 .0 .0 .
Revised: Epochs 0 .0 .0 .0 .0 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Top1-FP
Revised: Top1-FP

Original: Top1-MP
Revised: Top1-MP

Original: Top1-BF
Revised: Top1-BF

Original: Top1-Dyn
Revised: Top1-Dyn

Original: Top5-FP
Revised: Top5-FP

Original: Top5-MP
Revised: Top5-MP

Original: Top5-BF
Revised: Top5-BF

Original: Top5-Dyn
Revised: Top5-Dyn

Original: [Figure]
Revised: [Figure]

Revised: 99.93% of the FMAs in full BF16 precision (0.07% are in WU and BN layers), but the accuracy drops by almost 3% in Top-1 and 2% in Top-5.

Revised: This drop in accuracy happens just by doing an additional 5% BF16 FMAs.

Revised: This give us some space to improve the Dynamic approach by reducing the percentage of BF16 FMAs with the objective to increase the accuracy of the model.

Revised: Figure 3b shows the validation accuracy during 16 epochs for the Inception V2 model.

Original: [Table]
Revised: It shows ﬂuctuation on the accuracy evaluation during training due to its structure and hyperparameters tuning. Dynamic responds in a robust way to these changes, which highlights its general applicability. Table 1 shows that FP32 , MP , Dynamic , and BF16 reach top-5 accuracies of 93.36%, 92.67%, 92.02%, and 92.05% for Inception V2 after 16 epochs. Finally, the evaluation on ResNet-50 demonstrates that the Dynamic approach is effective when applied to deeper CNNs.

Revised: In this case, the precision of the model reaches state-of-the-art levels while using half-precision for 96.4% of the FMA instructions.

Revised: Figure 3c and Table 1 display the exact accuracy numbers we get from our evaluation after 32 epochs.

Revised: In this experiment the Topaccuracy drops just 1.2% comparing the BF16 and dynamic approaches, however we could improve the dynamic technique relaxing the quantity of BF16 FMA executed to gain more accuracy.

Revised: 6. S ENSITIVITY A NALYSIS FOR D YNAMIC P RECISION A LGORITHM

Revised: We provide a sensitivity analysis for the parameters employed in Algorithm 1.

Revised: The objective is to show that for a range of reasonable parameters the algorithm behaves as expected.

Revised: To do this analysis we set one of the parameters to the currently used value ( numBatchesMP to 10) to have a manageable number of combinations.

Revised: We then test all the possible combinations using numBatchesBF16 = { 500, 1000, 2000 } and emaThreshold = { 0.02, 0.04, 0.08 } , that is, a total of 9 different combinations.

Revised: As stated in Section 5.3, during our evaluation we used the conﬁguration { numBatchesMP , numBatchesBF16 , emaThreshold } = { 10, 1000, 0.04 } for all the evaluated networks.

Revised: Figure 4 shows, for a number of ResNet-50 epochs, the accuracy obtained for each of the 9 tested conﬁgurations as part of the sensitivity analysis.

Revised: The name convention for these conﬁgurations is

Revised: Dyn- < emaThreshold > < numBatchesBF16 > .

Revised: In addition, we include accuracy for BF16, MP, and

Revised: FP32 executions.

Revised: As shown in the ﬁgure, the accuracies obtained at each epoch are always above that of the BF technique.

Revised: For early epochs (i.e., 2 and 4) the dynamic conﬁgurations remain between BF16 and

Revised: FP32 accuracy, or even slightly above FP32, due to initial noise.

Revised: As training advances all dynamic techniques behave similarly and present accuracies that are above BF16 and similar to those obtained with MP and FP32, as we would expect.

Revised: The most important parameter is the emaThreshold , as it decides when a precision change occurs.

Revised: As long as this parameter is reasonably set to detect training loss improvement or degradation the algorithm is bound to behave as expected.

Original:  by the overﬂow rate displayed by the ﬁxed-point numbers.
Revised: [Figure]  R ELATED W ORK Prior work indicates that dynamic ﬁxed-point is effective to train deep neural networks with low precision multipliers (Courbariaux et al., 2014). This approach obtains state-of-the-art results by uniformly applying the dynamic ﬁxed point format with different scaling factors, which are driven by the overﬂow rate displayed by the ﬁxed-point numbers.

Original: Our proposals target deeper neural networks than this approach and do not uniformly apply the same format to all network parameters.
Revised: Our proposals target deeper neural networks than this approach and do not uniformly apply the same format to all network parameters.

Original: Instead, we differentiate between computations requiring FP32 during the whole training, like weight updates, from the ones that are well-suited for dynamic data representation schemes.
Revised: Instead, we differentiate between computations requiring FP32 during the whole training, like weight updates, from the ones that are well-suited for dynamic data representation schemes.

Original: Previous approaches show the beneﬁts of applying stochastic rounding to 16-bit ﬁxed-point multiply and add operators Gupta et al.
Revised: Previous approaches show the beneﬁts of applying stochastic rounding to 16-bit ﬁxed-point multiply and add operators Gupta et al.

Original: This previous work rely on FPGA emulation to show the beneﬁts of stochastic rounding when applied to a custom fully connected neural network dealing with the MNIST dataset.
Revised: This previous work rely on FPGA emulation to show the beneﬁts of stochastic rounding when applied to a custom fully connected neural network dealing with the MNIST dataset.

Original: The authors also consider a CNN similar to LeNet-5 to enlarge their experimental campaign.
Revised: The authors also consider a CNN similar to LeNet-5 to enlarge their experimental campaign.

Original: Previous approaches propose a training process of DNN using 8-bit ﬂoating point numbers (Wang et al., 2018).
Revised: Previous approaches propose a training process of DNN using 8-bit ﬂoating point numbers (Wang et al., 2018).

Original: They rely on a combination of 8-bit and 16-bit and additionally using stochastic rounding to obtain state-of-the-art results.
Revised: They rely on a combination of 8-bit and 16-bit and additionally using stochastic rounding to obtain state-of-the-art results.

Original: The neural networks used in this previous approach are much simpler than the ones we consider in this paper, which do not allow 8-bit arithmetic.
Revised: The neural networks used in this previous approach are much simpler than the ones we consider in this paper, which do not allow 8-bit arithmetic.

Original: The BF16 numerical format has been applied to speciﬁc-purpose hardware targeting deep neural networks (Wang & Kanwar, 2019).
Revised: The BF16 numerical format has been applied to speciﬁc-purpose hardware targeting deep neural networks (Wang & Kanwar, 2019).

Original: This speciﬁc hardware used an approach very similar to the mixed-precision techniques described in the paper by Kalamkar et al.
Revised: This speciﬁc hardware used an approach very similar to the mixed-precision techniques described in the paper by Kalamkar et al.

Original: and, consequently,
Revised: and, consequently,

Original: our
Revised: our

Original: Dynamic approach can be applied on top of them to reduce computing costs.
Revised: Dynamic approach can be applied on top of them to reduce computing costs.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: This paper analyzes the instruction breakdown of workloads focused on deep neural network training that rely on mixed-precision training.
Revised: This paper analyzes the instruction breakdown of workloads focused on deep neural network training that rely on mixed-precision training.

Original: We show that mixed-precision FMAs constitute around 60% of these workloads and propose two approaches based on half-precision FMAs to accelerate the training process without hurting accuracy.
Revised: We show that mixed-precision FMAs constitute around 60% of these workloads and propose two approaches based on half-precision FMAs to accelerate the training process without hurting accuracy.

Original: The ﬁrst approach uses BF16 FMAs for most of the training workload, except routines involved in weight updates or batch normalization layers.
Revised: The ﬁrst approach uses BF16 FMAs for most of the training workload, except routines involved in weight updates or batch normalization layers.

Original: This approach uses BF16 for more than 99% of the FMAs, which has a very strong potential for performance improvement, while reaching slightly smaller accuracy than the state-of-the-art.
Revised: This approach uses BF16 for more than 99% of the FMAs, which has a very strong potential for performance improvement, while reaching slightly smaller accuracy than the state-of-the-art.

Original: We propose a second approach that dynamically switches between different data representation formats.
Revised: We propose a second approach that dynamically switches between different data representation formats.

Original: This dynamic approach uses BF16 for around 96% of the FMAs while reaching the same precision levels as the standard single-precision and mixedprecision approaches.
Revised: This dynamic approach uses BF16 for around 96% of the FMAs while reaching the same precision levels as the standard single-precision and mixedprecision approaches.

Original: Our two proposals are evaluated considering three state-of-the-art deep neural networks and a binary analysis tool that applies the required precision for each instruction.
Revised: Our two proposals are evaluated considering three state-of-the-art deep neural networks and a binary analysis tool that applies the required precision for each instruction.

Original: To the best of our knowledge, this is the ﬁrst paper that demonstrates that half-precision can be used extensively on ≥ 94% of all
Revised: To the best of our knowledge, this is the ﬁrst paper that demonstrates that half-precision can be used extensively on ≥ 94% of all

Original: FMAs during the training of very deep models without the need for mixed-precision arithmetic.
Revised: FMAs during the training of very deep models without the need for mixed-precision arithmetic.