Original: T OWARD
Revised: T OWARD

Original: E FFICIENT
Revised: E FFICIENT

Original: L OW -P RECISION
Revised: L OW -P RECISION

Original: T RAINING :
Revised: T RAINING :

Original: D ATA
Revised: D ATA

Original: F ORMAT
Revised: F ORMAT

Original: O PTIMIZATION
Revised: O PTIMIZATION

Original: AND
Revised: AND

Original: H YSTERESIS
Revised: H YSTERESIS

Original: Q UANTIZATION
Revised: Q UANTIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As the complexity and size of deep neural networks continue to increase, lowprecision training has been extensively studied in the last few years to reduce hardware overhead.
Revised: As the complexity and size of deep neural networks continue to increase, lowprecision training has been extensively studied in the last few years to reduce hardware overhead.

Original: Training performance is largely affected by the numeric formats representing different values in low-precision training, but ﬁnding an optimal format typically requires numerous training runs, which is a very time-consuming process.
Revised: Training performance is largely affected by the numeric formats representing different values in low-precision training, but ﬁnding an optimal format typically requires numerous training runs, which is a very time-consuming process.

Original: In this paper, we propose a method to efﬁciently ﬁnd an optimal format  without actual training of deep neural networks.
Revised: In this paper, we propose a method to efﬁciently ﬁnd an optimal format for activations and errors without actual training.

Original: We employ this method to determine an 8-bit format suitable forthe trainingof various models.
Revised: We employ this method to determine an 8-bit format suitable for training various models.

Original: In addition, we propose anew quantization scheme that utilizes the hysteresis effect to suppress undesiredweight ﬂuctuation during training.
Revised: In addition, we propose a hysteresis quanization scheme to suppress undesired ﬂuctuation in quantized weights.

Original: Thisquantization scheme enables deeply quantized training using 4-bit weights, exhibiting only 0.2% degradation for ResNet-18 trained on ImageNet.
Revised: This scheme enables deeply quantized training using 4-bit weights, exhibiting only 0.2% degradation for ResNet-18 trained on ImageNet.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have been used in various ﬁelds such as vision, audio, natural language processing, and reinforcement learning.
Revised: Deep neural networks have been used in various ﬁelds such as vision, audio, natural language processing, and reinforcement learning.

Original: As larger and more complex neural networks are adopted, the energy and time consumed for training have become a critical issue in hardware implementation.
Revised: As larger and more complex neural networks are adopted, the energy and time consumed for training have become a critical issue in hardware implementation.

Original: Using low-bit representations in training signiﬁcantly reduces hardware overhead and memory footprint; hence, neural network training with limited precision has been extensively studied recently.
Revised: Using low-bit representations in training signiﬁcantly reduces hardware overhead and memory footprint; hence, neural network training with limited precision has been extensively studied recently.

Original: For instance, 16-bit formats are already adopted in commercial devices such as FP16 (IEEE, 2019) in NVidia GPU and bﬂoat16 (Kalamkar et al., 2019) in Google TPU (Wang et al., 2019). 
Revised: For instance, 16-bit formats are already adopted in commercial devices such as FP16 (IEEE, 2019) in NVidia GPU and bﬂoat16 (Kalamkar et al., 2019) in Google TPU (Wang et al., 2019). Also, K ¨ oster et al.

Original: Also, it has been demonstrated that even 8-bit formats could be adopted in deep neural network training with reasonable accuracy (Sun et al., 2019; Fox et al., 2020).
Revised: (2017) suggested a new data format using a shared exponent suitable for low-precision training. Recently, it has been demonstrated that even 8-bit formats could be adopted in deep neural network training with reasonable accuracy (Sun et al., 2019; Fox et al., 2020).

Original: However, there are various issues in realizing low-precision training in practical applications as detailed below.
Revised: However, there are various issues in realizing low-precision training in practical applications as detailed below.

Original: Optimal data format for low-precision training : Training performance is susceptible to the data format we use to represent variables in the network.
Revised: Optimal data format for low-precision training : Training performance is susceptible to the data format we use to represent variables in the network.

Original: When a value is represented using a  ﬁxed number of bits, there is a trade-off between dynamic range and precision.
Revised: When a value is represented using a ﬂoatingpoint format with a ﬁxed number of bits, there is a trade-off between dynamic range and precision.

Original: For instance, allocating more bits to the exponent part in a ﬂoating-point format enlarges the dynamic range but lowers precision due to fewer bits in the mantissa part.
Revised: For instance, allocating more bits to the exponent part in a ﬂoating-point format enlarges the dynamic range but lowers precision due to fewer bits in the mantissa part.

Original: Recent studies on 8-bit training suggest various ways to reduce the dynamic range required for number representation to enhance representation precision.
Revised: Recent studies on 8-bit training suggest various ways to reduce the dynamic range required for number representation to enhance representation precision.

Original: Early work on 8-bit training (Wang et al., 2018) adopts a 5-bit exponent to represent different variables using a single format, but Sun et al.
Revised: Early work on 8-bit training (Wang et al., 2018) adopts a 5-bit exponent to represent different variables using a single format, but Sun et al.

Original: (2019) examine the statistics of each variable and optimize the numeric formats separately.
Revised: (2019) examine the statistics of each variable and optimize the numeric formats separately.

Original: Speciﬁcally, the values used in the forward path (weight and activation) have a relatively narrow dynamic range, and only 4 bits are allocated to the exponent.
Revised: Speciﬁcally, the values used in the forward path (weight and activation) have a relatively narrow dynamic range, and only 4 bits are allocated to the exponent.

Original: Fox et al.
Revised: Fox et al.

Original: (2020) propose to divide data into smaller blocks and assign a shared exponent bias to each block.
Revised: (2020) propose to divide data into smaller blocks and assign a shared exponent bias to each block.

Original: Since the values in a block tend to exhibit similar statistics, the forward (weight and activation) and backward (error) paths could be represented using only 2-bit and 4-bit exponents, respectively. 
Revised: Since the values in a block tend to exhibit similar statistics, the forward (weight and activation) and backward (error) paths could be represented using only 2-bit and 4-bit exponents, respectively. Note that the shared exponent bias is effectively identical to the scaling factor.

Revised: If a variable has a value of m · 2 e and a shared exponent bias of b , then its actual value is m · 2 e + bias , which is identical to the scaling factor of 2 bias .

Original: However, these approaches are difﬁcult to generalize since we should empirically decide numeric formats for each task, neural network structure, and quantization scheme (Fig. 1).
Revised: However, these approaches are difﬁcult to generalize since we should empirically decide numeric formats for each task, neural network structure, and quantization scheme (Fig. 1).

Original: Furthermore, analyzing the statistics of each variable is not enough to determine an optimal format.
Revised: Furthermore, analyzing the statistics of each variable is not enough to determine an optimal format.

Original: Their distributions often have a long tail, andhence the dynamic range of the numeric format should be experimentally selected through many trial-and-errors in actual training.
Revised: Their distributions often have a long tail, and

Original: [Figure]
Revised: [Figure]

Original: +
Revised: +

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: ReLU
Revised: ReLU

Original: Quant +
Revised: Quant +

Revised: Conv

Original: [Figure]
Revised: [Figure]

Revised: ReLU

Original:  +
Revised: [Figure] quantization [Figure] dataset +

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: ReLU
Revised: ReLU

Original: Quant +
Revised: Quant +

Revised: Conv

Original: [Figure]
Revised: [Figure]

Original: ReLU
Revised: ReLU

Original: Quant
Revised: Quant

Original: Different quantization position
Revised: Different quantization position

Original: [Figure]
Revised: [Figure]

Original: Performance degradation in from-scratch training : Low-precision training differs from obtaining quantized neural network models. A quantized neural network refers to the model with quantized parameters (e.g., weight). The target model is ﬁrst trained in full precision, and then this pre-trained model is further ﬁne-tuned while the weights are now kept quantized, minimizing performance degradation by making the network adapt to weight quantization.
Revised: hence the dynamic range of the numeric format should be experimentally selected through many trial-and-errors in actual training.

Original:  Previous studies  show that a model could achieve comparable accuracy to full-precision models even using 1- or 2-bit weights (Choi et al., 2019;
Revised: Performance degradation in from-scratch training : Previous studies on quantized models show that a model could achieve comparable accuracy to full-precision models even using 1- or 2-bit weights (Choi et al., 2019;

Original: Martinez et al., 2020) through ﬁne-tuning a pre-trained model.
Revised: Martinez et al., 2020) through ﬁne-tuning a pre-trained model.

Original: Contrarily, in low-precision training, the entire process of neural network training must be processed in a deeply quantized environment; the neural network is trained from scratch while all the values in the network - not only parameters, but also other variables in the network (e.g., activation, error, and weight gradient) - are deeply quantized.

Original: The quantized model from low-precision training  typically shows a noticeable accuracy dropcompared to the quantized model obtained from ﬁne-tuning of a pre-trained model (Elhoushi et al., 2021).
Revised: However, in low-precision training where a neural network is trained from scratch using low-precision values and computations, the trained model typically shows a noticeable accuracy drop (Elhoushi et al., 2021).

Original: Fig. 1(b) shows the Top-1 validation accuracy of ResNet18 (
Revised: Fig. 1(b) shows the Top-1 validation accuracy of ResNet-18 (He et al., 2016) trained on

Original: He et al., 2016) trained on ImageNet (Deng et al., 2009) for different training schemes.
Revised: ImageNet (Deng et al., 2009) for different training schemes.

Original: The weights are quantized into a 4-bit base-2 logarithmic format.
Revised: The weights are quantized into a 4bit base-2 logarithmic format.

Original: From-scratch training of the model with quantized weights results in a 2.1% accuracy drop, whereas only 1.0% degradation is observed if we ﬁne-tune a pre-trained model.
Revised: From-scratch training of the model with quantized weights results in a 2.1% accuracy drop, whereas only 1.0% degradation is observed if we ﬁne-tune a pre-trained model.

Original: This suggests that even though a better solution exists for a given format, it cannot be reached through from-scratch training.
Revised: This suggests that even though a better solution exists for a given format, it cannot be reached through from-scratch training.

Original: To formalize the issues above, here we divide quantization in low-precision training into two types: network quantization and data ﬂow quantization.
Revised: To formalize the issues above, here we divide quantization in low-precision training into two types: network quantization and data ﬂow quantization.

Original: Network quantization refers to the quantization of the neural network model.
Revised: Network quantization refers to the quantization of the neural network model.

Original: An example of this type of quantization is weight quantization.
Revised: An example of this type of quantization is weight quantization.

Original: In network quantization, we need to reduce the performance difference between from-scratch training and ﬁne-tuning (Yang et al., 2019b).
Revised: In network quantization, we need to reduce the performance difference between from-scratch training and ﬁne-tuning (Yang et al., 2019b).

Original: On the other hand, data ﬂow quantization refers to the on-the-ﬂy quantization that occurs when data propagate through the network in low-precision training.
Revised: On the other hand, data ﬂow quantization refers to the on-the-ﬂy quantization that occurs when data propagate through the network in low-precision training.

Original: Examples include activation, error, and weight gradient quantizations.
Revised: Examples include activation, error, and weight gradient quantizations.

Original: Additional errors are introduced in weight update computation due to this type of quantization, which leads to performance degradation.
Revised: Additional errors are introduced in weight update computation due to this type of quantization, which leads to performance degradation.

Original: Hence, we need to ﬁnd an optimal format to minimize accuracy drop due to computation errors in data ﬂow quantization.
Revised: Hence, we need to ﬁnd an optimal format to minimize accuracy drop due to computation errors in data ﬂow quantization.

Original: In this paper, we present a systematic approach to implementing low-precision training on various models and tasks.
Revised: In this paper, we present a systematic approach to implementing low-precision training on various models and tasks.

Original: First, we present a method to efﬁciently ﬁnd an optimal format for data ﬂow quantization.
Revised: First, we present a method to efﬁciently ﬁnd an optimal format for data ﬂow quantization.

Original: In addition, we introduce a hysteresis quantization technique, a new quantization method for network quantization that can mitigate the issues of from-scratch training.
Revised: In addition, we introduce a hysteresis quantization technique, a new quantization method for network quantization that can mitigate the issues of from-scratch training.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We present a method that can predict the training performance of various numeric formats for data ﬂow quantization .
Revised: • We present a method that can predict the training performance of various numeric formats for data ﬂow quantization .

Original: This method allows us to determine an appropriate data format for different neural network structures, datasets, and tasks efﬁciently.
Revised: This method allows us to determine an appropriate data format for different neural network structures, datasets, and tasks efﬁciently.

Original: • Using the method above, we propose an optimal 8-bit format suitable for low-precision training of various models, which enables quantization of BatchNorm layer input and improves hardware efﬁciency with minimal performance degradation.
Revised: • Using the method above, we propose an optimal 8-bit format suitable for low-precision training of various models, which enables quantization of BatchNorm layer input and improves hardware efﬁciency with minimal performance degradation.

Original: • We propose a new quantization scheme that utilizes the hysteresis effect to improve the performance of from-scratch training.
Revised: • We propose a new quantization scheme that utilizes the hysteresis effect to improve the performance of from-scratch training.

Original: This scheme enables ultra-low-precision training using 4-bit logarithmic weights.
Revised: This scheme enables ultra-low-precision training using 4-bit logarithmic weights.

Original:  D ATA F LOW Q UANTIZATION 2. N UMERIC F ORMATS
Revised:  D ATA F LOW Q UANTIZATION 2. N UMERIC F ORMATS

Original: There are many numeric formats that can be constructed with n bits depending on how much dynamic range is required and how many valid bits are used for representing a value.
Revised: There are many numeric formats that can be constructed with n bits depending on how much dynamic range is required and how many valid bits are used for representing a value.

Original: For example, using 8 bits we could implement 8-bit ﬁxed point integer format, 8-bit ﬂoating-point formats such as FP152, FP143, and FP125 (FP1 xy represents 1 sign bit, x exponent bits, y mantissa bits), 8-bit posit format (Gustafson & Yonemoto, 2017), and 8-bit ﬂoat-ﬁx format (Han et al., 2019).
Revised: For example, using 8 bits we could implement 8-bit ﬁxed point integer format, 8-bit ﬂoating-point formats such as FP152, FP143, and FP125 (FP1 xy represents 1 sign bit, x exponent bits, y mantissa bits), 8-bit posit format (Gustafson & Yonemoto, 2017), and 8-bit ﬂoat-ﬁx format (Han et al., 2019).

Original: Since the diversity of formats that could be formulated using n bits is nearly unlimited, here we assume some constraints to limit candidates while still including widely used formats such as ﬁxed-point and ﬂoating-point formats as below:
Revised: Since the diversity of formats that could be formulated using n bits is nearly unlimited, here we assume some constraints to limit candidates while still including widely used formats such as ﬁxed-point and ﬂoating-point formats as below:

Original: • The MSB (Most Signiﬁcant Bit) is used as a sign bit and other bits represent magnitude.
Revised: • The MSB (Most Signiﬁcant Bit) is used as a sign bit and other bits represent magnitude.

Original: Accordingly, only symmetric formats  are considered.
Revised: Accordingly, only symmetric formats that have identical representable ranges for positive and negative numbers are considered.

Original: Two’s complement representation is slightly asymmetric, but it does not incur a signiﬁcant difference.
Revised: Two’s complement representation is slightly asym- metric since it can represent one more negative value, but it does not incur a signiﬁcant difference.

Original: • The number of valid bits of a larger value is greater than or equal to the number of valid bits of a smaller value.
Revised: • The number of valid bits of a larger value is greater than or equal to the number of valid bits of a smaller value.

Original: The valid bits stand for signiﬁcant digits in binary representation.
Revised: The valid bits stand for signiﬁcant digits in binary representation.

Original: • The ratio between consecutive representable values does not exceed 2.
Revised: • The ratio between consecutive representable values does not exceed 2.

Original: For example, the base- 4 logarithmic format is excluded.
Revised: For example, the base-4 logarithmic format is excluded.

Original: We could obtain 166 8-bit formats that meet these constraints.
Revised: We could obtain 166 8-bit formats that meet these constraints.

Original: Then, we reduce 1 and 2 valid bits in each format to obtain 7- and 6-bit formats, resulting in 498 formats in total.
Revised: Then, we reduce 1 and 2 valid bits in each format to obtain 7- and 6-bit formats, resulting in 498 formats in total.

Original: More information on the numeric formats considered in our experiments is provided in Appendix A.1.
Revised: More information on the numeric formats considered in our experiments is provided in Appendix A.1.

Original: 2. A CTIVATION AND E RROR Q UANTIZATION
Revised: 2. A CTIVATION AND E RROR Q UANTIZATION

Original: In a neural network consisting of n layers, the training process is described by
Revised: In a neural network consisting of n layers, the training process is described by

Original: [Equation]
Revised: [Equation]

Original: where A , E , W , and WG are activation, error, weight, and weight gradient, respectively.
Revised: where A , E , W , and G w are activation, error, weight, and weight gradient, respectively.

Original: f , g , h , and o are forward, backward, gradient, and update functions.
Revised: f , g , h , and o are forward, backward, gradient, and update functions.

Original: l and t represent the layer number and time step.
Revised: l and t represent the layer number and time step.

Original: We follow the quantized training scheme suggested by Fox et al.
Revised: We follow the quantized training scheme suggested by Fox et al.

Original: (2020), but
Revised: (2020), but

Original: with the following modiﬁcations to reduce hardware implementation costs.
Revised: with the following modiﬁcations to reduce hardware implementation costs.

Original: A and E are quantized not only for the GEMM input but also for the BatchNorm layer input.
Revised: A and E are quantized not only for the GEMM input but also for the BatchNorm layer input.

Original: BatchNorm layer normalizes input using the mean and variance of each channel, but these values are obtained only after observing all the inputs from the previous layer, necessitating that all input values are temporarily stored in memory.
Revised: BatchNorm layer normalizes input using the mean and variance of each channel, but these values are obtained only after observing all the inputs from the previous layer, necessitating that all input values are temporarily stored in memory.

Original: Therefore, quantizing the BatchNorm layer’s input signiﬁcantly reduces memory footprint and memory access overhead.
Revised: Therefore, quantizing the BatchNorm layer’s input signiﬁcantly reduces memory footprint and memory access overhead.

Original: Additionally, the scope of sharing exponent bias is extended to a layer ( A l and E l ) to avoid the overhead of aligning partial sums from different blocks in block-wise exponent sharing.
Revised: Additionally, the scope of sharing exponent bias is extended to a layer ( A l and E l ) to avoid the overhead of aligning partial sums from different blocks in block-wise exponent sharing.

Original: Finally, instead of determining the shared exponent bias by analyzing all values in the layer, we conservatively update it by detecting overﬂow and underutilization that occurred in the previous mini-batch.
Revised: Finally, instead of determining the shared exponent bias by analyzing all values in the layer, we conservatively update it by detecting overﬂow and underutilization that occurred in the previous mini-batch.

Original: [Figure]
Revised: [Figure]

Original: 2. I NDICATORS OF T RAINING P ERFORMANCE
Revised: 2. I NDICATORS OF T RAINING P ERFORMANCE

Original: Effect of quantized error :
Revised: Effect of quantized error :

Original: Quantizing the error E in the backward path is independent of how the forward path behaves since the loss surface of the model does not change.
Revised: Quantizing the error E in the backward path is independent of how the forward path behaves since the loss surface of the model does not change.

Original: Therefore, the optimal W that the network needs to reach through training remains the same regardless of the error quantization scheme.
Revised: Therefore, the optimal W that the network needs to reach through training remains the same regardless of the error quantization scheme.

Original: However, when the error is quantized, a quantization error ∆ E is introduced in E , which incurs a noise N ∆ E in WG through the gradient function in Eq.
Revised: However, when the error is quantized, a quantization error ∆ E is introduced in E , which incurs a noise N ∆ E in G w through the gradient function in Eq.

Original: 3 and potentially updates each weight in the wrong direction.
Revised: 3 and potentially updates each weight in the wrong direction.

Revised: While some amount of noise may improve the training performance through regularization, using low-precision formats already introduces a large noise in the network, incurring performance degradation (see Appendix A.8).

Original: Therefore, we suggest that the weight gradient error N ∆ E could be a good indicator of degradation in training performance.
Revised: Therefore, we suggest that the weight gradient error N ∆ E could be a good indicator of degradation in training performance.

Original: One way to implement this is predicting performance using the magnitude of N ∆ E ; however, if the noise is in the same direction as WG , it would only change the amount of each update and result in a less severe effect.
Revised: One way to implement this is predicting performance using the magnitude of N ∆ E ; however, if the noise is in the same direction as G w , it would only change the amount of each update and result in a less severe effect.

Original: Instead, we propose to measure the misalignment between WG + N ∆ E and WG for performance prediction.
Revised: Instead, we could measure the misalignment between G w + N ∆ E and G w for performance prediction.

Original: The misalignment between two vectors is estimated by
Revised: The misalignment between two vectors is estimated by

Original: [Equation]
Revised: [Equation]

Original: Then, the change in the update direction due to N ∆ E is ∠ ( WG, WG + N ∆ E ) .
Revised: Then, the change in the update direction due to N ∆ E is ∠ ( G w , G w + N ∆ E ) .

Original: We can expect that the smaller ∠ ( WG, WG + N ∆ E ) , the better the training performance.
Revised: We can expect that the smaller ∠ ( G w , G w + N ∆ E ) , the better the training performance.

Original: Effect of quantized activation : Contrary to error quantization, activation quantization affects the way the forward path operates, and the loss surface of the model changes.
Revised: Effect of quantized activation : Contrary to error quantization, activation quantization affects the way the forward path operates, and the loss surface of the model changes.

Original: Hence, the global optima of weight parameters shift, where the amount of shift would be proportional to the quantization noise.
Revised: Hence, the global optima of weight parameters shift, where the amount of shift would be proportional to the quantization noise.

Original: The displacement of global optima can be indirectly estimated using the direction of the weight gradients WG .
Revised: The displacement of global optima can be indirectly estimated using the direction of the weight gradients G w .

Original: If the angle ∠ ( WG, WG + N ∆ A ) is small, the deviation of the global optima is expected to be small as well, suggesting a better training performance.
Revised: If the angle ∠ ( G w , G w + N ∆ A ) is small, the deviation of the global optima is expected to be small as well, suggesting a better training performance.

Original: In the discussions above, we assumed that the angles ∠ ( WG, WG + N ∆ E ) and ∠ ( WG, WG + N ∆ A ) could be used to predict training performance.
Revised: In the discussions above, we assumed that the angles ∠ ( G w , G w + N ∆ E ) and ∠ ( G w , G w + N ∆ A ) could be used to predict training performance.

Original: We experimentally prove this by comparing the training performance of different numeric formats.
Revised: We experimentally prove this by comparing the training performance of different numeric formats.

Original: For 498 numeric formats in 6 to 8 bits, we compare the loss obtained from training with the proposed performance indicators. The results are summarized in Fig.
Revised: For 498 numeric formats in 6 to 8 bits, we compare the loss obtained from training with the proposed performance indicators (Fig. 2).

Original: Training loss is obtained by training ResNet-18 on CIFAR-10 dataset using 
Revised: Training loss is obtained by training ResNet-18 on CIFAR-10 dataset using SGD with a momentum of 0.

Original: SGD with a momentum of 0.9 for 60 epochs.
Revised: for 60 epochs.

Original: The batch size is 128 images and the initial learning rate is 0.1, which is decayed by a cosine scheduler.
Revised: The batch size is 128 images and the initial learning rate is 0.1, which is decayed by a cosine scheduler.

Original: Then, we measure each angle by processing 100 mini-batches on this pre-trained model.
Revised: We average angles from 100 mini-batches after quantizing a pre-trained model.

Original: Note that we use WG of the ﬁrst layer’s weight since it reﬂects the activation andquantization errors of all the layers in the network.
Revised: Note that we use G w of the ﬁrst layer since it can reﬂect quantization errors that occur in the activations and errors of all the layers in the network.

Revised: The weight gradients from the full-precision network, the network with quantized activations, and the network with quantized errors are G w ,

Revised: G w + N ∆ A , and G W + N ∆ E , respectively.

Original: Fig.
Revised: Fig.

Original: 2 shows that  the training performance has a higher Spearman’srank correlation with the misalignment angle than the magnitude of the error in the weight gradient, suggesting that the misalignment angle is a better metric to predict the ranking of various formats based on the training performance.
Revised: 2 shows that using the misalignment angle results in not only a higher Spearman’s correlation but also a more distinct shape for low training losses, making it a better metric for data format selection. For instance, using the error magnitude would predict the best format for transformer incorrectly (see Fig. 7(e) in Appendix A.3).

Revised: While obtaining the misalignment angle requires additional computations, its overhead is negligible since the part that requires the most time and computation is to obtain G w , G w + ∆ E , and G w + N ∆ A , which is still signiﬁcantly lower than actual training.

Original: Using this method, we could determine the optimal format for a speciﬁc neural network model, dataset, and task very efﬁciently as we only need to measure the misalignment angle between the two weight gradients without time-consuming network training.
Revised: Using this method, we could determine the optimal format for a speciﬁc neural network model, dataset, and task very efﬁciently as we only need to ∠ ( G W , G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 1. Fail (a) ResNet-18 (ImageNet) ∠ ( G W ,

Revised: G W + N ∆ A ) ∠ (G W , G W +N ∆E )

Revised: Fail (b) ResNet-101 (ImageNet) ∠ ( G W ,

Original: For experiments in Fig.
Revised: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 1. Fail (c) MobileNetV2 (ImageNet) ∠ ( G W , G W + N ∆ A ) ∠ (G W , G W +N ∆E ) [Figure]

Revised: [Figure]

Revised: layer LSTM (PTB) ∠ (

Revised: [Figure]

Revised: W ,

Original: 2, the amount of computation is reduced by 99.6%, and the reduction will be even larger for larger datasets and complex networks that need more epochs for training.
Revised: [Figure] N ∆ A ) ∠ (G W , G W +N [Figure] Fail (e) Transformer (IWLST) ∠ ( G W ,

Original: [Figure] [Figure] 1 +2m-1 bit α
Revised: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 2.

Revised: Fail (f) MobileNetV

Original: [Table]
Revised: [Figure]

Original: adder tree
Revised: SSDLite (VOC) measure the misalignment angle without time-consuming network training.

Original: [Table]
Revised: For experiments in Fig.

Original: [Table]
Revised: 2, the amount of computation is reduced by 99.6%, and the reduction will be even larger for larger datasets and complex networks that need more epochs for training.

Original: 2. O PTIMAL F ORMAT FOR D ATA F LOW Q UANTIZATION
Revised: 2. O PTIMAL F ORMAT FOR D ATA F LOW Q UANTIZATION

Original: Using the proposed performance estimation method above, here we  ﬁnd an optimal formatsuitable for training various models.
Revised: Here we show that we could ﬁnd an optimal format for training speciﬁc models using the proposed performance estimation method above.

Original: We select ﬁve models widely used in quantized training research for experiments: ResNet-18,  MobileNetV2 (Sandler et al., 2018), 2-layer LSTM, small transformer for translation on the IWSLT German to English dataset (Cettolo et al., 2014), and SSD-Lite (Liu et al., 2016) with MobileNetV2.
Revised: We select six models widely used in quantized training research for experiments: ResNet-18, ResNet-101, MobileNetV2 (Sandler et al., 2018), 2-layer LSTM, small transformer for translation on the IWSLT German to English dataset (Cettolo et al., 2014), and SSD-Lite (Liu et al., 2016) with MobileNetV2.

Original: We ﬁrst measure misalignment angles( ∠ ( WG, WG + N ∆ E ) and ∠ ( WG, WG + N ∆ A ) ) for 166 8-bit formats.
Revised: We ﬁrst measure misalignment angles for 166 8-bit formats.

Original: To verify the correlation between the training performance and the misalignment angles, we select four formats that exhibit low hardware implementation costs (INT8, FP152, FP143, and FP134) and train the networks using each format.
Revised: To verify the correlation between the training performance and the misalignment angles, we select four formats that exhibit low hardware implementation costs (INT8, FP152, FP143, and FP134) and train the networks using each format.

Original: While we may use different formats for activation and error, it requires a complicated datapath (Sun et al., 2019) and hence we only consider a single format for both variables.
Revised: While we may use different formats for activation and error, it requires a complicated datapath (Sun et al., 2019) and hence we only consider a single format for both variables.

Original: The experimental results in 
Revised: The experimental results in Fig.

Revised: 3 demonstrate that the training performance is higher if both misalignment angles are small in all tasks and models, conﬁrming that the proposed indicators could be used to determine the optimal numeric format.

Original: Fig.
Revised: Fig.

Original: 3 demonstrate that the training performance is higher if both misalignment angles are small in all tasks and models, conﬁrming that the proposed indicators could be used to determine the  optimal numeric format.
Revised: 3 suggests that FP134 and FP143 are the best candidates across all models. For hardware implementation, FP is the most optimal format due to its low implementation cost, which is discussed in Section 4.

Original: w
Revised: in detail.

Original: Qw       w
Revised: Note that using the error magnitude leads to the same conclusion that FP134 is the best format for the target models. See Appendix A.3 for more details.  N ETWORK Q UANTIZATION In quantized neural networks, the weight parameters are generally quantized in a way that minimizes the quantization error (Choi et al., 2019; Martinez et al., 2020).

Original: Qw
Revised: For instance, if x is quantized into a ﬁxed-point format through s (cid:98) xs (cid:101) , a proper value is selected for the scaling factor s to minimize the quantization error. However, as the weights continue to change during training, we need to calculate s for every update, which could cause signiﬁcant overhead. Therefore, prior studies on low-precision training suggest constraining the scaling factor to the power of 2 in the shared exponent (K¨oster et al., 2017) or the shared exponent bias (Fox et al., 2020). In this section, we analyze the issues behind weight quantization and propose a new quantization scheme to mitigate those issues.

Original: Q w    w t
Revised: w Qw       w Qw   w t

Revised: Q w Small ∆ w Large ∆ w w t

Original: Q w  wt (a) Conventional quantization      w
Revised: Q w Small ∆ w Large ∆ w (a) Conventional quantization      w

Original: Qw decreasing path increasing path      w
Revised: Qw decreasing path increasing path      w

Original: Qw decreasing path increasing path
Revised: Qw decreasing path increasing path

Original: Q ww t
Revised: Q w t

Original: Q w wt [Figure] From the results in Fig. 3, we could conclude that FP134 and FP143 are the best options across all models.
Revised: Small ∆ w Large ∆ w w

Original: Although they both use 8 bits, their hardware implementation costs may differ signiﬁcantly. For comparisons, we implemented a MAC unit using those formats as depicted in Fig. 4. For accumulation, we use FP169 which was conﬁrmed to exhibit no degradation through chunk-based accumulation (Wang et al., 2018).
Revised: Q w t

Original: In a conventional MAC unit with a single multiplier, the power and area are dominated by the 16-bit accumulation block, undermining the advantages of using a low-precision data format. Instead, we adopt a multi-way MAC unit with an adder tree proposed by
Revised: Small ∆ w Large ∆ w w

Original: Park et al. (2021) to minimize hardware overhead (Fig. 4). The outputs of multipliers are combined through integer addition, and only one ﬂoating-point addition is required to convert the combined value to a partial sum in FP169. In this structure, a smaller dynamic range is advantageous since all multiplication results need to be aligned and summed, and shifter overhead increases with the dynamic range.
Revised: [Figure]

Original: As a result, FP134 format exhibits the lowest cost as shown in Table 1. Therefore, we suggest that the FP134 format is the optimal format for low-precision training as it results in good training performance in the ﬁve models and low hardware cost.  N ETWORK Q UANTIZATION In quantized neural networks, the weight parameters are generally quantized in a way that minimizes the quantization error (Choi et al., 2019; Martinez et al., 2020).
Revised: Hysteresis

Original: For instance, if x is quantized into a ﬁxed-point format through s (cid:98) xs (cid:101) , a proper value is selected for the scaling factor s to minimize the quantization error. However, as the weights continue to change during training, we need to calculate s for every update, which could cause signiﬁcant overhead. Therefore, prior studies on low-precision training suggest constraining the scaling factor to the power of 2 in the shared exponent (K¨oster et al., 2017) or the shared exponent bias (Fox et al., 2020). In this section, we analyze the issues behind weight quantization and propose a new quantization scheme to mitigate those issues.
Revised: [Figure]

Original: 3. F LUCTUATION OF W EIGHT P ARAMETERS
Revised: 3. F LUCTUATION OF W EIGHT P ARAMETERS

Original: In typical low-precision training, a master copy of weight parameters is separately maintained in high precision, and those weights are updated based on the computed weight gradient.
Revised: In typical low-precision training, a master copy of weight parameters is separately maintained in high precision, and those weights are updated based on the computed weight gradient.

Original: This highprecision weight is quantized into a low-precision format and used for the forward path computation during training.
Revised: This highprecision weight is quantized into a low-precision format and used for the forward path computation during training.

Original: Since the scaling factor s is constrained to 2 n , the quantization threshold remains the same unless s is updated due to overﬂow or underutilization.
Revised: Since the scaling factor s is constrained to 2 n , the quantization threshold remains the same unless s is updated due to overﬂow or underutilization.

Original: If the optimal weight is located between two representable values of a data format, the quantized weight would ﬂuctuate alternately between the tworepresentable values in each update (Fig. 5(a)). Note that the amount of change in the quantized weight due to this ﬂuctuation is not necessarily proportional to the weight gradient. If the master weight is located close to the quantization threshold, even  a very small weight update may cause large ﬂuctuations, undermining training performance.
Revised: If the optimal weight is located between two representable values of a data format, the quantized weight would ﬂuctuate alternately between the two values in each update (Fig. 4(a)) even for a very small weight update, causing large ﬂuctuations and undermining training performance.

Original: 3. H YSTERESIS Q UANTIZATION
Revised: 3. H YSTERESIS Q UANTIZATION

Original: To mitigate the ﬂuctuation issue above, we propose to introduce the concept of hysteresis to quantization.
Revised: To mitigate the ﬂuctuation issue above, we propose to introduce the concept of hysteresis to quantization.

Original: More speciﬁcally, we quantize each weight differently depending on the direction of the weight update.
Revised: More speciﬁcally, we quantize each weight differently depending on the direction of the weight update.

Original: Fig. 5(b) describes the proposed hysteresis quantization scheme.The quantization [Figure]
Revised: Fig. 4(b) describes the proposed hysteresis quantization scheme.

Original:  threshold differs when the master weight increases and decreases.
Revised: The quantization threshold differs when the master weight increases and decreases.

Original: For instance, we could round down the weight for a positive weight update, and round it up for a negative weight update.
Revised: For instance, we could round down the weight for a positive weight update, and round it up for a negative weight update.

Original: The equation below describes the proposed quantization scheme.
Revised: The equation below describes the proposed quantization scheme.

Original: [Equation]
Revised: [Equation]

Original:  where w is the original value, Q w is its quantized value, and t is the time step.
Revised: ) where w is the original value, Q w is its quantized value, and t is the time step.

Original: The proposed hysteresis quantization reduces ﬂuctuation signiﬁcantly, stabilizing the training process and allowing the network to reach global optima more efﬁciently.
Revised: The proposed hysteresis quantization reduces ﬂuctuation signiﬁcantly, stabilizing the training process and allowing the network to reach global optima more efﬁciently.

Original: In addition, even if the weightﬂuctuates, the weight change  is now proportional to the weight update since enough amount of weight update should be accumulated to change the quantized weight.
Revised: In Fig. 4(b), if the weight change ∆ W is small, then enough number of those changes should be accumulated to ﬂip Q w . Hence, the update frequency is now proportional to the weight gradient.

Original: This helps the network to learn better while suppressing ﬂuctuations for small WG values.
Revised: This helps the network to learn better while suppressing ﬂuctuations for small G w values.

Revised: Alternatively, we may mitigate weight quantization errors by adopting AdaRound (Nagel et al.

Revised: (2020)), which learns whether each weight should be rounded up or down to produce the same output as high-precision weights.

Revised: However, whenever full-precision weights are updated, we need to re-train the learnable parameters (i.e., quantization scheme of each weight), incurring a large overhead and undermining the beneﬁt of low-precision training.

Original: 3. U LTRA -L OW -P RECISION F ORMAT FOR N ETWORK Q UANTIZATION
Revised: 3. U LTRA -L OW -P RECISION F ORMAT FOR N ETWORK Q UANTIZATION

Original: To verify the effectiveness of the proposed hysteresis quantization, we select 4-bit logarithmic representation as an ultra-low-precision format for weight parameters.
Revised: To verify the effectiveness of the proposed hysteresis quantization, we select 4-bit logarithmic representation as an ultra-low-precision format for weight parameters.

Original: This format has the same dynamic range as INT8 which is widely used for weight quantization, and is more hardware-efﬁcient as multiplication is implemented only using simple shift operations.
Revised: This format has the same dynamic range as INT8 which is widely used for weight quantization, and is more hardware-efﬁcient as multiplication is implemented only using simple shift operations.

Original: There have been attempts to use logarithmic weights in quantized neural networks (Lee et al., 2017; Elhoushi et al., 2021), but from-scratch training shows a signiﬁcant performance degradation.
Revised: There have been attempts to use logarithmic weights in quantized neural networks (Lee et al., 2017; Elhoushi et al., 2021), but from-scratch training shows a signiﬁcant performance degradation.

Original: In logarithmic data formats, the interval of quantization points is not uniform, making them more sensitive to ﬂuctuation.
Revised: In logarithmic data formats, the interval of quantization points is not uniform, making them more sensitive to ﬂuctuation.

Original: Fig.
Revised: Fig.

Original: 6 shows experimental results of ResNet-18 training on ImageNet using 4-bit logarithmic weights.
Revised: 5 shows experimental results of ResNet-18 training on ImageNet using 4-bit logarithmic weights.

Original: Note that we apply channel-wise quantization to the convolutional layers to compensate for  the insufﬁcient expression range and layer-wise quantization to other types of layers.
Revised: Note that we apply channel-wise quantization to the convolutional layers to compensate for [Figure] the insufﬁcient expression range and layer-wise quantization to other types of layers.

Original: Further details on the experimental setup are provided in Appendix A.4.1.
Revised: Further details on the experimental setup are provided in Appendix A.5.1.

Original: First, we measure how many quantized weights Q w change when the network performs one weight update using a mini-batch and average them over the ﬁrst 100 updates in the 60th epoch.
Revised: First, we measure how many quantized weights Q w change when the network performs one weight update using a mini-batch and average them over the ﬁrst 100 updates in the 60th epoch.

Original: The experimental result displayed in Fig.
Revised: The experimental result displayed in Fig.

Original: 6(a) clearly shows that using hysteresis signiﬁcantly reduces weight change frequency and stabilizes the training process.
Revised: 5(a) clearly shows that using hysteresis signiﬁcantly reduces weight change frequency and stabilizes the training process.

Original: Fig.
Revised: Fig.

Original: 6(b) compares the training performance of quantization with and without hysteresis.
Revised: 5(b) compares the training performance of quantization with and without hysteresis.

Original: Hysteresis quantization not only speeds up training but also achieves better results at the end of training.
Revised: Hysteresis quantization not only speeds up training but also achieves better results at the end of training.

Revised: Note that hysteresis quantization is applicable to other data formats, and additional experimental results can be found in Appendix A.4.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we experimentally show that the proposed training performance prediction method and hysteresis quantization scheme enable efﬁcient low-precision training.[Table]
Revised: In this section [Table] we experimentally show that the proposed training performance prediction method and hysteresis quantization scheme enable efﬁcient low-precision training.

Original: [Table]
Revised: [Table]

Original: 1 Fixed Point 4. 8- BIT L OW -P RECISION T RAINING
Revised: RECISION T RAINING

Original: In Section 2.4, we found that FP134 is the optimal format for low-precision training using the proposed performance prediction method.
Revised: In Section 2.4, we found that FP134 is the optimal format for the target models using the proposed performance prediction method.

Original: We measure the training performance of this format and compare it against other 8-bit data formats from recent studies by applying those formats to the training of various neural network models.
Revised: We measure the training performance of this format and compare it against other 8-bit data formats from recent studies by applying those formats to the training of various neural network models.

Original: More details on the experimental setup are provided in 
Revised: More details on the experimental setup are provided in Appendix

Original: Appendix A.4.
Revised: A.5.

Original: The performance of the proposed data format is summarized in Table 2.
Revised: The performance of the proposed data format is summarized in Table 1.

Original: Overall, 8-bit training using FP134 achieves nearly the same performance as the full-precision training on all models.
Revised: Overall, 8-bit training using FP134 achieves nearly the same performance as the full-precision training on all models.

Original: Even in MobileNetV2, which is known to be sensitive to quantization due to the small number of parameters, only 0.3% degradation occurred.
Revised: Even in MobileNetV2, which is known to be sensitive to quantization due to the small number of parameters, only 0.3% degradation occurred.

Original: Sun et al.
Revised: Sun et al.

Original: (2019) show that HFP8 also exhibits only 0.2% accuracy degradation in MobileNetV2 (71.81% vs. 71.61%), but they quantize BatchNorm input into 16 bits instead of 8 bits, roughly doubling the memory access and computational complexity.
Revised: (2019) show that HFP8 also exhibits only 0.2% accuracy degradation in MobileNetV2 (71.81% vs. 71.61%), but they quantize BatchNorm input into 16 bits instead of 8 bits, roughly doubling the memory access and computational complexity.

Original: Additionally, since the forward and backward paths employ different data formats, 
Revised: Additionally, since the forward and backward paths employ different data formats, HFP8 is [Table] Fixed Point [Table]

Revised: [Table]

Revised: input

Original: HFP8 is actually implemented using 9-bit MAC units in hardware (Agrawal et al., 2021).
Revised: [Table] input [Table] input

Revised: [Table]

Revised: input

Revised: [Table]

Revised: input

Revised: [Table]

Revised: oster

Revised: [Table]

Revised: 2017) actually implemented using 9-bit MAC units in hardware (Agrawal et al., 2021).

Original: Tablecompares the training performance of various data formats for ResNet-18 training.
Revised: Table 2 compares the training performance of various data formats for ResNet-18 training.

Revised: The columns w, x, dw, dx, and acc refer to weight, activation, weight gradient, error, and GEMM accumulation, respectively.

Original: Our FP134 format exhibits no accuracy drop compared to full-precision training.
Revised: Our FP134 format exhibits no accuracy drop compared to full-precision training.

Original: HFP8 (Sun et al., 2019) and BM (Fox et al., 2020) demonstrate similar performance, but they both use higher precision to represent BatchNorm inputs, and different formats are adopted in the forward and backward paths, necessitating complex computation units when implemented in hardware, as decribed above.
Revised: HFP8 (Sun et al., 2019) and BM8 (Fox et al., 2020) demonstrate similar performance, but they both use higher precision to represent BatchNorm inputs, and different formats are adopted in the forward and backward paths, necessitating complex computation units when implemented in hardware, as decribed above.

Original: In addition, BM8 assumes layer-wise sharing of exponent bias, incurring additional overhead in memory access and data alignment.
Revised: In addition, BM8 assumes block-wise sharing of exponent bias, incurring additional overhead in memory access and data alignment.

Original: FP8-SEB
Revised: FP8-SEB

Original: (Park et al., 2021) addresses this issue by employing layer-wise exponent bias sharing and multi-way MAC units, but it results in a 0.7% accuracy drop for ResNettraining.
Revised: (Park et al., 2021) addresses this issue by employing layer-wise exponent bias sharing and multi-way MAC units, but it results in a 0.7% accuracy drop for ResNet-18 training.

Original: Contrarily, our data format shows no performance degradation, while deeply quantizing 
Revised: Contrarily, our data format shows no performance degradation, while deeply quantizing BatchNorm inputs into the same format and allowing for a simple datapath by using an identical data format in the forward and backward paths. 4. H ARDWARE E VALUATION

Original: BatchNorm inputs into the same format andallowing for a simple datapath by using an identical data format in the forward and backward paths.
Revised: For hardware implementation cost comparisons, we implemented a conventional MAC unit and a multi-way MAC unit with integer-based accumulation (Tambe et al., 2020; Park et al., 2021) that support data formats presented in Section 4.1. For accumulation, we use FP169 with chunk-based accumulation (Wang et al., 2018). Experimental results in Table 3 show that FP134 exhibits lower cost than FP143 and other formats in previous studies.

Original: 4. U LTRA -L OW -P RECISION T RAINING WITH 4- BIT L OGARITHMIC W EIGHTS
Revised: Note that HFP8 (Sun et al., 2019) and BM (Fox et al., 2020) employ different formats for activation and error. Therefore, they need to be implemented in FP153 and FP145 to support all operations with a single MAC unit (Agrawal et al., 2021). Since Flex16+5 (K¨oster et al., 2017) requires 16-bit multiplication, its cost is signiﬁcantly higher than other 8-bit formats. More detailed analysis is provided in Appendix A.7.

Revised: -L OW -P RECISION T RAINING WITH 4- BIT L OGARITHMIC W

Original: Elhoushi et al.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: LSTM (PTB) [ppl.]

Revised: [Table]

Revised: Transformer (IWSLT)

Revised: [BLEU]

Revised: [Table]

Revised: MobileNetV

Revised: [Table]

Revised: SSDLite (VOC) [

Revised: [Table]

Revised: [Table]

Revised: Lower

Original:  (2021) recently demonstrated that 4-bit logarithmic weights could be used in quantized neural networks with minimal accuracy degradation.
Revised: [Table] better Elhoushi et al. (2021) recently demonstrated that 4-bit logarithmic weights could be used in quantized neural networks with minimal accuracy degradation.

Original: However, from-scratch training of the same model resulted in a 4.5% accuracy drop in ResNet-18 training (Table 4).
Revised: However, from-scratch training of the same model resulted in a 4.5% accuracy drop in ResNet-18 training (Table 4).

Original: Similarly, our experiments show 2.1% lower accuracy when training ResNet-18 using 4-bit logarithmic weights and[Table] [Table] Transformer (IWSLT) [
Revised: Similarly, our experiments show 2.1% lower accuracy when training ResNet-18 using 4-bit logarithmic weights and

Original: [Table]

Original: [Table]

Original: FP134 format for other variables.
Revised: FP134 format for other variables.

Original: However, using hysteresis quantization greatly improves the training performance, showing only 0.2% accuracy degradation.
Revised: However, using hysteresis quantization greatly improves the training performance, showing only 0.2% accuracy degradation.

Original: This is identical to the training performance achieved through ﬁne-tuning a pre-trained model by Elhoushi et al.
Revised: This is identical to the training performance achieved through ﬁne-tuning a pre-trained model by Elhoushi et al.

Original: (2021), conﬁrming that hysteresis quantization effectively solves the issue of sub-optimal solutions in from-scratch training.
Revised: (2021), conﬁrming that hysteresis quantization effectively solves the issue of sub-optimal solutions in from-scratch training.

Original: In addition, Table 5 demonstrates that hysteresis quantization improves the training performance in all target models.
Revised: In addition, Table 5 demonstrates that hysteresis quantization improves the training performance in all target models.

Original: Note that we quantized all trainable weights except for the BatchNorm parameters into 4 bits in experiments; the training performance could be further improved by using higher precision for error-sensitive parts such as the ﬁrst/last layers and residual connections.
Revised: Note that we quantized all trainable weights except for the BatchNorm parameters into 4 bits in experiments; the training performance could be further improved by using higher precision for error-sensitive parts such as the ﬁrst/last layers and residual connections.

Original: We also tested hysteresis quantization on the FP134 training experiments from Section 4.1, but there was no difference in performance across the ﬁve models.

Original: We suspect that weight ﬂuctuation is less severe in FP134 as it provides a much higher precision than 4-bit logarithmic representation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In low-precision training, the dynamic range of a tensor is data-dependant, and hence an optimal data format depends on various factors such as model, dataset, and quantization scheme.
Revised: In low-precision training, the dynamic range of a tensor is data-dependant, and hence an optimal data format depends on various factors such as model, dataset, and quantization scheme.

Original: We showed that the training performance of a speciﬁc data format  could be predicted by observing the errors introduced in the weight gradients.
Revised: We showed that the training performance of a speciﬁc data format for activation and error could be predicted by observing the errors introduced in the weight gradients.

Original: Based on this observation, we could determine an optimal 8bit format for low-precision training very efﬁciently without running numerous training runs.
Revised: Based on this observation, we could determine an optimal 8-bit format for low-precision training very efﬁciently without running numerous training runs.

Original: The proposed FP134 format achieved a similar or better accuracy compared to prior works, while allowing for efﬁcient hardware implementation through quantizing BatchNorm inputs and using a uniﬁed data format in both forward and backward paths.
Revised: The proposed FP134 format achieved a similar or better accuracy compared to prior works, while allowing for efﬁcient hardware implementation through quantizing BatchNorm inputs and using a uniﬁed data format in both forward and backward paths.

Original: In addition, it was demonstrated that frequent ﬂuctuations in weight parameters hinder reliable training with deeply quantized weights.

Original: We proposed the hysteresis quantization scheme, which improves training performance by suppressing undesired weight ﬂuctuations and stabilizing the training process.
Revised: In addition, we proposed the hysteresis quantization scheme for weight quantization, which improves training performance by suppressing undesired weight ﬂuctuations and stabilizing the training process.

Original: In ultra-low-precision training with 4-bit logarithmic weights, hysteresis quantization signiﬁcantly improves training performance by mitigating sub-optimal solutions, closely matching the performance obtained through ﬁne-tuning a pre-trained model. 
Revised: In ultra-low-precision training with 4-bit logarithmic weights, hysteresis quantization signiﬁcantly improves training performance by mitigating sub-optimal solutions, closely matching the performance obtained through ﬁne-tuning a pre-trained model. We expect that these two schemes can complement each other to enable practical low-precision training on various models and tasks.

Original: A A PPENDIX A. V ARIOUS F ORMATS A NALYZED IN S ECTION
Revised: A A PPENDIX A. V ARIOUS F ORMATS A NALYZED IN S ECTION

Original: In this paper, we made three assumptions on the quantization formats that were analyzed.
Revised: In this paper, we made three assumptions on the quantization formats that were analyzed.

Original: Firstly, 1-bit is allocated as a sign bit, so only symmetric formats are allowed, and secondly, the number of valid bits with a large absolute numerical value must be greater than or equal to the number of valid bits with a small absolute numerical value.
Revised: Firstly, 1-bit is allocated as a sign bit, so only symmetric formats are allowed, and secondly, the number of valid bits with a large absolute numerical value must be greater than or equal to the number of valid bits with a small absolute numerical value.

Original: Lastly, the base does not exceed 2.
Revised: Lastly, the base does not exceed 2.

Original: Considering the above assumptions, we provide a systematical approach for generating different quantization methods that were used for analysis in Section 2, in order to create quantization methods that have trade-offs in terms of dynamic range and the number of valid bits.
Revised: Considering the above assumptions, we provide a systematical approach for generating different quantization methods that were used for analysis in Section 2, in order to create quantization methods that have trade-offs in terms of dynamic range and the number of valid bits.

Original: In these quantization methods, we divide the data interval every 2 n (where n is an integer) and deﬁne the format by allocating valid bits to each interval. Since the entire interval can be shifted by the scaling factor, the format can be stated by arranging valid bits from the interval with the largest value to the interval  with  the smallest value without specifying n .
Revised: The quantization method is expressed with the following items: i) a list of decreasing positive real numbers P that contains the interval points (Eq. 7) and ii) a non-increasing integer list L that accompanies the interval list, with each item representing the number of signiﬁcand bits (Eq. 8). Here, s is shared exponent bias.

Original: The number of intervals to which valid bits are allocated becomes the dynamic range.
Revised: [Equation]

Original: When α -bit is given as a format to express data, there are a total of 2 α distinct numerical values that can be expressed in this format. Here, 1-bit is a sign bit, so the positive numerical value that can be expressed is 2 α − 1 .
Revised: The quantization points Q are generated in each of the intervals that are sliced with 2 l k − 1 evenly distributed datapoints. If the interval is { 2 s +1 , 2 s } , the quantization point Q can be expressed by

Original: Now, consider this value as the budget we can use.
Revised: Eq. 9. [Equation]

Original: If the number of valid bits for each interval is selected as m 1 , m 2 , .., m β , the following expression must be satisﬁed.
Revised: Notice that L for an α -bit quantization must satisfy

Original: [Equation]
Revised: [Equation]

Original: Since the leading digit is ﬁxed to 1, the number ofdistinct numerical values required when the valid bit is m is 2 m − 1  .
Revised: Since the format is symmetric, only half of the data points are assigned to positive numbers, so it is α − 1 , not α .

Original: Also, due to the previous constraint, m in the small interval must be less than or equal to m in the large interval. The list of valid bits determined for each interval deﬁnes a format, and we denote the valid bits in the order from the number of valid bits in the interval with the largest value to the interval with the smallest value. For example, if 8 bits are given, the number of valid bits can be selected as [7,6,5,4,3,2,1] (which translates to  a ﬁxed-point quantization without any exponents) or [6,6,6,5,4,3,2,1].
Revised: The reason for adding 1 is to include a zero value.

Original: When various 8-bit data representation formats are expressed as a list of the number of valid bits, it is as shown in Fig.
Revised: For example, when shared exponent bias is -1, an 8-bit ﬁxed-point quantization would be expressed as follows:

Original: [Figure]
Revised: [Equation]

Revised: The ﬁrst interval from 1 to 0.5 would be evenly sliced by 2 7 − 1 datapoints, the next interval from 0.

Revised: to 0.25 with 2 6 − 1 , etc.

Revised: Various cases are shown in Fig.

Revised: 6, with P plotted on the x-axis and L plotted on the y-axis.

Revised: Since P represents the range of values due to shared exponent bias that is independent of the data format, L can represent all of the various data formats we deal with in this paper.

Revised: [Figure]

Original: When selecting 8-bit formats, we chose the formats so that the intervals with the number of valid bits that are less than 3 do not appear for more than two digits to reduce the search space, as they have an unnecessarily large dynamic range.
Revised: When selecting 8-bit formats, we chose the formats so that the intervals with the number of valid bits that are less than 3 do not appear for more than two digits to reduce the search space, as they have an unnecessarily large dynamic range.

Original: Thus, formats such as [7,6,5,4,2,2,2,1] were excluded from the search space.
Revised: Thus, formats such as [7,6,5,4,2,2,2,1] were excluded from the search space.

Original: Considering all of the generation rules, we selected 166 distinct 8-bit formats with different dynamic range and valid bits from [7,6,5,4,3,2,1] to [3,3,3,...,3,2,1].
Revised: Considering all of the generation rules, we selected 166 distinct 8-bit formats with different dynamic range and valid bits from [7,6,5,4,3,2,1] to [3,3,3,...,3,2,1].

Original: After the number of valid bits for an 8-bit format is selected, 1 or 2 is subtracted from each value to create a corresponding 7-bit and 6-bit formats.
Revised: After the number of valid bits for an 8-bit format is selected, 1 or 2 is subtracted from each value to create a corresponding 7-bit and 6-bit formats.

Original: For example, in the case of [6,5,5,5,5,4,4,4,3,2,1] 8-bit format, the 7-bit corresponding format is [5,4,4,4,4,3,3,3,2,1] and 6-bit corresponding format is [4,3,3,3,3,2,2,2,1].
Revised: For example, in the case of [6,5,5,5,5,4,4,4,3,2,1] 8-bit format, the 7-bit corresponding format is [5,4,4,4,4,3,3,3,2,1] and 6-bit corresponding format is [4,3,3,3,3,2,2,2,1].

Original: From the generated 166 8-bit formats, 7-bit and 6-bit formats were also generated using this rule.
Revised: From the generated 166 8-bit formats, 7-bit and 6-bit formats were also generated using this rule.

Original: A. S OFTWARE I MPLEMENTATION D ETAILS
Revised: A. S OFTWARE I MPLEMENTATION D ETAILS

Original: To support quantized training for various formats, custom C++ and CUDA codes to emulate quantized data were written.
Revised: To support quantized training for various formats [Figure] custom C++ and CUDA codes to emulate quantized data were written.

Original: Our custom C++ and CUDA extension code could perform quantizationrelated functions through utilizing the Python APIs in PyTorch for extendable research while maintaining high performance.
Revised: Our custom C++ and CUDA extension code could perform quantizationrelated functions through utilizing the Python APIs in PyTorch for extendable research while maintaining high performance.

Original: We emulate the quantized value using custom code in the part that needs quantization throughout the network, and PyTorch built-in functions are used for computation kernels such as convolution and matrix multiplication.
Revised: We emulate the quantized value using custom code in the part that needs quantization throughout the network [Figure] and PyTorch built-in functions are used for computation kernels such as convolution and matrix multiplication.

Original: We created a package named lptorch, short for low precision PyTorch, and the code can be found in the supplementary material.
Revised: We created a package named lptorch, short for low precision PyTorch [Figure] and the code can be found in the supplementary material [Figure]

Original: A. A NGLE VS . A BSOLUTE V ALUE TO P REDICT P ERFORMANCE
Revised: A. A NGLE VS . M AGNITUDE TO P REDICT P ERFORMANCE

Original: [Figure]
Revised: [Figure]

Original: In addition to the misalignment angles of WG ( ∠ ( WG, WG + N ∆ A ) and ∠ ( WG, WG + N ∆ E ) ), as deﬁned in Section 2.3, we used the absolute values of noise ( | N ∆
Revised: In addition to the misalignment angles of G w ( ∠ ( G w , G w + N ∆ A ) and ∠ ( G w , G w + N ∆ E ) ), as deﬁned in Section 2.3, we used the absolute values of noise ( | N ∆

Original: A | and | N ∆ E | ) in order to predict the ﬁnal trained performance, and the results are shown in Fig. 8.
Revised: A | and | N ∆ E | ) in order to predict the ﬁnal trained performance, and the results are shown in Fig. Fig.

Original: In Fig. 8(d), although
Revised: 3 and Fig. 7 show that both the error magnitude and the misalignment angle are good metrics for determining optimal data format. For the six target models, both metrics suggest FP134 as the best format. However, the misalignment angle still better captures the training performance.

Original:  FP134 shows smaller noise in terms of absolute value, the actual training loss is smaller for FP143.
Revised: For instance, in Fig. 7(e), although FP134 shows smaller noise magnitude, the actual training loss is smaller for FP143.

Original: Similarly, in Fig. 8(b) and (e), although INT8 failed and FP152 succeeded in training, the absolute value of noise did not indicate a clear superior of the two formats.
Revised: Similarly, in Fig. 7(b), (c) and (f), although INT8 failed and FP152 succeeded in training, the absolute value of noise did not indicate a clear superior of the two formats.

Original: Based on these observations, we conclude that the misalignment angles are more suitable for predicting training performance compared against using the absolute value of noise.
Revised: Based on these observations, we conclude that the misalignment angles are more suitable for predicting training performance compared against using the absolute value of noise.

Original: A. E XPERIMENTAL D ETAILS A.4. R ES N ET -18 (I MAGE N ET )
Revised: A. H YSTERESIS Q UANTIZATION WITH I NTEGER W EIGHTS

Revised: In addition to 4-bit logarithmic weights, we also tested the hysteresis quantization scheme on a lowprecision integer format (INT4) that uses uniform quantization. The results are shown in Table 6. Experimental results show that using hysteresis improves the performance in most cases.

Revised: In addition, in MobileNetV2 training with INT4 weights, training initially failed, but using hysteresis enables reliable training, which suggests that hysteresis quantization not only helps the network to reach the optimal point but also prevents divergence in an unwanted direction during the training process.

Revised: However, it is interesting to see that the hysteresis quantization is less effective on the LSTM model for the INT4 format.

Revised: We suspect that this is due to the weight distribution characteristics of the

Revised: Model

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ppl

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: LSTM model.

Revised: As shown in Fig.

Revised: 8, most of the weights have a relatively large magnitude in the

Revised: LSTM model when normalized, contrary to ResNet-18 in which the weights are more evenly distributed.

Revised: In logarithmic formats, the relative amount of quantization error is similar for all values.

Revised: In contrast, the relative amount of quantization error is smaller for large values in uniform quantization.

Revised: Therefore, the weight parameters of LSTM are more severely affected by ﬂuctuation in logarithmic formats, making our hysteresis quantization scheme more effective in those formats compared to uniform quantization.

Revised: A.

Revised: E XPERIMENTAL D ETAILS

Original:  We conducted ImageNet experiments using SGD with a momentum of 0.9 for 90 epochs with a batch size of 256 images and an initial learning rate of 0.1 which is decayed by a factor of 10 at the 30th and 60th epochs.
Revised: A.5. [Equation] 18 (I MAGE N ET ) We conducted ImageNet experiments using SGD with a momentum of 0.9 for 90 epochs with a batch size of 256 images and an initial learning rate of 0.1 which is decayed by a factor of 10 at the 30th and 60th epochs.

Original: We used the ResNet-18 architecture from the ofﬁcial PyTorch implementation 1 .
Revised: We used the ResNet-18 architecture from the ofﬁcial PyTorch implementation 1 .

Original: Fig.shows Top-1 training & validation accuracy graphs.
Revised: Fig.shows Top-1 training & validation accuracy graphs.

Original: Observation of the training graph indicates that all of the results are close to the baseline within 0.2% with the exception of FP130 without hysteresis quantization. 
Revised: Observation of the training graph indicates that all of the results are close to the baseline within 0.2% with the exception of FP130 without hysteresis quantization. [Figure]

Revised: We trained ResNet-101 by applying the same training method as ResNet-18.

Original: A.4. M OBILE N ET V2 (I MAGE N ET )
Revised: We conducted ImageNet experiments using SGD with a momentum of 0.9 for 90 epochs with a batch size ofimages and an initial learning rate of 0.1 which is decayed by a factor of 10 at the 30th and 60th epochs. We used the ResNet-101 architecture from the ofﬁcial PyTorch implementation 2 . Fig.shows Top-1 training & validation accuracy graphs. Observation of the training graph indicates that all of the results are close to the baseline with less than 0.3% performance drop except for FP without hysteresis quantization.

Original:  We conducted ImageNet experiments using SGD with a momentum of 0.9 for 270 epochs with a batch size of 256 images and cosine annealing with an initial learning rate of 0.05.
Revised: [Figure] We conducted ImageNet experiments using SGD with a momentum of 0.9 for 270 epochs with a batch size of 256 images and cosine annealing with an initial learning rate of 0.05.

Original: We used the MobileNetV2 architecture from the ofﬁcial PyTorch implementation 2 .
Revised: We used the MobileNetV2 architecture from the ofﬁcial PyTorch implementation 3 .

Original: Fig.
Revised: Fig.

Original: 10 shows Top1 training & validation accuracy graphs.
Revised: 11 shows Top1 training & validation accuracy graphs.

Original: Observation of the training graph indicates that FP without hysteresis leads to very unstable ﬂuctuations throughout the training.
Revised: Observation of the training graph indicates that FP without hysteresis leads to very unstable ﬂuctuations throughout the training.

Original: On the other hand, in FP130 with hysteresis, training is less susceptible to ﬂuctuations and follows the baseline (FP32) training closely until the learning rate decreases toward the latter part of learning, where both FP with hysteresis and FP134 show some degradation from the baseline.
Revised: On the other hand, in FP130 with hysteresis, training is less susceptible to ﬂuctuations and follows the baseline (FP32) training closely until the learning rate decreases toward the latter part of learning, where both FP with hysteresis and FP134 show some degradation from the baseline.

Original: This is seen as a limitation due to the low precision of each format.
Revised: This is seen as a limitation due to the low precision of each format.

Original: A.4. 2- LAYER LSTM (PTB)
Revised: A.5. 2- LAYER LSTM (PTB)

Original: We adopted the 2-layer Long Short Term Memory (LSTM) network from PyTorch Examples3 for language modeling on the Penn Treebank dataset (Marcus et al., 1993).
Revised: We adopted the 2-layer Long Short Term Memory (LSTM) network from PyTorch Examples for language modeling on the Penn Treebank dataset (Marcus et al., 1993).

Original: We ran experiments in batches of 20 sentences with an initial learning rate of 20 which is decayed by a factor of 4 at epoch 11, 16, 26, 31 and 37.
Revised: We ran experiments in batches of 20 sentences with an initial learning rate of 20 which is decayed by a factor of 4 at epoch 11, 16, 26, 31 and 37.

Original: The embedding and hidden dimensions are 650 and the sequence length is 35.
Revised: The embedding and hidden dimensions are 650 and the sequence length is 35.

Original: Fig.
Revised: Fig.

Original: 11 shows training & validation perplexity.
Revised: 12 shows training & validation perplexity.

Original: A.4. T RANSFORMER M ODEL (IWLST)
Revised: A.5. T RANSFORMER M ODEL (IWLST)

Original: We adopted the Transformer Base model from the FairSeq 4 repository on the IWSLT’14 German to English translation task.
Revised: We adopted the Transformer Base model from the FairSeq 5 repository on the IWSLT’14 German to English translation task.

Original: We used Adam optimizer and default training parameters found in the repository and trained from scratch for 25 epochs.
Revised: We used Adam optimizer and default training parameters found in the repository and trained from scratch for 25 epochs.

Original: BLEU scores were calculated using the script from the repository. 
Revised: BLEU scores were calculated using the script from the repository. A.5. M OBILE N ET

Original: [Figure]
Revised: [Equation]

Original: We adopted a PyTorch implementation of SSDLite from the online repository 5 .
Revised: We adopted a PyTorch implementation of SSDLite from the online repository 6 .

Original: The base network is MobileNetV2 which was pretrained with each format in Appendix A.4.2.
Revised: The base network is MobileNetV2 which was pretrained with each format in Appendix A.5.3.

Original: In the case of INT4 without hysteresis, which failed training in MobileNetV2, a model pretrained with INT4 with hysteresis was used.

Original: The entire network is trained on VOC2012 and VOC2007 trainval datasets and evaluated on VOC2007 validation dataset.
Revised: The entire network is trained on VOC2012 and VOC2007 trainval datasets and evaluated on VOC2007 validation dataset.

Original: We used SGD with a momentum of 0.9 for 200 epochs in batches of 32 images and cosine annealing with an initial learning rate of 0.01.
Revised: We used SGD with a momentum of 0.9 for 200 epochs in batches of 32 images and cosine annealing with an initial learning rate of 0.01.

Original: Fig.
Revised: Fig.

Original: 12 shows validation loss at every 5 epochs.
Revised: 13 shows validation loss at every 5 epochs.

Original: Even in this experiment, in the case of FP130 without hysteresis the loss ﬂuctuates signiﬁcantly, whereas in FP130 with hysteresis learning proceeds much more stably.
Revised: Even in this experiment, in the case of FP130 without hysteresis the loss ﬂuctuates signiﬁcantly, whereas in FP130 with hysteresis learning proceeds much more stably.

Original: FP134 showed similar results to the baseline regardless of hysteresis quantization.
Revised: FP134 showed similar results to the baseline regardless of hysteresis quantization.

Original: A. M ODEL Q UANTIZATION M ETHODS
Revised: A. M ODEL Q UANTIZATION M ETHODS

Original: We quantized GEMM input and batchnorm input in all quantized training experiments.
Revised: We quantized GEMM input and batchnorm input in all quantized training experiments.

Original: Among the ﬁve models used in the experiment, the quantization details for three representative structures are+
Revised: Among the six models used in the experiment, the quantization details for three representative structures are

Original: [Figure]
Revised: [Figure]

Original: BN Quant
Revised: Quant

Original: ReLU
Revised: ReLU

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: Quant +
Revised: Quant +

Original: [Figure]

Original:  Quant
Revised: Conv Quant

Revised: BN Quant

Original: [Figure]
Revised: ReLU

Original:  Quant
Revised: Conv Quant

Original: ReLU [Figure]
Revised: BN Quant

Original:  Quant
Revised: Conv Quant

Original: [Figure] Quant[Figure]
Revised: BN Quant

Original: Quant
Revised: Quant

Original: [Figure]
Revised: [Figure]

Original: Residual Block

Original: [Figure]
Revised: shown in the Fig.

Original: ResNet
Revised: In each structure of ﬁgure, inputs such as x, c, h, V, K, and Q are also all quantized in 8 bits.

Original: [Figure] c
Revised: A conventional MAC unit consists of a multiplier and an accumulator. In the multiplier, the exponents of two input operands are summed while their mantissas are multiplied.

Original: [Figure] + [Figure] shown in the Fig. 13.
Revised: The multiplication part is more complex, and hence it dominates the area of the multiplier.

Original: In each structure of ﬁgure, inputs such as x, c, h, V, K, and Q are also all quantized in 8 bits.
Revised: As a result, the size of the multiplier is larger when more bits are allocated to mantissa. In the accumulator, a ﬂoating-point adder adds the multiplication results to a partial sum in FP169. The adder is decomposed into a shifter that aligns the mantissa by the exponent difference, an integer adder that sums aligned mantissas, and a quantization unit that converts the result back to FP169. Since the result is re-quantized into FP169, the addition operation of aligned mantissas does not need to be lossless.

Revised: FP169 format has a 10-bit mantissa including one hidden bit.

Revised: Therefore, we only need to accurately calculate higher 10 bits, which necessitates a 12-bit adder considering rounding.

Revised: Therefore, shifting by more than 12 bits is not needed even if the result of the multiplier has a larger exponent range.

Revised: Therefore, the shifter, adder, and quantization unit, which are the components of the accumulator, are not affected by the input format.

Revised: There are minor differences such as an adder that calculates the difference between exponents and a shifter with a different bit width of the input, but their costs are ignorable.

Revised: Contrarily, a multi-way MAC consists of a multiplier, a shifter for alignment, an adder tree, a normalization unit, and a ﬁnal accumulator.

Original: {+The multiplier and the ﬁnal accumulator are identical to sign exp man sign exp man x + XOR n bit m bit 1 bit subnormal << 2m+2 bit n+1 bit signed shifted number 2 n+
+}
Revised: {+The multiplier and the ﬁnal accumulator are identical to sign exp man sign exp man x + XOR n bit m bit 1 bit subnormal << 2m+2 bit n+1 bit signed shifted number 2 n+
+}

Revised: +2m-2 bit 2 n+1 +2m-1 bit α α-input adder tree +

Revised: FP

Revised: FP sign exp man sign exp man x + XOR n bit m bit 1 bit subnormal << 2m+2 bit n+1 bit signed shifted number 2 n+1 +2m-2 bit 2 n+1 +2m-1 bit α α-input adder tree +

Revised: FP

Revised: FP

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ) those of the conventional MAC.

Revised: However, since only one normalization unit and one ﬁnal accumulator are shared across multiple inputs, their implementation cost becomes insigniﬁcant for a larger number of inputs.

Revised: The shifter for alignment converts the multiplier output to an integer format since the cost of integer addition is lower than that of ﬂoating-point addition.

Revised: Then, the adder tree sums those integer values, and the normalization unit converts the result back to a ﬂoating-point format.

Revised: The cost of the shifter for alignment, adder tree, and normalization unit is all determined by the integer bit width, and the larger the exponent range of the input operands, the larger the required bit width, as shown in Fig.

Revised: In FP134, FP143, and FP152, the minimum integer bit widths are 23, 37, and 67 bits, respectively.

Revised: Since the bit width is sufﬁciently large, the cost difference of these units exceeds the cost difference of the multiplier.

Revised: Therefore, the cost of a multi-way MAC increases with the number of exponent bits.

Revised: In addition to the synthesis result for ASIC implementation in Table 3, we measured the hardware overhead of MAC units of different data formats on FPGA.

Revised: Table 7 shows the synthesis results on Xilinx Artix-7 FPGA (XC7A100TCSG324-1).

Revised: Those MAC units do not need block RAMs (BRAMs), and we used a compiler directive to avoid using DSP modules for fair comparisons.

Revised: Table 7 shows a similar trend to Table 3; the cost of one MAC gradually decreases as the number of inputs increases in the multi-way MAC.

Revised: Also, due to integer-based addition in the adder tree, the cost of FP134, which has the smallest dynamic range, exhibits lower costs than the other formats.

Revised: A. E FFECT OF Q UANTIZATION N OISE ON D ATA F LOW Q UANTIZATION

Revised: [Table]

Revised: SSDLite(VOC)

Revised: [Table]

Revised: Table 8 shows the training results when both activation and error are quantized in various data formats.

Revised: If an appropriate amount of noise is introduced in the network during training, it will increase the training loss but reduce the validation loss, suggesting that the model has been improved due to the regularization effect.

Revised: However, if the noise level continues to increase, the model’s performance will start to degrade at some point.

Revised: For instance, when MobileNetV2 is quantized in FP134, its performance is improved through the regularization effect since the training loss increases while the validation loss decreases compared to FP32.

Revised: However, both the training and validation losses increase when quantized in most cases, resulting in lower accuracy.

Revised: This suggests that using a very low precision data format already introduces a large amount of noise in the network, incurring performance degradation.

Revised: Hence, it is necessary to reduce error in the network to improve the training performance in low-precision training.