Original: T OWARD
Revised: T OWARD

Original: E FFICIENT
Revised: E FFICIENT

Original: L OW -P RECISION
Revised: L OW -P RECISION

Original: T RAINING :
Revised: T RAINING :

Original: D ATA
Revised: D ATA

Original: F ORMAT
Revised: F ORMAT

Original: O PTIMIZATION
Revised: O PTIMIZATION

Original: AND
Revised: AND

Original: H YSTERESIS
Revised: H YSTERESIS

Original: Q UANTIZATION
Revised: Q UANTIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As the complexity and size of deep neural networks continue to increase, lowprecision training has been extensively studied in the last few years to reduce hardware overhead.
Revised: As the complexity and size of deep neural networks continue to increase, lowprecision training has been extensively studied in the last few years to reduce hardware overhead.

Original: Training performance is largely affected by the numeric formats representing different values in low-precision training, but ﬁnding an optimal format typically requires numerous training runs, which is a very time-consuming process.
Revised: Training performance is largely affected by the numeric formats representing different values in low-precision training, but ﬁnding an optimal format typically requires numerous training runs, which is a very time-consuming process.

Original: In this paper, we propose a method to efﬁciently ﬁnd an optimal format for activations and errors without actual training.
Revised: In this paper, we propose a method to efﬁciently ﬁnd an optimal format for activations and errors without actual training.

Original: We employ this method to determine an 8-bit format suitable for training various models.
Revised: We employ this method to determine an 8-bit format suitable for training various models.

Original: In addition, we propose a hysteresis quanization scheme to suppress undesired ﬂuctuation in quantized weights.
Revised: In addition, we propose a hysteresis quanization scheme to suppress undesired ﬂuctuation in quantized weights.

Original: This scheme enables deeply quantized training using 4-bit weights, exhibiting only 0.2% degradation for ResNet-18 trained on ImageNet.
Revised: This scheme enables deeply quantized training using 4-bit weights, exhibiting only 0.2% degradation for ResNet-18 trained on ImageNet.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have been used in various ﬁelds such as vision, audio, natural language processing, and reinforcement learning.
Revised: Deep neural networks have been used in various ﬁelds such as vision, audio, natural language processing, and reinforcement learning.

Original: As larger and more complex neural networks are adopted, the energy and time consumed for training have become a critical issue in hardware implementation.
Revised: As larger and more complex neural networks are adopted, the energy and time consumed for training have become a critical issue in hardware implementation.

Original: Using low-bit representations in training signiﬁcantly reduces hardware overhead and memory footprint; hence, neural network training with limited precision has been extensively studied recently.
Revised: Using low-bit representations in training signiﬁcantly reduces hardware overhead and memory footprint; hence, neural network training with limited precision has been extensively studied recently.

Original: For instance, 16-bit formats are already adopted in commercial devices such as FP16 (IEEE, 2019) in NVidia GPU and bﬂoat16 (Kalamkar et al., 2019) in Google TPU (Wang et al., 2019).
Revised: For instance, 16-bit formats are already adopted in commercial devices such as FP16 (IEEE, 2019) in NVidia GPU and bﬂoat16 (Kalamkar et al., 2019) in Google TPU (Wang et al., 2019).

Original: Also,
Revised: Also,

Original: K ¨ oster et al.
Revised: K ¨ oster et al.

Original: (2017) suggested a new data format using a shared exponent suitable for low-precision training.
Revised: (2017) suggested a new data format using a shared exponent suitable for low-precision training.

Original: Recently, it has been demonstrated that even 8-bit formats could be adopted in deep neural network training with reasonable accuracy (Sun et al., 2019; Fox et al., 2020).
Revised: Recently, it has been demonstrated that even 8-bit formats could be adopted in deep neural network training with reasonable accuracy (Sun et al., 2019; Fox et al., 2020).

Original: However, there are various issues in realizing low-precision training in practical applications as detailed below.
Revised: However, there are various issues in realizing low-precision training in practical applications as detailed below.

Original: Optimal data format for low-precision training : Training performance is susceptible to the data format we use to represent variables in the network.
Revised: Optimal data format for low-precision training : Training performance is susceptible to the data format we use to represent variables in the network.

Original: When a value is represented using a ﬂoatingpoint format with a ﬁxed number of bits, there is a trade-off between dynamic range and precision.
Revised: When a value is represented using a ﬂoatingpoint format with a ﬁxed number of bits, there is a trade-off between dynamic range and precision.

Original: For instance, allocating more bits to the exponent part in a ﬂoating-point format enlarges the dynamic range but lowers precision due to fewer bits in the mantissa part.
Revised: For instance, allocating more bits to the exponent part in a ﬂoating-point format enlarges the dynamic range but lowers precision due to fewer bits in the mantissa part.

Original: Recent studies on 8-bit training suggest various ways to reduce the dynamic range required for number representation to enhance representation precision.
Revised: Recent studies on 8-bit training suggest various ways to reduce the dynamic range required for number representation to enhance representation precision.

Original: Early work on 8-bit training (Wang et al., 2018) adopts a 5-bit exponent to represent different variables using a single format, but Sun et al.
Revised: Early work on 8-bit training (Wang et al., 2018) adopts a 5-bit exponent to represent different variables using a single format, but Sun et al.

Original: (2019) examine the statistics of each variable and optimize the numeric formats separately.
Revised: (2019) examine the statistics of each variable and optimize the numeric formats separately.

Original: Speciﬁcally, the values used in the forward path (weight and activation) have a relatively narrow dynamic range, and only 4 bits are allocated to the exponent.
Revised: Speciﬁcally, the values used in the forward path (weight and activation) have a relatively narrow dynamic range, and only 4 bits are allocated to the exponent.

Original: Fox et al.
Revised: Fox et al.

Original: (2020) propose to divide data into smaller blocks and assign a shared exponent bias to each block.
Revised: (2020) propose to divide data into smaller blocks and assign a shared exponent bias to each block.

Original: Since the values in a block tend to exhibit similar statistics, the forward (weight and activation) and backward (error) paths could be represented using only 2-bit and 4-bit exponents, respectively.
Revised: Since the values in a block tend to exhibit similar statistics, the forward (weight and activation) and backward (error) paths could be represented using only 2-bit and 4-bit exponents, respectively.

Original: Note that the shared exponent bias is effectively identical to the scaling factor.
Revised: Note that the shared exponent bias is effectively identical to the scaling factor.

Original: If a variable has a value of m · 2 e and a shared exponent bias of b , then its actual value is m · 2 e + bias , which is identical to the scaling factor of 2 bias .
Revised: If a variable has a value of m · 2 e and a shared exponent bias of b , then its actual value is m · 2 e + bias , which is identical to the scaling factor of 2 bias .

Original: However, these approaches are difﬁcult to generalize since we should empirically decide numeric formats for each task, neural network structure, and quantization scheme (Fig. 1).
Revised: However, these approaches are difﬁcult to generalize since we should empirically decide numeric formats for each task, neural network structure, and quantization scheme (Fig. 1).

Original: Furthermore, analyzing the statistics of each variable is not enough to determine an optimal format.
Revised: Furthermore, analyzing the statistics of each variable is not enough to determine an optimal format.

Original: Their distributions often have a long tail, and
Revised: Their distributions often have a long tail, and

Original: [Figure]
Revised: [Figure]

Original: +
Revised: +

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: ReLU
Revised: ReLU

Original: Quant +
Revised: Quant +

Original: Conv
Revised: Conv

Original: [Figure]
Revised: [Figure]

Original: ReLU
Revised: ReLU

Original: [Figure]
Revised: [Figure]

Original: quantization
Revised: quantization

Original: [Figure]
Revised: [Figure]

Original: dataset +
Revised: dataset +

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: ReLU
Revised: ReLU

Original: Quant +
Revised: Quant +

Original: Conv
Revised: Conv

Original: [Figure]
Revised: [Figure]

Original: ReLU
Revised: ReLU

Original: Quant
Revised: Quant

Original: Different quantization position
Revised: Different quantization position

Original: [Figure]
Revised: [Figure]

Original: hence the dynamic range of the numeric format should be experimentally selected through many trial-and-errors in actual training.
Revised: hence the dynamic range of the numeric format should be experimentally selected through many trial-and-errors in actual training.

Original: Performance degradation in from-scratch training : Previous studies on quantized models show that a model could achieve comparable accuracy to full-precision models even using 1- or 2-bit weights (Choi et al., 2019;
Revised: Performance degradation in from-scratch training : Previous studies on quantized models show that a model could achieve comparable accuracy to full-precision models even using 1- or 2-bit weights (Choi et al., 2019;

Original: Martinez et al., 2020) through ﬁne-tuning a pre-trained model.
Revised: Martinez et al., 2020) through ﬁne-tuning a pre-trained model.

Original: However, in low-precision training where a neural network is trained from scratch using low-precision values and computations, the trained model typically shows a noticeable accuracy drop (Elhoushi et al., 2021).
Revised: However, in low-precision training where a neural network is trained from scratch using low-precision values and computations, the trained model typically shows a noticeable accuracy drop (Elhoushi et al., 2021).

Original: Fig. 1(b) shows the Top-1 validation accuracy of ResNet-18 (He et al., 2016) trained on
Revised: Fig. 1(b) shows the Top-1 validation accuracy of ResNet-18 (He et al., 2016) trained on

Original: ImageNet (Deng et al., 2009) for different training schemes.
Revised: ImageNet (Deng et al., 2009) for different training schemes.

Original: The weights are quantized into a 4bit base-2 logarithmic format.
Revised: The weights are quantized into a 4bit base-2 logarithmic format.

Original: From-scratch training of the model with quantized weights results in a 2.1% accuracy drop, whereas only 1.0% degradation is observed if we ﬁne-tune a pre-trained model.
Revised: From-scratch training of the model with quantized weights results in a 2.1% accuracy drop, whereas only 1.0% degradation is observed if we ﬁne-tune a pre-trained model.

Original: This suggests that even though a better solution exists for a given format, it cannot be reached through from-scratch training.
Revised: This suggests that even though a better solution exists for a given format, it cannot be reached through from-scratch training.

Original: To formalize the issues above, here we divide quantization in low-precision training into two types: network quantization and data ﬂow quantization.
Revised: To formalize the issues above, here we divide quantization in low-precision training into two types: network quantization and data ﬂow quantization.

Original: Network quantization refers to the quantization of the neural network model.
Revised: Network quantization refers to the quantization of the neural network model.

Original: An example of this type of quantization is weight quantization.
Revised: An example of this type of quantization is weight quantization.

Original: In network quantization, we need to reduce the performance difference between from-scratch training and ﬁne-tuning (Yang et al., 2019b).
Revised: In network quantization, we need to reduce the performance difference between from-scratch training and ﬁne-tuning (Yang et al., 2019b).

Original: On the other hand, data ﬂow quantization refers to the on-the-ﬂy quantization that occurs when data propagate through the network in low-precision training.
Revised: On the other hand, data ﬂow quantization refers to the on-the-ﬂy quantization that occurs when data propagate through the network in low-precision training.

Original: Examples include activation, error, and weight gradient quantizations.
Revised: Examples include activation, error, and weight gradient quantizations.

Original: Additional errors are introduced in weight update computation due to this type of quantization, which leads to performance degradation.
Revised: Additional errors are introduced in weight update computation due to this type of quantization, which leads to performance degradation.

Original: Hence, we need to ﬁnd an optimal format to minimize accuracy drop due to computation errors in data ﬂow quantization.
Revised: Hence, we need to ﬁnd an optimal format to minimize accuracy drop due to computation errors in data ﬂow quantization.

Original: In this paper, we present a systematic approach to implementing low-precision training on various models and tasks.
Revised: In this paper, we present a systematic approach to implementing low-precision training on various models and tasks.

Original: First, we present a method to efﬁciently ﬁnd an optimal format for data ﬂow quantization.
Revised: First, we present a method to efﬁciently ﬁnd an optimal format for data ﬂow quantization.

Original: In addition, we introduce a hysteresis quantization technique, a new quantization method for network quantization that can mitigate the issues of from-scratch training.
Revised: In addition, we introduce a hysteresis quantization technique, a new quantization method for network quantization that can mitigate the issues of from-scratch training.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We present a method that can predict the training performance of various numeric formats for data ﬂow quantization .
Revised: • We present a method that can predict the training performance of various numeric formats for data ﬂow quantization .

Original: This method allows us to determine an appropriate data format for different neural network structures, datasets, and tasks efﬁciently.
Revised: This method allows us to determine an appropriate data format for different neural network structures, datasets, and tasks efﬁciently.

Original: • Using the method above, we propose an optimal 8-bit format suitable for low-precision training of various models, which enables quantization of BatchNorm layer input and improves hardware efﬁciency with minimal performance degradation.
Revised: • Using the method above, we propose an optimal 8-bit format suitable for low-precision training of various models, which enables quantization of BatchNorm layer input and improves hardware efﬁciency with minimal performance degradation.

Original: • We propose a new quantization scheme that utilizes the hysteresis effect to improve the performance of from-scratch training.
Revised: • We propose a new quantization scheme that utilizes the hysteresis effect to improve the performance of from-scratch training.

Original: This scheme enables ultra-low-precision training using 4-bit logarithmic weights.
Revised: This scheme enables ultra-low-precision training using 4-bit logarithmic weights.

Original:  D ATA F LOW Q UANTIZATION 2. N UMERIC F ORMATS
Revised:  D ATA F LOW Q UANTIZATION 2. N UMERIC F ORMATS

Original: There are many numeric formats that can be constructed with n bits depending on how much dynamic range is required and how many valid bits are used for representing a value.
Revised: There are many numeric formats that can be constructed with n bits depending on how much dynamic range is required and how many valid bits are used for representing a value.

Original: For example, using 8 bits we could implement 8-bit ﬁxed point integer format, 8-bit ﬂoating-point formats such as FP152, FP143, and FP125 (FP1 xy represents 1 sign bit, x exponent bits, y mantissa bits), 8-bit posit format (Gustafson & Yonemoto, 2017), and 8-bit ﬂoat-ﬁx format (Han et al., 2019).
Revised: For example, using 8 bits we could implement 8-bit ﬁxed point integer format, 8-bit ﬂoating-point formats such as FP152, FP143, and FP125 (FP1 xy represents 1 sign bit, x exponent bits, y mantissa bits), 8-bit posit format (Gustafson & Yonemoto, 2017), and 8-bit ﬂoat-ﬁx format (Han et al., 2019).

Original: Since the diversity of formats that could be formulated using n bits is nearly unlimited, here we assume some constraints to limit candidates while still including widely used formats such as ﬁxed-point and ﬂoating-point formats as below:
Revised: Since the diversity of formats that could be formulated using n bits is nearly unlimited, here we assume some constraints to limit candidates while still including widely used formats such as ﬁxed-point and ﬂoating-point formats as below:

Original: • The MSB (Most Signiﬁcant Bit) is used as a sign bit and other bits represent magnitude.
Revised: • The MSB (Most Signiﬁcant Bit) is used as a sign bit and other bits represent magnitude.

Original: Accordingly, only symmetric formats that have identical representable ranges for positive and
Revised: Accordingly, only symmetric formats that have identical representable ranges for positive and

Original: negative
Revised: negative

Original: numbers are considered.
Revised: numbers are considered.

Original: Two’s
Revised: Two’s

Original: complement
Revised: complement

Original: representation is slightly asym-
Revised: representation is slightly asym-

Original: metric since it can represent one more negative value, but it does not incur a signiﬁcant difference.
Revised: metric since it can represent one more negative value, but it does not incur a signiﬁcant difference.

Original: • The number of valid bits of a larger value is greater than or equal to the number of valid bits of a smaller value.
Revised: • The number of valid bits of a larger value is greater than or equal to the number of valid bits of a smaller value.

Original: The valid bits stand for signiﬁcant digits in binary representation.
Revised: The valid bits stand for signiﬁcant digits in binary representation.

Original: • The ratio between consecutive representable values does not exceed 2.
Revised: • The ratio between consecutive representable values does not exceed 2.

Original: For example, the base-4 logarithmic format is excluded.
Revised: For example, the base-4 logarithmic format is excluded.

Original: We could obtain 166 8-bit formats that meet these constraints.
Revised: We could obtain 166 8-bit formats that meet these constraints.

Original: Then, we reduce 1 and 2 valid bits in each format to obtain 7- and 6-bit formats, resulting in 498 formats in total.
Revised: Then, we reduce 1 and 2 valid bits in each format to obtain 7- and 6-bit formats, resulting in 498 formats in total.

Original: More information on the numeric formats considered in our experiments is provided in Appendix A.1.
Revised: More information on the numeric formats considered in our experiments is provided in Appendix A.1.

Original: 2. A CTIVATION AND E RROR Q UANTIZATION
Revised: 2. A CTIVATION AND E RROR Q UANTIZATION

Original: In a neural network consisting of n layers, the training process is described by
Revised: In a neural network consisting of n layers, the training process is described by

Original: [Equation]
Revised: [Equation]

Original: where A , E , W , and G w are activation, error, weight, and weight gradient, respectively.
Revised: where A , E , W , and G w are activation, error, weight, and weight gradient, respectively.

Original: f , g , h , and o are forward, backward, gradient, and update functions.
Revised: f , g , h , and o are forward, backward, gradient, and update functions.

Original: l and t represent the layer number and time step.
Revised: l and t represent the layer number and time step.

Original: We follow the quantized training scheme suggested by Fox et al.
Revised: We follow the quantized training scheme suggested by Fox et al.

Original: (2020), but
Revised: (2020), but

Original: with the following modiﬁcations to reduce hardware implementation costs.
Revised: with the following modiﬁcations to reduce hardware implementation costs.

Original: A and E are quantized not only for the GEMM input but also for the BatchNorm layer input.
Revised: A and E are quantized not only for the GEMM input but also for the BatchNorm layer input.

Original: BatchNorm layer normalizes input using the mean and variance of each channel, but these values are obtained only after observing all the inputs from the previous layer, necessitating that all input values are temporarily stored in memory.
Revised: BatchNorm layer normalizes input using the mean and variance of each channel, but these values are obtained only after observing all the inputs from the previous layer, necessitating that all input values are temporarily stored in memory.

Original: Therefore, quantizing the BatchNorm layer’s input signiﬁcantly reduces memory footprint and memory access overhead.
Revised: Therefore, quantizing the BatchNorm layer’s input signiﬁcantly reduces memory footprint and memory access overhead.

Original: Additionally, the scope of sharing exponent bias is extended to a layer ( A l and E l ) to avoid the overhead of aligning partial sums from different blocks in block-wise exponent sharing.
Revised: Additionally, the scope of sharing exponent bias is extended to a layer ( A l and E l ) to avoid the overhead of aligning partial sums from different blocks in block-wise exponent sharing.

Original: Finally, instead of determining the shared exponent bias by analyzing all values in the layer, we conservatively update it by detecting overﬂow and underutilization that occurred in the previous mini-batch.
Revised: Finally, instead of determining the shared exponent bias by analyzing all values in the layer, we conservatively update it by detecting overﬂow and underutilization that occurred in the previous mini-batch.

Original: [Figure]
Revised: [Figure]

Original: 2. I NDICATORS OF T RAINING P ERFORMANCE
Revised: 2. I NDICATORS OF T RAINING P ERFORMANCE

Original: Effect of quantized error :
Revised: Effect of quantized error :

Original: Quantizing the error E in the backward path is independent of how the forward path behaves since the loss surface of the model does not change.
Revised: Quantizing the error E in the backward path is independent of how the forward path behaves since the loss surface of the model does not change.

Original: Therefore, the optimal W that the network needs to reach through training remains the same regardless of the error quantization scheme.
Revised: Therefore, the optimal W that the network needs to reach through training remains the same regardless of the error quantization scheme.

Original: However, when the error is quantized, a quantization error ∆ E is introduced in E , which incurs a noise N ∆ E in G w through the gradient function in Eq.
Revised: However, when the error is quantized, a quantization error ∆ E is introduced in E , which incurs a noise N ∆ E in G w through the gradient function in Eq.

Original: 3 and potentially updates each weight in the wrong direction.
Revised: 3 and potentially updates each weight in the wrong direction.

Original: While some amount of noise may improve the training performance through regularization, using low-precision formats already introduces a large noise in the network, incurring performance degradation (see Appendix A.8).
Revised: While some amount of noise may improve the training performance through regularization, using low-precision formats already introduces a large noise in the network, incurring performance degradation (see Appendix A.8).

Original: Therefore, we suggest that the weight gradient error
Revised: Therefore, we suggest that the weight gradient error

Original: N ∆ E could be a good indicator of degradation in training performance.
Revised: N ∆ E could be a good indicator of degradation in training performance.

Original: One way to implement this is predicting performance using the magnitude of N ∆ E ; however, if the noise is in the same direction as G w , it would only change the amount of each update and result in a less severe effect.
Revised: One way to implement this is predicting performance using the magnitude of N ∆ E ; however, if the noise is in the same direction as G w , it would only change the amount of each update and result in a less severe effect.

Original: Instead, we could measure the misalignment between G w + N ∆ E and G w for performance prediction.
Revised: Instead, we could measure the misalignment between G w + N ∆ E and G w for performance prediction.

Original: The misalignment between two vectors is estimated by
Revised: The misalignment between two vectors is estimated by

Original: [Equation]
Revised: [Equation]

Original: Then, the change in the update direction due to N ∆ E is ∠ ( G w , G w + N ∆ E ) .
Revised: Then, the change in the update direction due to N ∆ E is ∠ ( G w , G w + N ∆ E ) .

Original: We can expect that the smaller ∠ ( G w , G w + N ∆ E ) , the better the training performance.
Revised: We can expect that the smaller ∠ ( G w , G w + N ∆ E ) , the better the training performance.

Original: Effect of quantized activation : Contrary to error quantization, activation quantization affects the way the forward path operates, and the loss surface of the model changes.
Revised: Effect of quantized activation : Contrary to error quantization, activation quantization affects the way the forward path operates, and the loss surface of the model changes.

Original: Hence, the global optima of weight parameters shift, where the amount of shift would be proportional to the quantization noise.
Revised: Hence, the global optima of weight parameters shift, where the amount of shift would be proportional to the quantization noise.

Original: The displacement of global optima can be indirectly estimated using the direction of the weight gradients G w .
Revised: The displacement of global optima can be indirectly estimated using the direction of the weight gradients G w .

Original: If the angle ∠ ( G w , G w + N ∆ A ) is small, the deviation of the global optima is expected to be small as well, suggesting a better training performance.
Revised: If the angle ∠ ( G w , G w + N ∆ A ) is small, the deviation of the global optima is expected to be small as well, suggesting a better training performance.

Original: In the discussions above, we assumed that the angles ∠ ( G w , G w + N ∆ E ) and ∠ ( G w , G w + N ∆ A ) could be used to predict training performance.
Revised: In the discussions above, we assumed that the angles ∠ ( G w , G w + N ∆ E ) and ∠ ( G w , G w + N ∆ A ) could be used to predict training performance.

Original: We experimentally prove this by comparing the training performance of different numeric formats.
Revised: We experimentally prove this by comparing the training performance of different numeric formats.

Original: For 498 numeric formats in 6 to 8 bits, we compare the loss obtained from training with the proposed performance indicators (Fig. 2).
Revised: For 498 numeric formats in 6 to 8 bits, we compare the loss obtained from training with the proposed performance indicators (Fig. 2).

Original: Training loss is obtained by training ResNet-18 on CIFAR-10 dataset using SGD with a momentum of 0.
Revised: Training loss is obtained by training ResNet-18 on CIFAR-10 dataset using SGD with a momentum of 0.

Original: for 60 epochs.
Revised: for 60 epochs.

Original: The batch size is 128 images and the initial learning rate is 0.1, which is decayed by a cosine scheduler.
Revised: The batch size is 128 images and the initial learning rate is 0.1, which is decayed by a cosine scheduler.

Original: We average angles from 100 mini-batches after quantizing a pre-trained model.
Revised: We average angles from 100 mini-batches after quantizing a pre-trained model.

Original: Note that we use G w of the ﬁrst layer since it can reﬂect quantization errors that occur in the activations and errors of all the layers in the network.
Revised: Note that we use G w of the ﬁrst layer since it can reﬂect quantization errors that occur in the activations and errors of all the layers in the network.

Original: The weight gradients from the full-precision network, the network with quantized activations, and the network with quantized errors are G w ,
Revised: The weight gradients from the full-precision network, the network with quantized activations, and the network with quantized errors are G w ,

Original: G w + N ∆ A , and G W + N ∆ E , respectively.
Revised: G w + N ∆ A , and G W + N ∆ E , respectively.

Original: Fig.
Revised: Fig.

Original: 2 shows that using the misalignment angle results in not only a higher Spearman’s correlation but also a more distinct shape for low training losses, making it a better metric for data format selection.
Revised: 2 shows that using the misalignment angle results in not only a higher Spearman’s correlation but also a more distinct shape for low training losses, making it a better metric for data format selection.

Original: For instance, using the error magnitude would predict the best format for transformer incorrectly (see Fig. 7(e) in Appendix A.3).
Revised: For instance, using the error magnitude would predict the best format for transformer incorrectly (see Fig. 7(e) in Appendix A.3).

Original: While obtaining the misalignment angle requires additional computations, its overhead is negligible since the part that requires the most time and computation is to obtain G w , G w + ∆ E , and G w + N ∆ A , which is still signiﬁcantly lower than actual training.
Revised: While obtaining the misalignment angle requires additional computations, its overhead is negligible since the part that requires the most time and computation is to obtain G w , G w + ∆ E , and G w + N ∆ A , which is still signiﬁcantly lower than actual training.

Original: Using this method, we could determine the optimal format for a speciﬁc neural network model, dataset, and task very efﬁciently as we only need to
Revised: Using this method, we could determine the optimal format for a speciﬁc neural network model, dataset, and task very efﬁciently as we only need to

Original: ∠ ( G W ,
Revised: ∠ ( G W ,

Original: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 1.
Revised: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 1.

Original: Fail (a) ResNet-18 (ImageNet) ∠ ( G W ,
Revised: Fail (a) ResNet-18 (ImageNet) ∠ ( G W ,

Original: G W + N ∆ A ) ∠ (G W , G W +N ∆E )
Revised: G W + N ∆ A ) ∠ (G W , G W +N ∆E )

Original: Fail (b) ResNet-101 (ImageNet) ∠ ( G W ,
Revised: Fail (b) ResNet-101 (ImageNet) ∠ ( G W ,

Original: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 1.
Revised: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 1.

Original: Fail (c) MobileNetV2 (ImageNet) ∠ ( G W ,
Revised: Fail (c) MobileNetV2 (ImageNet) ∠ ( G W ,

Original: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 
Revised: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 4.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: layer LSTM (PTB) ∠ ( 
Revised: layer LSTM (PTB) ∠ ( G W ,

Original: [Figure]

Original:  W,
Revised: G W

Original: [Figure]
Revised: [Figure]

Original: N ∆ A ) ∠ (G W , G W +N [Figure]
Revised: N ∆ A ) ∠ (G W , G W +N ∆E ) 3.621 3.

Original: Fail (e) Transformer (IWLST) ∠ ( G W ,
Revised: Fail (e) Transformer (IWLST) ∠ ( G W ,

Original: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 2.
Revised: G W + N ∆ A ) ∠ (G W , G W +N ∆E ) 2.

Original: Fail (f) MobileNetV [Figure] SSDLite (VOC) measure the misalignment angle without time-consuming network training.
Revised: Fail (f) MobileNetV2 + SSDLite (VOC) measure the misalignment angle without time-consuming network training.

Original: For experiments in Fig.
Revised: For experiments in Fig.

Original: 2, the amount of computation is reduced by 99.6%, and the reduction will be even larger for larger datasets and complex networks that need more epochs for training.
Revised: 2, the amount of computation is reduced by 99.6%, and the reduction will be even larger for larger datasets and complex networks that need more epochs for training.

Original: 2. O PTIMAL F ORMAT FOR D ATA F LOW Q UANTIZATION
Revised: 2. O PTIMAL F ORMAT FOR D ATA F LOW Q UANTIZATION

Original: Here we show that we could ﬁnd an optimal format for training speciﬁc models using the proposed performance estimation method above.
Revised: Here we show that we could ﬁnd an optimal format for training with quantized errors and activations using the proposed performance estimation method above.

Original: We select six models  widely used in quantized training research for experiments: ResNet-18, ResNet-101, MobileNetV2 (Sandler et al., 2018), 2-layer 
Revised: To ﬁnd a format suitable for a wide range of models, we select six models with different architectures, layer types, and target tasks that are widely used in quantized training research for experiments: ResNet-18, ResNet-101, MobileNetV (Sandler et al., 2018), 2-layer LSTM, small transformer for translation on the IWSLT German to

Original: LSTM, small transformer for translation on the IWSLT German to English dataset (Cettolo et al., 2014), and SSD-Lite (Liu et al., 2016) with MobileNetV2.
Revised: English dataset (Cettolo et al., 2014), and SSD-Lite (Liu et al., 2016) with MobileNetV2.

Original: We ﬁrst measure misalignment angles for 166 8-bit formats.
Revised: We ﬁrst measure misalignment angles for 166 8-bit formats.

Original: To verify the correlation between the training performance and the misalignment angles, we select four formats that exhibit low hardware implementation costs (INT8, FP152, FP143, and FP134) and train the networks using each format.
Revised: To verify the correlation between the training performance and the misalignment angles, we select four formats that exhibit low hardware implementation costs (INT8, FP152, FP143, and FP134) and train the networks using each format.

Original: While we may use different formats for activation and error, it requires a complicated datapath (Sun et al., 2019) and hence we only consider a single format for both variables.
Revised: While we may use different formats for activation and error, it requires a complicated datapath (Sun et al., 2019) and hence we only consider a single format for both variables.

Original: The experimental results in Fig.
Revised: The experimental results in Fig.

Original: 3 demonstrate that the training performance is higher if both misalignment angles are small in all tasks and models, conﬁrming that the proposed indicators could be used to determine the optimal numeric format.
Revised: 3 demonstrate that the training performance is higher if both misalignment angles are small in all tasks and models, conﬁrming that the proposed indicators could be used to determine the optimal numeric format.

Original: Fig.
Revised: Fig.

Original: 3 suggests that FP134 and FP143 are the best candidates across all models.
Revised: 3 suggests that FP134 and FP143 are the best candidates across all models.

Original: For hardware implementation, FP is the most optimal format due to its low implementation cost, which is discussed in Section 4. in detail.
Revised: For hardware implementation, FP134 is the most optimal format due to its low implementation cost, which is discussed in Section 4.2 in detail.

Original: Note that using the error magnitude leads to the same conclusion that FP134 is the best format for the target models.
Revised: Note that using the error magnitude leads to the same conclusion that FP134 is the best format for the target models.

Original: See Appendix A.3 for more details.
Revised: See Appendix A.3 for more details.

Original:  N ETWORK Q UANTIZATION
Revised:  N ETWORK Q UANTIZATION

Original: In quantized neural networks, the weight parameters are generally quantized in a way that minimizes the quantization error (Choi et al., 2019; Martinez et al., 2020).
Revised: In quantized neural networks, the weight parameters are generally quantized in a way that minimizes the quantization error (Choi et al., 2019; Martinez et al., 2020).

Original: For instance, if x is quantized into a ﬁxed-point format through s (cid:98) xs (cid:101) , a proper value is selected for the scaling factor s to minimize the quantization error.
Revised: For instance, if x is quantized into a ﬁxed-point format through s (cid:98) xs (cid:101) , a proper value is selected for the scaling factor s to minimize the quantization error.

Original: However, as the weights continue to change during training, we need to calculate s for every update, which could cause signiﬁcant overhead.
Revised: However, as the weights continue to change during training, we need to calculate s for every update, which could cause signiﬁcant overhead.

Original: Therefore, prior studies on low-precision training suggest constraining the scaling factor to the power of 2 in the shared exponent (K¨oster et al.,2017) or the shared exponent bias (Fox et al., 2020).
Revised: Therefore, prior studies on low-precision training suggest constraining the scaling factor to the power of 2 in the shared exponent (K¨oster et al.,

Original: In this section, we analyze the issues behind weight quantization and propose a new quantization scheme to mitigate those issues.

Original: w
Revised: w

Original: Qw       w Qw   w t
Revised: Qw       w Qw   w t

Original: Q w Small ∆ w Large ∆ w w t
Revised: Q w Small ∆ w Large ∆ w w t

Original: Q w Small ∆ w Large ∆ w (a) Conventional quantization      w
Revised: Q w Small ∆ w Large ∆ w (a) Conventional quantization      w

Original: Qw decreasing path increasing path      w
Revised: Qw decreasing path increasing path      w

Original: Qw decreasing path increasing path
Revised: Qw decreasing path increasing path

Original: Q w t
Revised: Q w t

Original: Small ∆ w Large ∆ w w
Revised: Small ∆ w Large ∆ w w

Original: Q w t
Revised: Q w t

Original: Small ∆ w Large ∆ w w
Revised: Small ∆ w Large ∆ w w

Original: [Figure]
Revised: [Figure]

Original: Hysteresis
Revised: 2017) or the shared exponent bias (Fox et al., 2020).

Original: [Figure]
Revised: In this section, we analyze the issues behind weight quantization and propose a new quantization scheme to mitigate those issues.

Original: 3. F LUCTUATION OF W EIGHT P ARAMETERS
Revised: 3. F LUCTUATION OF W EIGHT P ARAMETERS

Original: In typical low-precision training, a master copy of weight parameters is separately maintained in high precision, and those weights are updated based on the computed weight gradient.
Revised: In typical low-precision training, a master copy of weight parameters is separately maintained in high precision, and those weights are updated based on the computed weight gradient.

Original: This highprecision weight is quantized into a low-precision format and used for the forward path computation during training.
Revised: This highprecision weight is quantized into a low-precision format and used for the forward path computation during training.

Original: Since the scaling factor s is constrained to 2 n , the quantization threshold remains the same unless s is updated due to overﬂow or underutilization.
Revised: Since the scaling factor s is constrained to 2 n , the quantization threshold remains the same unless s is updated due to overﬂow or underutilization.

Original: If the optimal weight is located between two representable values of a data format, the quantized weight would ﬂuctuate alternately between the two values in each update (Fig. 4(a)) even for a very small weight update, causing large ﬂuctuations and undermining training performance.
Revised: If the optimal weight is located between two representable values of a data format, the quantized weight would ﬂuctuate alternately between the two values in each update (Fig. 4(a)) even for a very small weight update, causing large ﬂuctuations and undermining training performance.

Original: 3. H YSTERESIS Q UANTIZATION
Revised: 3. H YSTERESIS Q UANTIZATION

Original: To mitigate the ﬂuctuation issue above, we propose to introduce the concept of hysteresis to quantization.
Revised: To mitigate the ﬂuctuation issue above, we propose to introduce the concept of hysteresis to quantization.

Original: More speciﬁcally, we quantize each weight differently depending on the direction of the weight update.
Revised: More speciﬁcally, we quantize each weight differently depending on the direction of the weight update.

Original: Fig. 4(b) describes the proposed hysteresis quantization scheme.
Revised: Fig. 4(b) describes the proposed hysteresis quantization scheme.

Original: The quantization threshold differs when the master weight increases and decreases.
Revised: The quantization threshold differs when the master weight increases and decreases.

Original: For instance, we could round down the weight for a positive weight update, and round it up for a negative weight update.
Revised: For instance, we could round down the weight for a positive weight update, and round it up for a negative weight update.

Original: The equation below describes the proposed quantization scheme.
Revised: The equation below describes the proposed quantization scheme.

Original: [Equation]
Revised: [Equation]

Original: ) where w is the original value, Q w is its quantized value, and t is the time step.
Revised: where w is the original value, Q w is its quantized value, and t is the time step.

Original: The proposed hysteresis quantization reduces ﬂuctuation signiﬁcantly, stabilizing the training process and allowing the network to reach global optima more efﬁciently.
Revised: The proposed hysteresis quantization reduces ﬂuctuation signiﬁcantly, stabilizing the training process and allowing the network to reach global optima more efﬁciently.

Original: In Fig. 4(b), if the weight change ∆ W is small, then enough number of those changes should be accumulated to ﬂip Q w .
Revised: In Fig. 4(b), if the weight change ∆ W is small, then enough number of those changes should be accumulated to ﬂip Q w .

Original: Hence, the update frequency is now proportional to the weight gradient.
Revised: Hence, the update frequency is now proportional to the weight gradient.

Original: This helps the network to learn better while suppressing ﬂuctuations for small G w values.
Revised: This helps the network to learn better while suppressing ﬂuctuations for small G w values.

Original: Alternatively, we may mitigate weight quantization errors by adopting AdaRound (Nagel et al.
Revised: Alternatively, we may mitigate weight quantization errors by adopting AdaRound (Nagel et al.

Original: (2020)), which learns whether each weight should be rounded up or down to produce the same output as high-precision weights.
Revised: (2020)), which learns whether each weight should be rounded up or down to produce the same output as high-precision weights.

Original: However, whenever full-precision weights are updated, we need to re-train the learnable parameters (i.e., quantization scheme of each weight), incurring a large overhead and undermining the beneﬁt of low-precision training.
Revised: However, whenever full-precision weights are updated, we need to re-train the learnable parameters (i.e., quantization scheme of each weight), incurring a large overhead and undermining the beneﬁt of low-precision training.

Original: 3. U LTRA -L OW -P RECISION F ORMAT FOR N ETWORK Q UANTIZATION
Revised: 3. U LTRA -L OW -P RECISION F ORMAT FOR N ETWORK Q UANTIZATION

Original: To verify the effectiveness of the proposed hysteresis quantization, we select 4-bit logarithmic representation as an ultra-low-precision format for weight parameters.
Revised: To verify the effectiveness of the proposed hysteresis quantization, we select 4-bit logarithmic representation as an ultra-low-precision format for weight parameters.

Original: This format has the same dynamic range as INT8 which is widely used for weight quantization, and is more hardware-efﬁcient as multiplication is implemented only using simple shift operations.
Revised: This format has the same dynamic range as INT8 which is widely used for weight quantization, and is more hardware-efﬁcient as multiplication is implemented only using simple shift operations.

Original: There have been attempts to use logarithmic weights in quantized neural networks (Lee et al., 2017; Elhoushi et al., 2021), but from-scratch training shows a signiﬁcant performance degradation.
Revised: There have been attempts to use logarithmic weights in quantized neural networks (Lee et al., 2017; Elhoushi et al., 2021), but from-scratch training shows a signiﬁcant performance degradation.

Original: In logarithmic data formats, the interval of quantization points is not uniform, making them more sensitive to ﬂuctuation. 
Revised: In logarithmic data formats, the interval of quantization points is not uniform, making them more sensitive to ﬂuctuation. [Figure]

Original: Fig.
Revised: Fig.

Original: 5 shows experimental results of ResNet-18 training on ImageNet using 4-bit logarithmic weights.
Revised: 5 shows experimental results of ResNet-18 training on ImageNet using 4-bit logarithmic weights.

Original: Note that we apply channel-wise quantization to the convolutional layers to compensate for[Figure] the insufﬁcient expression range and layer-wise quantization to other types of layers.
Revised: Note that we apply channel-wise quantization to the convolutional layers to compensate for the insufﬁcient expression range and layer-wise quantization to other types of layers.

Original: Further details on the experimental setup are provided in Appendix A.5.1.
Revised: Further details on the experimental setup are provided in Appendix A.5.1.

Original: First, we measure how many quantized weights Q w change when the network performs one weight update using a mini-batch and average them over the ﬁrst 100 updates in the 60th epoch.
Revised: First, we measure how many quantized weights Q w change when the network performs one weight update using a mini-batch and average them over the ﬁrst 100 updates in the 60th epoch.

Original: The experimental result displayed in Fig.
Revised: The experimental result displayed in Fig.

Original: 5(a) clearly shows that using hysteresis signiﬁcantly reduces weight change frequency and stabilizes the training process.
Revised: 5(a) clearly shows that using hysteresis signiﬁcantly reduces weight change frequency and stabilizes the training process.

Original: Fig.
Revised: Fig.

Original: 5(b) compares the training performance of quantization with and without hysteresis.
Revised: 5(b) compares the training performance of quantization with and without hysteresis.

Original: Hysteresis quantization not only speeds up training but also achieves better results at the end of training.
Revised: Hysteresis quantization not only speeds up training but also achieves better results at the end of training.

Original: Note that hysteresis quantization is applicable to other data formats, and additional experimental results can be found in Appendix A.4.
Revised: Note that hysteresis quantization is applicable to other data formats, and additional experimental results can be found in Appendix A.4.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section

Original: [Table]
Revised: [Table]

Original: we experimentally show that the proposed training performance prediction method and hysteresis quantization scheme enable efﬁcient low-precision training.
Revised: MobileNetV2 + SSDLite (VOC) [mAP]

Original: [Table]
Revised: [Table]

Original:   RECISION T RAINING
Revised:  8- BIT L OW -P RECISION T RAINING

Original: In Section 2.4, we found that FP134 is the optimal format for the target models using the proposed performance prediction method.
Revised: In Section 2.4, we found that FP134 is the optimal format for low-precision training using the proposed performance prediction method.

Original: We measure the training performance of this format and compare it against other 8-bit data formats from recent studies by applying those formats to the training of various neural network models.
Revised: We measure the training performance of this format and compare it against other 8-bit data formats from recent studies by applying those formats to the training of various neural network models.

Original: More details on the experimental setup are provided inAppendix
Revised: More details on the experimental setup are provided in

Original:  A.5.
Revised: Appendix A.5.

Original: The performance of the proposed data format is summarized in Table 1.
Revised: The performance of the proposed data format is summarized in Table 1.

Original: Overall, 8-bit training using FP134 achieves nearly the same performance as the full-precision training on all models.
Revised: Overall, 8-bit training using FP134 achieves nearly the same performance as the full-precision training on all models.

Original: Even in MobileNetV2, which is known to be sensitive to quantization due to the small number of parameters, only 0.3% degradation occurred.
Revised: Even in MobileNetV2, which is known to be sensitive to quantization due to the small number of parameters, only 0.3% degradation occurred.

Original: Sun et al.
Revised: Sun et al.

Original: (2019) show that HFP8 also exhibits only 0.2% accuracy degradation in MobileNetV2 (71.81% vs. 71.61%), but they quantize BatchNorm input into 16 bits instead of 8 bits, roughly doubling the memory access and computational complexity.
Revised: (2019) show that HFP8 also exhibits only 0.2% accuracy degradation in MobileNetV2 (71.81% vs. 71.61%), but they quantize BatchNorm input into 16 bits instead of 8 bits, roughly doubling the memory access and computational complexity.

Original: Additionally, since the forward and backward paths employ different data formats,HFP8 is
Revised: Additionally, since the forward and backward paths employ different data formats,

Revised: HFP8 is actually implemented using 9-bit MAC units in hardware (Agrawal et al., 2021).

Revised: Table

Original: [Table]
Revised: [Table]

Original: Fixed Point
Revised: Fixed Point

Original: [Table] [Table]
Revised: [Table] [Table]

Original: input
Revised: input

Original: [Table]
Revised: [Table]

Original: input
Revised: input

Original: [Table]
Revised: [Table]

Original: input
Revised: input

Original: [Table]
Revised: [Table]

Original: input
Revised: input

Original: [Table]
Revised: [Table]

Original: input          
Revised: input          1 Sun

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: oster
Revised: al.

Original: [Table]
Revised: [Table]

Original: 2017) actually implemented using 9-bit MAC units in hardware (Agrawal et al., 2021).
Revised: K ¨ oster et al.

Original: Table 2 compares the training performance of various data formats for ResNet-18 training.
Revised: (2017) compares the training performance of various data formats for ResNet-18 training.

Original: The columns w, x, dw, dx, and acc refer to weight, activation, weight gradient, error, and GEMM accumulation, respectively.
Revised: The columns w, x, dw, dx, and acc refer to weight, activation, weight gradient, error, and GEMM accumulation, respectively.

Original: Our FP134 format exhibits no accuracy drop compared to full-precision training.
Revised: Our FP134 format exhibits no accuracy drop compared to full-precision training.

Original: HFP8 (Sun et al., 2019) and BM8 (Fox et al., 2020) demonstrate similar performance, but they both use higher precision to represent BatchNorm inputs, and different formats are adopted in the forward and backward paths, necessitating complex computation units when implemented in hardware, as decribed above.
Revised: HFP (Sun et al., 2019) and BM8 (Fox et al., 2020) demonstrate similar performance, but they both use higher precision to represent BatchNorm inputs, and different formats are adopted in the forward and backward paths, necessitating complex computation units when implemented in hardware, as decribed above.

Original: In addition, BM8 assumes block-wise sharing of exponent bias, incurring additional overhead in memory access and data alignment.
Revised: In addition, BM8 assumes block-wise sharing of exponent bias, incurring additional overhead in memory access and data alignment.

Original: FP8-SEB
Revised: FP8-SEB

Original: (Park et al., 2021) addresses this issue by employing layer-wise exponent bias sharing and multi-way MAC units, but it results in a 0.7% accuracy drop for ResNet-18 training.
Revised: (Park et al., 2021) addresses this issue by employing layer-wise exponent bias sharing and multi-way MAC units, but it results in a 0.7% accuracy drop for ResNet-18 training.

Original: Contrarily, our data format shows no performance degradation, while deeply quantizing BatchNorm inputs into the same format and allowing for a simple datapath by using an identical data format in the forward and backward paths.
Revised: Contrarily, our data format shows no performance degradation, while deeply quantizing BatchNorm inputs into the same format and allowing for a simple datapath by using an identical data format in the forward and backward paths.

Original: 4. H ARDWARE E VALUATION
Revised: 4. H ARDWARE E VALUATION

Original: For hardware implementation cost comparisons, we implemented a conventional MAC unit and a multi-way MAC unit with integer-based accumulation (Tambe et al., 2020;
Revised: For hardware implementation cost comparisons, we implemented a conventional MAC unit and a multi-way MAC unit with integer-based accumulation (Tambe et al., 2020;

Original: Park et al., 2021) that support data formats presented in Section 4.1.
Revised: Park et al., 2021) that support data formats presented in Section 4.1.

Original: For accumulation, we use FP169 with chunk-based accumulation (Wang et al., 2018).
Revised: For accumulation, we use FP169 with chunk-based accumulation (Wang et al., 2018).

Original: Experimental results in Table 3 show that FP134 exhibits lower cost than FP143 and other formats in previous studies.
Revised: Experimental results in Table 3 show that FP134 exhibits lower cost than FP143 and other formats in previous studies.

Original: Note that HFP8 (Sun et al., 2019) and BM (Fox et al., 2020) employ different formats for activation and error.
Revised: Note that HFP8 (Sun et al., 2019) and BM (Fox et al., 2020) employ different formats for activation and error.

Original: Therefore, they need to be implemented in FP153 and FP145 to support all operations with a single MAC unit (Agrawal et al., 2021).
Revised: Therefore, they need to be implemented in FP153 and FP145 to support all operations with a single MAC unit (Agrawal et al., 2021).

Original: Since Flex16+5 (K¨oster et al., 2017) requires 16-bit multiplication, its cost is signiﬁcantly higher than other 8-bit formats.
Revised: Since Flex16+5 (K¨oster et al., 2017) requires 16-bit multiplication, its cost is signiﬁcantly higher than other 8-bit formats.

Original: More detailed analysis is provided in Appendix A.7.
Revised: More detailed analysis is provided in Appendix A.7.

Original: -L OW -P RECISION T RAINING WITH 4- BIT L OGARITHMIC W
Revised: -L OW -P RECISION T RAINING WITH 4- BIT L OGARITHMIC W

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: LSTM (PTB) [ppl.]
Revised: LSTM (PTB) [ppl.]

Original: [Table]
Revised: [Table]

Original: Transformer (IWSLT)
Revised: Transformer (IWSLT)

Original: [BLEU]
Revised: [BLEU]

Original: [Table]
Revised: [Table]

Original: MobileNetV
Revised: MobileNetV

Original: [Table]
Revised: [Table]

Original: SSDLite (VOC) [
Revised: SSDLite (VOC) [

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Lower
Revised: Lower

Original: [Table]
Revised: [Table]

Original: better
Revised: better

Original: Elhoushi et al.
Revised: Elhoushi et al.

Original: (2021) recently demonstrated that 4-bit logarithmic weights could be used in quantized neural networks with minimal accuracy degradation.
Revised: (2021) recently demonstrated that 4-bit logarithmic weights could be used in quantized neural networks with minimal accuracy degradation.

Original: However, from-scratch training of the same model resulted in a 4.5% accuracy drop in ResNet-18 training (Table 4).
Revised: However, from-scratch training of the same model resulted in a 4.5% accuracy drop in ResNet-18 training (Table 4).

Original: Similarly, our experiments show 2.1% lower accuracy when training ResNet-18 using 4-bit logarithmic weights and
Revised: Similarly, our experiments show 2.1% lower accuracy when training ResNet-18 using 4-bit logarithmic weights and

Original: FP134 format for other variables.
Revised: FP134 format for other variables.

Original: However, using hysteresis quantization greatly improves the training performance, showing only 0.2% accuracy degradation.
Revised: However, using hysteresis quantization greatly improves the training performance, showing only 0.2% accuracy degradation.

Original: This is identical to the training performance achieved through ﬁne-tuning a pre-trained model by Elhoushi et al.
Revised: This is identical to the training performance achieved through ﬁne-tuning a pre-trained model by Elhoushi et al.

Original: (2021), conﬁrming that hysteresis quantization effectively solves the issue of sub-optimal solutions in from-scratch training.
Revised: (2021), conﬁrming that hysteresis quantization effectively solves the issue of sub-optimal solutions in from-scratch training.

Original: In addition, Table 5 demonstrates that hysteresis quantization improves the training performance in all target models.
Revised: In addition, Table 5 demonstrates that hysteresis quantization improves the training performance in all target models.

Original: Note that we quantized all trainable weights except for the BatchNorm parameters into 4 bits in experiments; the training performance could be further improved by using higher precision for error-sensitive parts such as the ﬁrst/last layers and residual connections.
Revised: Note that we quantized all trainable weights except for the BatchNorm parameters into 4 bits in experiments; the training performance could be further improved by using higher precision for error-sensitive parts such as the ﬁrst/last layers and residual connections.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In low-precision training, the dynamic range of a tensor is data-dependant, and hence an optimal data format depends on various factors such as model, dataset, and quantization scheme.
Revised: In low-precision training, the dynamic range of a tensor is data-dependant, and hence an optimal data format depends on various factors such as model, dataset, and quantization scheme.

Original: We showed that the training performance of a speciﬁc data format for activation and error could be predicted by observing the errors introduced in the weight gradients.
Revised: We showed that the training performance of a speciﬁc data format for activation and error could be predicted by observing the errors introduced in the weight gradients.

Original: Based on this observation, we could determine an optimal 8-bit format for low-precision training very efﬁciently without running numerous training runs.
Revised: Based on this observation, we could determine an optimal 8-bit format for low-precision training very efﬁciently without running numerous training runs.

Original: The proposed FP134 format achieved a similar or better accuracy compared to prior works, while allowing for efﬁcient hardware implementation through quantizing BatchNorm inputs and using a uniﬁed data format in both forward and backward paths.
Revised: The proposed FP134 format achieved a similar or better accuracy compared to prior works, while allowing for efﬁcient hardware implementation through quantizing BatchNorm inputs and using a uniﬁed data format in both forward and backward paths.

Original: In addition, we proposed the hysteresis quantization scheme for weight quantization, which improves training performance by suppressing undesired weight ﬂuctuations and stabilizing the training process.
Revised: In addition, we proposed the hysteresis quantization scheme for weight quantization, which improves training performance by suppressing undesired weight ﬂuctuations and stabilizing the training process.

Original: In ultra-low-precision training with 4-bit logarithmic weights, hysteresis quantization signiﬁcantly improves training performance by mitigating sub-optimal solutions, closely matching the performance obtained through ﬁne-tuning a pre-trained model.
Revised: In ultra-low-precision training with 4-bit logarithmic weights, hysteresis quantization signiﬁcantly improves training performance by mitigating sub-optimal solutions, closely matching the performance obtained through ﬁne-tuning a pre-trained model.

Original: We expect that these two schemes can complement each other to enable practical low-precision training on various models and tasks.
Revised: We expect that these two schemes can complement each other to enable practical low-precision training on various models and tasks.

Original: A A PPENDIX A. V ARIOUS F ORMATS A NALYZED IN S ECTION
Revised: A A PPENDIX A. V ARIOUS F ORMATS A NALYZED IN S ECTION

Original: In this paper, we made three assumptions on the quantization formats that were analyzed.
Revised: In this paper, we made three assumptions on the quantization formats that were analyzed.

Original: Firstly, 1-bit is allocated as a sign bit, so only symmetric formats are allowed, and secondly, the number of valid bits with a large absolute numerical value must be greater than or equal to the number of valid bits with a small absolute numerical value.
Revised: Firstly, 1-bit is allocated as a sign bit, so only symmetric formats are allowed, and secondly, the number of valid bits with a large absolute numerical value must be greater than or equal to the number of valid bits with a small absolute numerical value.

Original: Lastly, the base does not exceed 2.
Revised: Lastly, the base does not exceed 2.

Original: Considering the above assumptions, we provide a systematical approach for generating different quantization methods that were used for analysis in Section 2, in order to create quantization methods that have trade-offs in terms of dynamic range and the number of valid bits.
Revised: Considering the above assumptions, we provide a systematical approach for generating different quantization methods that were used for analysis in Section 2, in order to create quantization methods that have trade-offs in terms of dynamic range and the number of valid bits.

Original: The quantization method is expressed with the following items: i) a list of decreasing positive real numbers P that contains the interval points (Eq. 7) and ii) a non-increasing integer list L that accompanies the interval list, with each item representing the number of signiﬁcand bits (Eq. 8).
Revised: The quantization method is expressed with the following items: i) a list of decreasing positive real numbers P that contains the interval points (Eq. 7) and ii) a non-increasing integer list L that accompanies the interval list, with each item representing the number of signiﬁcand bits (Eq. 8).

Original: Here, s is shared exponent bias.
Revised: Here, s is shared exponent bias.

Original: [Equation]
Revised: [Equation]

Original: The quantization points Q are generated in each of the intervals that are sliced with 2 l k − 1 evenly distributed datapoints.
Revised: The quantization points Q are generated in each of the intervals that are sliced with 2 l k − 1 evenly distributed datapoints.

Original: If the interval is { 2 s +1 , 2 s } , the quantization point Q can be expressed by
Revised: If the interval is { 2 s +1 , 2 s } , the quantization point Q can be expressed by

Original: Eq. 9.
Revised: Eq. 9.

Original: [Equation]
Revised: [Equation]

Original: Notice that L for an α -bit quantization must satisfy
Revised: Notice that L for an α -bit quantization must satisfy

Original: [Equation]
Revised: [Equation]

Original: Since the format is symmetric, only half of the data points are assigned to positive numbers, so it is α − 1 , not α .
Revised: Since the format is symmetric, only half of the data points are assigned to positive numbers, so it is α − 1 , not α .

Original: The reason for adding 1 is to include a zero value.
Revised: The reason for adding 1 is to include a zero value.

Original: For example, when shared exponent bias is -1, an 8-bit ﬁxed-point quantization would be expressed as follows:
Revised: For example, when shared exponent bias is -1, an 8-bit ﬁxed-point quantization would be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst interval from 1 to 0.5 would be evenly sliced by 2 7 − 1 datapoints, the next interval from 0.
Revised: The ﬁrst interval from 1 to 0.5 would be evenly sliced by 2 7 − 1 datapoints, the next interval from 0.

Original: to 0.25 with 2 6 − 1 , etc.
Revised: to 0.25 with 2 6 − 1 , etc.

Original: Various cases are shown in Fig.
Revised: Various cases are shown in Fig.

Original: 6, with P plotted on the x-axis and L plotted on the y-axis.
Revised: 6, with P plotted on the x-axis and L plotted on the y-axis.

Original: Since P represents the range of values due to shared exponent bias that is independent of the data format, L can represent all of the various data formats we deal with in this paper.
Revised: Since P represents the range of values due to shared exponent bias that is independent of the data format, L can represent all of the various data formats we deal with in this paper.

Original: [Figure]
Revised: [Figure]

Original: When selecting 8-bit formats, we chose the formats so that the intervals with the number of valid bits that are less than 3 do not appear for more than two digits to reduce the search space, as they have an unnecessarily large dynamic range.
Revised: When selecting 8-bit formats, we chose the formats so that the intervals with the number of valid bits that are less than 3 do not appear for more than two digits to reduce the search space, as they have an unnecessarily large dynamic range.

Original: Thus, formats such as [7,6,5,4,2,2,2,1] were excluded from the search space.
Revised: Thus, formats such as [7,6,5,4,2,2,2,1] were excluded from the search space.

Original: Considering all of the generation rules, we selected 166 distinct 8-bit formats with different dynamic range and valid bits from [7,6,5,4,3,2,1] to [3,3,3,...,3,2,1].
Revised: Considering all of the generation rules, we selected 166 distinct 8-bit formats with different dynamic range and valid bits from [7,6,5,4,3,2,1] to [3,3,3,...,3,2,1].

Original: After the number of valid bits for an 8-bit format is selected, 1 or 2 is subtracted from each value to create a corresponding 7-bit and 6-bit formats.
Revised: After the number of valid bits for an 8-bit format is selected, 1 or 2 is subtracted from each value to create a corresponding 7-bit and 6-bit formats.

Original: For example, in the case of [6,5,5,5,5,4,4,4,3,2,1] 8-bit format, the 7-bit corresponding format is [5,4,4,4,4,3,3,3,2,1] and 6-bit corresponding format is [4,3,3,3,3,2,2,2,1].
Revised: For example, in the case of [6,5,5,5,5,4,4,4,3,2,1] 8-bit format, the 7-bit corresponding format is [5,4,4,4,4,3,3,3,2,1] and 6-bit corresponding format is [4,3,3,3,3,2,2,2,1].

Original: From the generated 166 8-bit formats, 7-bit and 6-bit formats were also generated using this rule.
Revised: From the generated 166 8-bit formats, 7-bit and 6-bit formats were also generated using this rule.

Original: A. S OFTWARE I MPLEMENTATION D ETAILS
Revised: A. S OFTWARE I MPLEMENTATION D ETAILS

Original: To support quantized training for various formats
Revised: To support quantized training for various formats

Original: [Figure]
Revised: [Figure]

Original: custom C++ and CUDA codes to emulate quantized data were written.
Revised: custom C++ and CUDA codes to emulate quantized data were written.

Original: Our custom C++ and CUDA extension code could perform quantizationrelated functions through utilizing the Python APIs in PyTorch for extendable research while maintaining high performance.
Revised: Our custom C++ and CUDA extension code could perform quantizationrelated functions through utilizing the Python APIs in PyTorch for extendable research while maintaining high performance.

Original: We emulate the quantized value using custom code in the part that needs quantization throughout the network
Revised: We emulate the quantized value using custom code in the part that needs quantization throughout the network

Original: [Figure]
Revised: [Figure]

Original: and PyTorch built-in functions are used for computation kernels such as convolution and matrix multiplication.
Revised: and PyTorch built-in functions are used for computation kernels such as convolution and matrix multiplication.

Original: We created a package named lptorch, short for low precision PyTorch
Revised: We created a package named lptorch, short for low precision PyTorch

Original: [Figure]
Revised: [Figure]

Original: and the code can be found in the supplementary material
Revised: and the code can be found in the supplementary material

Original: [Figure]
Revised: [Figure]

Original: A. A NGLE VS . M AGNITUDE TO P REDICT P ERFORMANCE
Revised: A. A NGLE VS . M AGNITUDE TO P REDICT P ERFORMANCE

Original: [Figure]
Revised: [Figure]

Original: In addition to the misalignment angles of G w ( ∠ ( G w , G w + N ∆ A ) and ∠ ( G w , G w + N ∆ E ) ), as deﬁned in Section 2.3, we used the absolute values of noise ( | N ∆
Revised: In addition to the misalignment angles of G w ( ∠ ( G w , G w + N ∆ A ) and ∠ ( G w , G w + N ∆ E ) ), as deﬁned in Section 2.3, we used the absolute values of noise ( | N ∆

Original: A | and | N ∆ E | ) in order to predict the ﬁnal trained performance, and the results are shown in Fig.
Revised: A | and | N ∆ E | ) in order to predict the ﬁnal trained performance, and the results are shown in Fig.

Original: Fig.
Revised: Fig.

Original: 3 and Fig.
Revised: 3 and Fig.

Original: 7 show that both the error magnitude and the misalignment angle are good metrics for determining optimal data format.
Revised: 7 show that both the error magnitude and the misalignment angle are good metrics for determining optimal data format.

Original: For the six target models, both metrics suggest FP134 as the best format.
Revised: For the six target models, both metrics suggest FP134 as the best format.

Original: However, the misalignment angle still better captures the training performance.
Revised: However, the misalignment angle still better captures the training performance.

Original: For instance, in Fig.
Revised: For instance, in Fig.

Original: 7(e), although FP134 shows smaller noise magnitude, the actual training loss is smaller for FP143.
Revised: 7(e), although FP134 shows smaller noise magnitude, the actual training loss is smaller for FP143.

Original: Similarly, in Fig. 7(b), (c) and (f), although INT8 failed and FP152 succeeded in training, the absolute value of noise did not indicate a clear superior of the two formats.
Revised: Similarly, in Fig. 7(b), (c) and (f), although INT8 failed and FP152 succeeded in training, the absolute value of noise did not indicate a clear superior of the two formats.

Original: Based on these observations, we conclude that the misalignment angles are more suitable for predicting training performance compared against using the absolute value of noise.
Revised: Based on these observations, we conclude that the misalignment angles are more suitable for predicting training performance compared against using the absolute value of noise.

Original: A. H YSTERESIS Q UANTIZATION WITH I NTEGER W EIGHTS
Revised: A. H YSTERESIS Q UANTIZATION WITH I NTEGER W EIGHTS

Original: In addition to 4-bit logarithmic weights, we also tested the hysteresis quantization scheme on a lowprecision integer format (INT4) that uses uniform quantization.
Revised: In addition to 4-bit logarithmic weights, we also tested the hysteresis quantization scheme on a lowprecision integer format (INT4) that uses uniform quantization.

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: Experimental results show that using hysteresis improves the performance in most cases.
Revised: Experimental results show that using hysteresis improves the performance in most cases.

Original: In addition, in MobileNetV2 training with INT4 weights, training initially failed, but using hysteresis enables reliable training, which suggests that hysteresis quantization not only helps the network to reach the optimal point but also prevents divergence in an unwanted direction during the training process.
Revised: In addition, in MobileNetV2 training with INT4 weights, training initially failed, but using hysteresis enables reliable training, which suggests that hysteresis quantization not only helps the network to reach the optimal point but also prevents divergence in an unwanted direction during the training process.

Original: However, it is interesting to see that the hysteresis quantization is less effective on the LSTM model for the INT4 format.
Revised: However, it is interesting to see that the hysteresis quantization is less effective on the LSTM model for the INT4 format.

Original: We suspect that this is due to the weight distribution characteristics of the
Revised: We suspect that this is due to the weight distribution characteristics of the

Original: Model
Revised: Model

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ppl
Revised: ppl

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: LSTM model.
Revised: LSTM model.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 8, most of the weights have a relatively large magnitude in the
Revised: 8, most of the weights have a relatively large magnitude in the

Original: LSTM model when normalized, contrary to ResNet-18 in which the weights are more evenly distributed.
Revised: LSTM model when normalized, contrary to ResNet-18 in which the weights are more evenly distributed.

Original: In logarithmic formats, the relative amount of quantization error is similar for all values.
Revised: In logarithmic formats, the relative amount of quantization error is similar for all values.

Original: In contrast, the relative amount of quantization error is smaller for large values in uniform quantization.
Revised: In contrast, the relative amount of quantization error is smaller for large values in uniform quantization.

Original: Therefore, the weight parameters of LSTM are more severely affected by ﬂuctuation in logarithmic formats, making our hysteresis quantization scheme more effective in those formats compared to uniform quantization.
Revised: Therefore, the weight parameters of LSTM are more severely affected by ﬂuctuation in logarithmic formats, making our hysteresis quantization scheme more effective in those formats compared to uniform quantization.

Original: A.
Revised: A.

Original: E XPERIMENTAL D ETAILS
Revised: E XPERIMENTAL D ETAILS

Original: A.5.
Revised: A.5.

Original: [Equation]
Revised: [Equation]

Original: 18 (I MAGE N ET )
Revised: 18 (I MAGE N ET )

Original: We conducted ImageNet experiments using SGD with a momentum of 0.9 for 90 epochs with a batch size of 256 images and an initial learning rate of 0.1 which is decayed by a factor of 10 at the 30th and 60th epochs.
Revised: We conducted ImageNet experiments using SGD with a momentum of 0.9 for 90 epochs with a batch size of 256 images and an initial learning rate of 0.1 which is decayed by a factor of 10 at the 30th and 60th epochs.

Original: We used the ResNet-18 architecture from the ofﬁcial PyTorch implementation 1 .
Revised: We used the ResNet-18 architecture from the ofﬁcial PyTorch implementation 1 .

Original: Fig.shows Top-1 training & validation accuracy graphs.
Revised: Fig.shows Top-1 training & validation accuracy graphs.

Original: Observation of the training graph indicates that all of the results are close to the baseline within 0.2% with the exception of FP130 without hysteresis quantization.
Revised: Observation of the training graph indicates that all of the results are close to the baseline within 0.2% with the exception of FP130 without hysteresis quantization.

Original: [Figure]
Revised: [Figure]

Original: We trained ResNet-101 by applying the same training method as ResNet-18.
Revised: We trained ResNet-101 by applying the same training method as ResNet-18.

Original: We conducted ImageNet experiments using SGD with a momentum of 0.9 for 90 epochs with a batch size ofimages and an initial learning rate of 0.1 which is decayed by a factor of 10 at the 30th and 60th epochs.
Revised: We conducted ImageNet experiments using SGD with a momentum of 0.9 for 90 epochs with a batch size ofimages and an initial learning rate of 0.1 which is decayed by a factor of 10 at the 30th and 60th epochs.

Original: We used the ResNet-101 architecture from the ofﬁcial PyTorch implementation 2 .
Revised: We used the ResNet-101 architecture from the ofﬁcial PyTorch implementation 2 .

Original: Fig.shows Top-1 training & validation accuracy graphs.
Revised: Fig.shows Top-1 training & validation accuracy graphs.

Original: Observation of the training graph indicates that all of the results are close to the baseline with less than 0.3% performance drop except for FP without hysteresis quantization.
Revised: Observation of the training graph indicates that all of the results are close to the baseline with less than 0.3% performance drop except for FP without hysteresis quantization.

Original: [Figure]
Revised: [Figure]

Original: We conducted ImageNet experiments using SGD with a momentum of 0.9 for 270 epochs with a batch size of 256 images and cosine annealing with an initial learning rate of 0.05.
Revised: We conducted ImageNet experiments using SGD with a momentum of 0.9 for 270 epochs with a batch size of 256 images and cosine annealing with an initial learning rate of 0.05.

Original: We used the MobileNetV2 architecture from the ofﬁcial PyTorch implementation 3 .
Revised: We used the MobileNetV2 architecture from the ofﬁcial PyTorch implementation 3 .

Original: Fig.
Revised: Fig.

Original: 11 shows Top1 training & validation accuracy graphs.
Revised: 11 shows Top1 training & validation accuracy graphs.

Original: Observation of the training graph indicates that FP without hysteresis leads to very unstable ﬂuctuations throughout the training.
Revised: Observation of the training graph indicates that FP without hysteresis leads to very unstable ﬂuctuations throughout the training.

Original: On the other hand, in FP130 with hysteresis, training is less susceptible to ﬂuctuations and follows the baseline (FP32) training closely until the learning rate decreases toward the latter part of learning, where both FP with hysteresis and FP134 show some degradation from the baseline.
Revised: On the other hand, in FP130 with hysteresis, training is less susceptible to ﬂuctuations and follows the baseline (FP32) training closely until the learning rate decreases toward the latter part of learning, where both FP with hysteresis and FP134 show some degradation from the baseline.

Original: This is seen as a limitation due to the low precision of each format.
Revised: This is seen as a limitation due to the low precision of each format.

Original: A.5. 2- LAYER LSTM (PTB)
Revised: A.5. 2- LAYER LSTM (PTB)

Original: We adopted the 2-layer Long Short Term Memory (LSTM) network from PyTorch Examples for language modeling on the Penn Treebank dataset (Marcus et al., 1993).
Revised: We adopted the 2-layer Long Short Term Memory (LSTM) network from PyTorch Examples for language modeling on the Penn Treebank dataset (Marcus et al., 1993).

Original: We ran experiments in batches of 20 sentences with an initial learning rate of 20 which is decayed by a factor of 4 at epoch 11, 16, 26, 31 and 37.
Revised: We ran experiments in batches of 20 sentences with an initial learning rate of 20 which is decayed by a factor of 4 at epoch 11, 16, 26, 31 and 37.

Original: The embedding and hidden dimensions are 650 and the sequence length is 35.
Revised: The embedding and hidden dimensions are 650 and the sequence length is 35.

Original: Fig.
Revised: Fig.

Original: 12 shows training & validation perplexity.
Revised: 12 shows training & validation perplexity.

Original: A.5. T RANSFORMER M ODEL (IWLST)
Revised: A.5. T RANSFORMER M ODEL (IWLST)

Original: We adopted the Transformer Base model from the FairSeq 5 repository on the IWSLT’14 German to English translation task.
Revised: We adopted the Transformer Base model from the FairSeq 5 repository on the IWSLT’14 German to English translation task.

Original: We used Adam optimizer and default training parameters found in the repository and trained from scratch for 25 epochs.
Revised: We used Adam optimizer and default training parameters found in the repository and trained from scratch for 25 epochs.

Original: BLEU scores were calculated using the script from the repository.
Revised: BLEU scores were calculated using the script from the repository.

Original: A.5. M OBILE N ET
Revised: A.5. M OBILE N ET

Original: [Equation]
Revised: [Equation]

Original: + SSDL ITE (VOC)
Revised: + SSDL ITE (VOC)

Original: We adopted a PyTorch implementation of SSDLite from the online repository 6 .
Revised: We adopted a PyTorch implementation of SSDLite from the online repository 6 .

Original: The base network is MobileNetV2 which was pretrained with each format in Appendix A.5.3.
Revised: The base network is MobileNetV2 which was pretrained with each format in Appendix A.5.3.

Original: The entire network is trained on VOC2012 and VOC2007 trainval datasets and evaluated on VOC2007 validation dataset.
Revised: The entire network is trained on VOC2012 and VOC2007 trainval datasets and evaluated on VOC2007 validation dataset.

Original: We used SGD with a momentum of 0.9 for 200 epochs in batches of 32 images and cosine annealing with an initial learning rate of 0.01.
Revised: We used SGD with a momentum of 0.9 for 200 epochs in batches of 32 images and cosine annealing with an initial learning rate of 0.01.

Original: Fig.
Revised: Fig.

Original: 13 shows validation loss at every 5 epochs.
Revised: 13 shows validation loss at every 5 epochs.

Original: Even in this experiment, in the case of FP130 without hysteresis the loss ﬂuctuates signiﬁcantly, whereas in FP130 with hysteresis learning proceeds much more stably.
Revised: Even in this experiment, in the case of FP130 without hysteresis the loss ﬂuctuates signiﬁcantly, whereas in FP130 with hysteresis learning proceeds much more stably.

Original: FP134 showed similar results to the baseline regardless of hysteresis quantization.
Revised: FP134 showed similar results to the baseline regardless of hysteresis quantization.

Original: A. M ODEL Q UANTIZATION M ETHODS
Revised: A. M ODEL Q UANTIZATION M ETHODS

Original: We quantized GEMM input and batchnorm input in all quantized training experiments.
Revised: We quantized GEMM input and batchnorm input in all quantized training experiments.

Original: Among the six models used in the experiment, the quantization details for three representative structures are
Revised: Among the six models used in the experiment, the quantization details for three representative structures are

Original: [Figure]
Revised: [Figure]

Original: Quant
Revised: Quant

Original: ReLU
Revised: ReLU

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: Quant +
Revised: Quant +

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: ReLU
Revised: ReLU

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: Conv Quant
Revised: Conv Quant

Original: BN Quant
Revised: BN Quant

Original: Quant
Revised: Quant

Original: [Figure]
Revised: [Figure]

Original: shown in the Fig.
Revised: shown in the Fig.

Original: In each structure of ﬁgure, inputs such as x, c, h, V, K, and Q are also all quantized in 8 bits.
Revised: In each structure of ﬁgure, inputs such as x, c, h, V, K, and Q are also all quantized in 8 bits.

Original: A. D ETAILS ON H ARDWARE I MPLEMENTATION
Revised: A. D ETAILS ON H ARDWARE I MPLEMENTATION

Original: A conventional MAC unit consists of a multiplier and an accumulator.
Revised: A conventional MAC unit consists of a multiplier and an accumulator.

Original: In the multiplier, the exponents of two input operands are summed while their mantissas are multiplied.
Revised: In the multiplier, the exponents of two input operands are summed while their mantissas are multiplied.

Original: The multiplication part is more complex, and hence it dominates the area of the multiplier.
Revised: The multiplication part is more complex, and hence it dominates the area of the multiplier.

Original: As a result, the size of the multiplier is larger when more bits are allocated to mantissa.
Revised: As a result, the size of the multiplier is larger when more bits are allocated to mantissa.

Original: In the accumulator, a ﬂoating-point adder adds the multiplication results to a partial sum in FP169.
Revised: In the accumulator, a ﬂoating-point adder adds the multiplication results to a partial sum in FP169.

Original: The adder is decomposed into a shifter that aligns the mantissa by the exponent difference, an integer adder that sums aligned mantissas, and a quantization unit that converts the result back to FP169.
Revised: The adder is decomposed into a shifter that aligns the mantissa by the exponent difference, an integer adder that sums aligned mantissas, and a quantization unit that converts the result back to FP169.

Original: Since the result is re-quantized into FP169, the addition operation of aligned mantissas does not need to be lossless.
Revised: Since the result is re-quantized into FP169, the addition operation of aligned mantissas does not need to be lossless.

Original: FP169 format has a 10-bit mantissa including one hidden bit.
Revised: FP169 format has a 10-bit mantissa including one hidden bit.

Original: Therefore, we only need to accurately calculate higher 10 bits, which necessitates a 12-bit adder considering rounding.
Revised: Therefore, we only need to accurately calculate higher 10 bits, which necessitates a 12-bit adder considering rounding.

Original: Therefore, shifting by more than 12 bits is not needed even if the result of the multiplier has a larger exponent range.
Revised: Therefore, shifting by more than 12 bits is not needed even if the result of the multiplier has a larger exponent range.

Original: Therefore, the shifter, adder, and quantization unit, which are the components of the accumulator, are not affected by the input format.
Revised: Therefore, the shifter, adder, and quantization unit, which are the components of the accumulator, are not affected by the input format.

Original: There are minor differences such as an adder that calculates the difference between exponents and a shifter with a different bit width of the input, but their costs are ignorable.
Revised: There are minor differences such as an adder that calculates the difference between exponents and a shifter with a different bit width of the input, but their costs are ignorable.

Original: Contrarily, a multi-way MAC consists of a multiplier, a shifter for alignment, an adder tree, a normalization unit, and a ﬁnal accumulator.
Revised: Contrarily, a multi-way MAC consists of a multiplier, a shifter for alignment, an adder tree, a normalization unit, and a ﬁnal accumulator.

Original: The multiplier and the ﬁnal accumulator are identical to sign exp man sign exp man x + XOR n bit m bit 1 bit subnormal << 2m+2 bit n+1 bit signed shifted number 2 n+

Revised: The multiplier and the ﬁnal accumulator are identical to sign exp man sign exp man x + XOR n bit m bit 1 bit subnormal << 2m+2 bit n+1 bit signed shifted number 2 n+

Original: +2m-2 bit 2 n+1 +2m-1 bit α α-input adder tree +
Revised: +2m-2 bit 2 n+1 +2m-1 bit α α-input adder tree +

Original: FP
Revised: FP

Original: FP sign exp man sign exp man x + XOR n bit m bit 1 bit subnormal << 2m+2 bit n+1 bit signed shifted number 2 n+1 +2m-2 bit 2 n+1 +2m-1 bit α α-input adder tree +
Revised: FP sign exp man sign exp man x + XOR n bit m bit 1 bit subnormal << 2m+2 bit n+1 bit signed shifted number 2 n+1 +2m-2 bit 2 n+1 +2m-1 bit α α-input adder tree +

Original: FP FP
Revised: FP FP

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) those of the conventional MAC.
Revised: ) those of the conventional MAC.

Original: However, since only one normalization unit and one ﬁnal accumulator are shared across multiple inputs, their implementation cost becomes insigniﬁcant for a larger number of inputs.
Revised: However, since only one normalization unit and one ﬁnal accumulator are shared across multiple inputs, their implementation cost becomes insigniﬁcant for a larger number of inputs.

Original: The shifter for alignment converts the multiplier output to an integer format since the cost of integer addition is lower than that of ﬂoating-point addition.
Revised: The shifter for alignment converts the multiplier output to an integer format since the cost of integer addition is lower than that of ﬂoating-point addition.

Original: Then, the adder tree sums those integer values, and the normalization unit converts the result back to a ﬂoating-point format.
Revised: Then, the adder tree sums those integer values, and the normalization unit converts the result back to a ﬂoating-point format.

Original: The cost of the shifter for alignment, adder tree, and normalization unit is all determined by the integer bit width, and the larger the exponent range of the input operands, the larger the required bit width, as shown in Fig.
Revised: The cost of the shifter for alignment, adder tree, and normalization unit is all determined by the integer bit width, and the larger the exponent range of the input operands, the larger the required bit width, as shown in Fig.

Original: In FP134, FP143, and FP152, the minimum integer bit widths are 23, 37, and 67 bits, respectively.
Revised: In FP134, FP143, and FP152, the minimum integer bit widths are 23, 37, and 67 bits, respectively.

Original: Since the bit width is sufﬁciently large, the cost difference of these units exceeds the cost difference of the multiplier.
Revised: Since the bit width is sufﬁciently large, the cost difference of these units exceeds the cost difference of the multiplier.

Original: Therefore, the cost of a multi-way MAC increases with the number of exponent bits.
Revised: Therefore, the cost of a multi-way MAC increases with the number of exponent bits.

Original: In addition to the synthesis result for ASIC implementation in Table 3, we measured the hardware overhead of MAC units of different data formats on FPGA.
Revised: In addition to the synthesis result for ASIC implementation in Table 3, we measured the hardware overhead of MAC units of different data formats on FPGA.

Original: Table 7 shows the synthesis results on Xilinx Artix-7 FPGA (XC7A100TCSG324-1).
Revised: Table 7 shows the synthesis results on Xilinx Artix-7 FPGA (XC7A100TCSG324-1).

Original: Those MAC units do not need block RAMs (BRAMs), and we used a compiler directive to avoid using DSP modules for fair comparisons.
Revised: Those MAC units do not need block RAMs (BRAMs), and we used a compiler directive to avoid using DSP modules for fair comparisons.

Original: Table 7 shows a similar trend to Table 3; the cost of one MAC gradually decreases as the number of inputs increases in the multi-way MAC.
Revised: Table 7 shows a similar trend to Table 3; the cost of one MAC gradually decreases as the number of inputs increases in the multi-way MAC.

Original: Also, due to integer-based addition in the adder tree, the cost of FP134, which has the smallest dynamic range, exhibits lower costs than the other formats.
Revised: Also, due to integer-based addition in the adder tree, the cost of FP134, which has the smallest dynamic range, exhibits lower costs than the other formats.

Original: A. E FFECT OF Q UANTIZATION N OISE ON D ATA F LOW Q UANTIZATION
Revised: A. E FFECT OF Q UANTIZATION N OISE ON D ATA F LOW Q UANTIZATION

Original: [Table]
Revised: [Table]

Original: SSDLite(VOC)

Original: [Table] Table 8 shows the training results when both activation and error are quantized in various data formats.
Revised: Table 8 shows the training results when both activation and error are quantized in various data formats.

Original: If an appropriate amount of noise is introduced in the network during training, it will increase the training loss but reduce the validation loss, suggesting that the model has been improved due to the regularization effect.
Revised: If an appropriate amount of noise is introduced in the network during training, it will increase the training loss but reduce the validation loss, suggesting that the model has been improved due to the regularization effect.

Original: However, if the noise level continues to increase, the model’s performance will start to degrade at some point.
Revised: However, if the noise level continues to increase, the model’s performance will start to degrade at some point.

Original: For instance, when MobileNetV2 is quantized in FP134, its performance is improved through the regularization effect since the training loss increases while the validation loss decreases compared to FP32.
Revised: For instance, when MobileNetV2 is quantized in FP134, its performance is improved through the regularization effect since the training loss increases while the validation loss decreases compared to FP32.

Original: However, both the training and validation losses increase when quantized in most cases, resulting in lower accuracy.
Revised: However, both the training and validation losses increase when quantized in most cases, resulting in lower accuracy.

Original: This suggests that using a very low precision data format already introduces a large amount of noise in the network, incurring performance degradation.
Revised: This suggests that using a very low precision data format already introduces a large amount of noise in the network, incurring performance degradation.

Original: Hence, it is necessary to reduce error in the network to improve the training performance in low-precision training.
Revised: Hence, it is necessary to reduce error in the network to improve the training performance in low-precision training.

Revised: A. O THER H ARDWARE C OMPONENTS A FFECTED BY D ATA F ORMAT

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: When designing a neural network training processor, some parts of the hardware (e.g., batch normalization, non-linear activation functions such as tanh and sigmoid, and softmax function) are typically implemented with higher precision to avoid performance drop.

Revised: Hence, we need to consider data format conversion overheads when comparing different formats.

Revised: If we consider various 8-bit data formats with different representation methods, as we did in Table 3, and assume that computations other than MAC operations are implemented in full precision, the processing architecture (except MAC units) will be identical for all formats.

Revised: In addition, the on/off-chip memory space, control logics, and on-chip interconnects will remain the same.

Revised: The only difference would be the low-precision MAC units and the data conversion units between full-precision and low-precision formats.

Revised: However, the cost of conversion between low-precision and high-precision ﬂoating-point formats is typically very low and does not vary much with the low-precision format.

Revised: For low-precision to high-precision conversion, we only have to add a bias-correction term to the exponent and add 0 after the mantissa.

Revised: For high-precision to low-precision conversion, we need to add a bias-correction term to the exponent, clamp the overﬂowed value to the maximum, and round off the mantissa.

Revised: The cost is very low compared to the MAC operation, and the cost difference between different low-precision formats is negligible.

Revised: We have synthesized the conversion units for different formats, and their costs are presented in Table 9.

Revised: The experimental results conﬁrm that the overhead of data format conversion is signiﬁcantly lower than MAC operations.

Revised: In addition, all formats except Flexpoint exhibit similar conversion costs.