Original: B EYOND
Revised: B EYOND

Original: Q UANTIZATION
Revised: Q UANTIZATION

Original: : P OWER AWARE NEURAL NETWORKS
Revised: : P OWER AWARE NEURAL NETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Power consumption is a major obstacle in the deployment of deep neural networks (DNNs) on end devices.
Revised: Power consumption is a major obstacle in the deployment of deep neural networks (DNNs) on end devices.

Original: Existing approaches for reducing power consumption rely on quite general principles, including avoidance of multiplication operations and aggressive quantization of weights and activations.
Revised: Existing approaches for reducing power consumption rely on quite general principles, including avoidance of multiplication operations and aggressive quantization of weights and activations.

Original: However, these methods do not take into account the precise power consumed by each module in the network, and are therefore far from optimal.
Revised: However, these methods do not take into account the precise power consumed by each module in the network, and are therefore far from optimal.

Original: In this paper we develop accurate power consumption models for all arithmetic operations in the DNN, under various working conditions.
Revised: In this paper we develop accurate power consumption models for all arithmetic operations in the DNN, under various working conditions.

Original: Surprisingly, we reveal several important factors that have been overlooked to date.
Revised: Surprisingly, we reveal several important factors that have been overlooked to date.

Original: Based on our analysis, we present PANN (power-aware neural network), a simple approach for approximating any full-precision network by a lowpower ﬁxed-precision variant.
Revised: Based on our analysis, we present PANN (power-aware neural network), a simple approach for approximating any full-precision network by a lowpower ﬁxed-precision variant.

Original: Our method can be applied to a pre-trained network, and can also be used during training to achieve improved performance.
Revised: Our method can be applied to a pre-trained network, and can also be used during training to achieve improved performance.

Original: In contrast to previous methods, PANN incurs only a minor degradation in accuracy w.r.t.
Revised: In contrast to previous methods, PANN incurs only a minor degradation in accuracy w.r.t.

Original: the full-precision version of the network, even when working at the power-budget of a 2-bit quantized variant.
Revised: the full-precision version of the network, even when working at the power-budget of a 2-bit quantized variant.

Original: In addition, our scheme enables to seamlessly traverse the power-accuracy trade-off at deployment time, which is a major advantage over existing quantization methods that are constrained to speciﬁc bit widths.
Revised: In addition, our scheme enables to seamlessly traverse the power-accuracy trade-off at deployment time, which is a major advantage over existing quantization methods that are constrained to speciﬁc bit widths.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the ever increasing popularity of deep neural networks (DNNs) for tasks like face detection, voice recognition, and image enhancement, power consumption has become one of the major considerations in the design of DNNs for resource-limited end-devices.
Revised: With the ever increasing popularity of deep neural networks (DNNs) for tasks like face detection, voice recognition, and image enhancement, power consumption has become one of the major considerations in the design of DNNs for resource-limited end-devices.

Original: Over the last several years, a plethora of approaches have been introduced for achieving power efﬁciency in DNNs.
Revised: Over the last several years, a plethora of approaches have been introduced for achieving power efﬁciency in DNNs.

Original: These range from specialized architectures (Sandler et al., 2018; Huang et al., 2019; Tan et al., 2019; Radosavovic et al., 2020), to hardware oriented methods like multiplier-free designs and low-precision arithmetic.
Revised: These range from specialized architectures (Sandler et al., 2018; Huang et al., 2019; Tan et al., 2019; Radosavovic et al., 2020), to hardware oriented methods like multiplier-free designs and low-precision arithmetic.

Original: Multiplier aware methods attempt to reduce power consumption by avoiding the costly multiplication operations, which dominate the computations in a DNN.
Revised: Multiplier aware methods attempt to reduce power consumption by avoiding the costly multiplication operations, which dominate the computations in a DNN.

Original: Several works replaced multiplications by additions (Courbariaux et al., 2015;
Revised: Several works replaced multiplications by additions (Courbariaux et al., 2015;

Original: Li et al., 2016; Chen et al., 2020) or by bit shift operations (Elhoushi et al., 2019) or both (You et al., 2020).
Revised: Li et al., 2016; Chen et al., 2020) or by bit shift operations (Elhoushi et al., 2019) or both (You et al., 2020).

Original: Others employed efﬁcient matrix multiplication operators (Tschannen et al., 2018; Lavin & Gray, 2016).
Revised: Others employed efﬁcient matrix multiplication operators (Tschannen et al., 2018; Lavin & Gray, 2016).

Original: However, most methods in this category introduce dedicated architectures, which require training the network from scratch.
Revised: However, most methods in this category introduce dedicated architectures, which require training the network from scratch.

Original: This poses a severe limitation, as different variants of the network need to be trained for different power constraints.
Revised: This poses a severe limitation, as different variants of the network need to be trained for different power constraints.

Original: Low-precision DNNs reduce power consumption by using low-precision arithmetic.
Revised: Low-precision DNNs reduce power consumption by using low-precision arithmetic.

Original: This is done either via quantization-aware training (QAT) or with post-training quantization techniques.
Revised: This is done either via quantization-aware training (QAT) or with post-training quantization techniques.

Original: The latter avoid the need for retraining the network but often still require access to a small number of calibration samples in order to adapt the network’s weights.
Revised: The latter avoid the need for retraining the network but often still require access to a small number of calibration samples in order to adapt the network’s weights.

Original: Such techniques include approaches like re-training, ﬁne-tuning, calibration and optimization (Banner et al., 2019;
Revised: Such techniques include approaches like re-training, ﬁne-tuning, calibration and optimization (Banner et al., 2019;

Original: Jacob et al., 2018; Nahshan et al., 2019; Li et al., 2021).
Revised: Jacob et al., 2018; Nahshan et al., 2019; Li et al., 2021).

Original: All existing methods in this category suffer from a large drop in accuracy with respect to the full-precision version of the network. Especially, when working at very low bit widths.
Revised: All existing methods in this category suffer from a large drop in accuracy with respect to the full-precision version of the network, especially when working at very low bit widths.

Original: Moreover, similarly to the multiplier-free approaches, they do not provide a mechanism for traversing the power-accuracy trade-off without actually changing the hardware ( e.g. , replacing an 8 -bit multiplier by a 4 -bit one).
Revised: Moreover, similarly to the multiplier-free approaches, they do not provide a mechanism for traversing the power-accuracy trade-off without actually changing the hardware ( e.g. , replacing an 8 -bit multiplier by a 4 -bit one).

Original: In this work, we introduce a power-aware neural network (PANN) approach that allows to dramatically cut down the power consumption of any model.
Revised: In this work, we introduce a power-aware neural network (PANN) approach that allows to dramatically cut down the power consumption of any model.

Original: Our method can be applied at post-training to improve the power efﬁciency of a pre-trained model, or in a QAT setting
Revised: Our method can be applied at post-training to improve the power efﬁciency of a pre-trained model, or in a QAT setting

Original: [Figure]
Revised: [Figure]

Original: for the level See more examples to obtain even improved results.
Revised: to obtain even improved results.

Original: Our approach is based on careful analysis of the power consumed by additions and multiplications, as functions of several factors.
Revised: Our approach is based on careful analysis of the power consumed by additions and multiplications, as functions of several factors.

Original: We rely on bit toggling activity, which is known to be the main factor affecting dynamic power consumption, and support our theoretical analysis with accurate gate-level simulations on a 5 nm process.
Revised: We rely on bit toggling activity, which is known to be the main factor affecting dynamic power consumption and support our theoretical analysis with accurate gate-level simulations on a 5 nm process.

Original: Our ﬁrst important observation is that a major portion of the power consumed by a DNN is due to the use of signed integers.
Revised: Our ﬁrst important observation is that a major portion of the power consumed by a DNN is due to the use of signed integers.

Original: We therefore present a simple method for converting any pre-trained model to use unsigned arithmetic.
Revised: We therefore present a simple method for converting any pre-trained model to use unsigned arithmetic.

Original: This conversion does not change the functionality of the model and, as can be seen in Fig.
Revised: This conversion does not change the functionality of the model and, as can be seen in Fig.

Original: 1, dramatically reduces power consumption on common hardware conﬁgurations.
Revised: 1, dramatically reduces power consumption on common hardware conﬁgurations.

Original: Our second observation is that the multiplier’s power consumption is dominated by the larger bit width among its two inputs.
Revised: Our second observation is that the multiplier’s power consumption is dominated by the larger bit width among its two inputs.

Original: Therefore, although high accuracy can often be achieved with quite drastic quantization of only the weights, this common practice turns out to be ineffective in terms of power consumption.
Revised: Therefore, although high accuracy can often be achieved with quite drastic quantization of only the weights, this common practice turns out to be ineffective in terms of power consumption.

Original: To be able to take advantage of drastic weight quantization, here we introduce a method that allows removing the multiplier altogether.
Revised: To be able to take advantage of drastic weight quantization, here we introduce a method that allows removing the multiplier altogether.

Original: Our approach cancan work in combination with any activation quantization method.
Revised: Our approach can work in combination with any activation quantization method.

Original: We show theoretically and experimentally that this method is far advantageous over existing quantization methods at low power budgets, both at post-training and in QAT settings (see Fig. 1 and Sec. 6).
Revised: We show theoretically and experimentally that this method is far advantageous over existing quantization methods at low power budgets, both at post-training and in QAT settings (see Fig. 1 and Sec. 6).

Original: Our method allows working under any power constraint by tuning the number of additions used to approximate each mutltiply-accumulate (MAC) operation.
Revised: Our method allows working under any power constraint by tuning the number of additions used to approximate each mutltiply-accumulate (MAC) operation.

Original: This is in contrast to regular quantization methods, which are limited to particular values.
Revised: This is in contrast to regular quantization methods, which are limited to particular values.

Original: We can thus traverse the power-accuracy trade-off without changing the architecture ( e.g. , bit width of the multiplier), as required by existing methods.
Revised: We can thus traverse the power-accuracy trade-off without changing the architecture ( e.g. , bit width of the multiplier), as required by existing methods.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Avoiding multiplications In ﬁxed point (integer) representation, additions are typically much more power-efﬁcient than multiplications (Horowitz, 2014b;a).
Revised: Avoiding multiplications In ﬁxed point (integer) representation, additions are typically much more power-efﬁcient than multiplications (Horowitz, 2014b;a).

Original: Some works suggested to binarize or ternarize the weights to enable working with additions only (Courbariaux et al., 2015;
Revised: Some works suggested to binarize or ternarize the weights to enable working with additions only (Courbariaux et al., 2015;

Original: Lin et al., 2015; Li et al., 2016).
Revised: Lin et al., 2015; Li et al., 2016).

Original: However, this often severely impairs the network’s accuracy.
Revised: However, this often severely impairs the network’s accuracy.

Original: Recent works suggested to replace multiplications by bit shifts (Elhoushi et al., 2019) or additions (Chen et al., 2020) or both (You et al., 2020).
Revised: Recent works suggested to replace multiplications by bit shifts (Elhoushi et al., 2019) or additions (Chen et al., 2020) or both (You et al., 2020).

Original: Other methods reduce the number of multiplications by inducingsparsity (Venkatesh et al., 2016; Mahmoud et al., 2020), decomposition into smaller intermediate
Revised: Other methods reduce the number of multiplications by inducing

Original: [Figure]
Revised: [Figure]

Original: p li e r
Revised: p li e r

Original: C onv e r t e rsummed with the
Revised: C onv e r t e r

Original: [Figure]
Revised: [Figure]

Original: toggles

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: MAC .
Revised: MAC .

Original: The b-bit multiplier inputs are drawn uniformly from [ − 2 b − 1 , 2 b − 1 ) and its b acc = 2 b bit output is summedwith the B -bit number in the FF. products (Kim et al., 2016), Winograd based convolutions (Lavin & Gray, 2016), or Strassen’s matrix multiplication algorithm (Tschannen et al., 2018).
Revised: The b bit drawn summed the sparsity (Venkatesh et al., 2016; Mahmoud et al., 2020), decomposition into smaller intermediate products (Kim et al., 2016), Winograd based convolutions (Lavin & Gray, 2016), or Strassen’s matrix multiplication algorithm (Tschannen et al., 2018).

Original: Some of these methods require internal changes in the model, a dedicated backpropagation scheme, or other modiﬁcations to the training process.
Revised: Some of these methods require internal changes in the model, a dedicated backpropagation scheme, or other modiﬁcations to the training process.

Original: Quantization DNN quantization approaches include post-training quantization (PTQ), which is applied to a pre-trained model, and quantization-aware training (QAT), where the network’s weights are adapted to the quantization during training (Gupta et al., 2015; Louizos et al., 2018; Achterhold et al., 2018; Esser et al., 2019).
Revised: Quantization DNN quantization approaches include post-training quantization (PTQ), which is applied to a pre-trained model, and quantization-aware training (QAT), where the network’s weights are adapted to the quantization during training (Gupta et al., 2015; Louizos et al., 2018; Achterhold et al., 2018; Esser et al., 2019).

Original: PTQ methods are more ﬂexible in that they do not require access to the training set.
Revised: PTQ methods are more ﬂexible in that they do not require access to the training set.

Original: These methods show optimal results for 8 -bit quantization, but tend to incur a large drop in accuracy at low bit widths.
Revised: These methods show optimal results for 8 -bit quantization, but tend to incur a large drop in accuracy at low bit widths.

Original: To battle this effect, some PTQ methods minimize the quantization errors of each layer individually by optimizing the parameters over a calibration set (Nahshan et al., 2019; Nagel et al., 2020; Hubara et al., 2020).
Revised: To battle this effect, some PTQ methods minimize the quantization errors of each layer individually by optimizing the parameters over a calibration set (Nahshan et al., 2019; Nagel et al., 2020; Hubara et al., 2020).

Original: Others use nonuniform quantization (Liu et al., 2021; Fang et al., 2020).
Revised: Others use nonuniform quantization (Liu et al., 2021; Fang et al., 2020).

Original: Effort is also invested in avoiding the need of any data sample for calibration (Cai et al., 2020; Shoukai et al., 2020; Nagel et al., 2019; Haroush et al., 2020).
Revised: Effort is also invested in avoiding the need of any data sample for calibration (Cai et al., 2020; Shoukai et al., 2020; Nagel et al., 2019; Haroush et al., 2020).

Original: These methods, however, still show a signiﬁcant drop in accuracy at the lower bit widths, while frequently requiring additional computational resources.
Revised: These methods, however, still show a signiﬁcant drop in accuracy at the lower bit widths, while frequently requiring additional computational resources.

Original: Common to all quantization works is that they lack analysis of the power consumed by each arithmetic operation as a function of bit-width, and thus cannot strive for optimal power-accuracy trade-offs.
Revised: Common to all quantization works is that they lack analysis of the power consumed by each arithmetic operation as a function of bit-width, and thus cannot strive for optimal power-accuracy trade-offs.

Original:  P OWER CONSUMPTION OF A CONVENTIONAL DNN
Revised:  P OWER CONSUMPTION OF A CONVENTIONAL DNN

Original: The total amount of power consumed by a logic circuit can be attributed to two main sources: a static power component and a dynamic one.
Revised: The total amount of power consumed by a logic circuit can be attributed to two main sources: a static power component and a dynamic one.

Original: The static power is due to a constant leakage current.
Revised: The static power is due to a constant leakage current.

Original: It does not depend on the circuit’s activity and is typically the smaller component among the two.
Revised: It does not depend on the circuit’s activity and is typically the smaller component among the two.

Original: The dynamic power consumed by each node in the circuit is given by P = CV 2 fα , where C is the node capacitance, V is the supply voltage, f is the operating frequency, and α is the switching activity factor (the average number of bit ﬂips per clock)
Revised: The dynamic power consumed by each node in the circuit is given by P = CV 2 fα , where C is the node capacitance, V is the supply voltage, f is the operating frequency, and α is the switching activity factor (the average number of bit ﬂips per clock)

Original: (Nasser et al., 2017).
Revised: (Nasser et al., 2017).

Original: Here we focus on dynamic power, which is  the dominant factor (Butts & Sohi, 2000).
Revised: Here we focus on dynamic power, which is a major contributor to the overall power consumption (see Appendix A.1 and (Karimi et al., 2019; Kim et al., 2020)). Also, this is the only factor affected by the DNN architecture.

Original: Most of the computation in a forward pass of a DNN can be attributed to MAC operations.
Revised: Most of the computation in a forward pass of a DNN can be attributed to MAC operations.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, MACs involve a multiplier that accepts two b -bit numbers and outputs a b acc -bit result ( b acc = 2 b to account for the largest possible product), and an accumulator with a large bit width B to which the multiplier’s output is added repeatedly.
Revised: 2, MACs involve a multiplier that accepts two b -bit numbers and outputs a b acc -bit result ( b acc = 2 b to account for the largest possible product), and an accumulator with a large bit width B to which the multiplier’s output is added repeatedly.

Original: To understand how much power each of these components consumes, we simulated them in Python.
Revised: To understand how much power each of these components consumes, we simulated them in Python.

Original: For the multiplier, we used the Booth-encoding architecture, which is considered efﬁcient in terms of bit toggling (Asif & Kong, 2015).
Revised: For the multiplier, we used the Booth-encoding architecture, which is considered efﬁcient in terms of bit toggling (Asif & Kong, 2015).

Original: For the accumulator, we simulated a serial adder.
Revised: For the accumulator, we simulated a serial adder.

Original: Our Python simulation allows measuring the total number of bit ﬂips in each MAC operation, including at the inputs, at the outputs, in the ﬂip-ﬂop (FF) register holding the previous sum, and within each of the internal components ( e.g. , the full-adders) of the multiplier.
Revised: Our Python simulation allows measuring the total number of bit ﬂips in each MAC operation, including at the inputs, at the outputs, in the ﬂip-ﬂop (FF) register holding the previous sum, and within each of the internal components ( e.g. , the full-adders) of the multiplier.

Original: We also veriﬁed our analysis with an accurate physical gate-level simulation on a 5 nm process and found good agreement in terms of the dependence of power consumption on the bit widths (see details in Appendix A.1).
Revised: We also veriﬁed our analysis with an accurate physical gate-level simulation on a 5 nm process and found good agreement in terms of the dependence of power consumption on the bit widths (see details in Appendix A.1).

Original: Table 1 shows the average number of bit ﬂips per MAC when both inputs to the multiplier are drawn uniformly at random from [ − 2 b − 1 , 2 b − 1 ) (Gaussian inputs lead to similar results; please see
Revised: Table 1 shows the average number of bit ﬂips per MAC when both inputs to the multiplier are drawn uniformly at random from [ − 2 b − 1 , 2 b − 1 ) (Gaussian inputs lead to similar results; please see

Original: Appendix Figs 7-8).
Revised: Appendix Figs 7-8).

Original: As can be seen, the power consumed by the multiplier is given by
Revised: As can be seen, the power consumed by the multiplier is given by

Original: [Equation]
Revised: [Equation]

Original: ) where 0 .
Revised: ) where 0 .

Original: 5 b 2 is due to the bit toggling in the internal units, and 0 .
Revised: 5 b 2 is due to the bit toggling in the internal units, and 0 .

Original: 5 b is contributed by the bit ﬂips in each input.
Revised: 5 b is contributed by the bit ﬂips in each input.

Original: The power consumed by the accumulator is given by
Revised: The power consumed by the accumulator is given by

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where 0 .
Revised: ) where 0 .

Original: 5 B is due to the bit toggling in its input coming from the multiplier, 0 .
Revised: 5 B is due to the bit toggling in its input coming from the multiplier, 0 .

Original: 5 b acc =
Revised: 5 b acc =

Original: b (recall b acc = 2 b ) to the bit ﬂips at the output, and an additional 0 .
Revised: b (recall b acc = 2 b ) to the bit ﬂips at the output, and an additional 0 .

Original: 5 b acc =
Revised: 5 b acc =

Original: b to the bit ﬂips in the FF.
Revised: b to the bit ﬂips in the FF.

Original: These results lead us to our ﬁrst important observation.
Revised: These results lead us to our ﬁrst important observation.

Original: Observation 1.
Revised: Observation 1.

Original: A dominant source of power consumption is the bit toggling at the input of the accumulator ( 0 .
Revised: A dominant source of power consumption is the bit toggling at the input of the accumulator ( 0 .

Original: 5 B ).
Revised: 5 B ).

Original: Suppose, for example, we use b = 4 bits for representing the weights and activations and employ a
Revised: Suppose, for example, we use b = 4 bits for representing the weights and activations and employ a

Original: B = 32 bit accumulator, as common in modern architectures (Kalamkar et al., 2019; Rodriguez et al., 2018).
Revised: B = 32 bit accumulator, as common in modern architectures (Kalamkar et al., 2019; Rodriguez et al., 2018).

Original: Then the toggling at the input of the accumulator ( 0 .
Revised: Then the toggling at the input of the accumulator ( 0 .

Original: 5 B = 16 ) is responsible for 44 .
Revised: 5 B = 16 ) is responsible for 44 .

Original: 4% of the total power consumption ( P mult + P acc = 36 ).
Revised: 4% of the total power consumption ( P mult + P acc = 36 ).

Original: At lower bit widths, this percentage is even larger.
Revised: At lower bit widths, this percentage is even larger.

Original: Unfortunately, existing quantization methods and multiplier-free designs do not battle this source of power consumption.
Revised: Unfortunately, existing quantization methods and multiplier-free designs do not battle this source of power consumption.

Original: Ni et al.
Revised: Ni et al.

Original: (2021) have recently shown that the bit-width B of the accumulator can be somewhat reduced by explicitly accounting for overﬂows.
Revised: (2021) have recently shown that the bit-width B of the accumulator can be somewhat reduced by explicitly accounting for overﬂows.

Original: However, this approach requires dedicated training, and degrades the network’s classiﬁcation accuracy at low values of B .
Revised: However, this approach requires dedicated training, and degrades the network’s classiﬁcation accuracy at low values of B .

Original: As we now show, it is possible to drastically reduce the bit toggles at the input of the accumulator at post-training without changing the model’s functionality (thus retaining the same classiﬁcation accuracy).
Revised: As we now show, it is possible to drastically reduce the bit toggles at the input of the accumulator at post-training without changing the model’s functionality (thus retaining the same classiﬁcation accuracy).

Original:  S WITCHING TO UNSIGNED ARITHMETIC
Revised:  S WITCHING TO UNSIGNED ARITHMETIC

Original: Since the output of the multiplier has only 2 b bits, one could expect to experience no more than b bit ﬂips on average at the accumulator’s input.
Revised: Since the output of the multiplier has only 2 b bits, one could expect to experience no more than b bit ﬂips on average at the accumulator’s input.

Original: Why do we have 0 .
Revised: Why do we have 0 .

Original: 5 B bit ﬂips instead?
Revised: 5 B bit ﬂips instead?

Original: The reason is rooted in the use of signed arithmetic.
Revised: The reason is rooted in the use of signed arithmetic.

Original: Speciﬁcally, negative numbers are represented using two’s complement, and thus switching between positive and negative numbers results in ﬂipping of many of the higher bits.
Revised: Speciﬁcally, negative numbers are represented using two’s complement, and thus switching between positive and negative numbers results in ﬂipping of many of the higher bits.

Original: For example, when using a 32 bit accumulator, if the output of the multiplier switches from +2 to − 2 , then the bits at the input of the accumulator switch from 00000000000000000000000000000010 to 11111111111111111111111111111110 .
Revised: For example, when using a 32 bit accumulator, if the output of the multiplier switches from +2 to − 2 , then the bits at the input of the accumulator switch from 00000000000000000000000000000010 to 11111111111111111111111111111110 .

Original: Note that this effect is dominant only at the accumulator’s input simply because sign changes at the output are rare.
Revised: Note that this effect is dominant only at the accumulator’s input simply because sign changes at the output are rare.

Original: If we could work with unsigned integers, then the higher bits at the accumulator’s input would always remain zero, which would lead to a substantial reduction in power consumption without any performance degradation.
Revised: If we could work with unsigned integers, then the higher bits at the accumulator’s input would always remain zero, which would lead to a substantial reduction in power consumption without any performance degradation.

Original: To quantify this, we repeated the experiment of Sec. 3, but with the b -bit inputs to the multiplier now drawn uniformly from [0 , 2 b − 1 ) (see Appendix A.2 for details).
Revised: To quantify this, we repeated the experiment of Sec. 3, but with the b -bit inputs to the multiplier now drawn uniformly from [0 , 2 b − 1 ) (see Appendix A.2 for details).

Original: In this case, the average number of bit ﬂips at the input of the accumulator reduced from 0 .
Revised: In this case, the average number of bit ﬂips at the input of the accumulator reduced from 0 .

Original: 5 B to 0 .
Revised: 5 B to 0 .

Original: 5 b acc = b .
Revised: 5 b acc = b .

Original: Speciﬁcally, the average power consumption of an unsigned MAC operation was measured to be
Revised: Speciﬁcally, the average power consumption of an unsigned MAC operation was measured to be

Original: [Equation]
Revised: [Equation]

Original: due to the multiplier and
Revised: due to the multiplier and

Original: [Equation]
Revised: [Equation]

Original: due to the accumulator.
Revised: due to the accumulator.

Original: In (4), 2 b bit ﬂips occur at the accumulator’s output and the FF, and b bit ﬂips occur at the accumulator’s input coming from the multiplier.
Revised: In (4), 2 b bit ﬂips occur at the accumulator’s output and the FF, and b bit ﬂips occur at the accumulator’s input coming from the multiplier.

Original: Thus, although the mutliplier’s power (3) turns out to be the same as in the signed setting (1), the accumulator’s power (4) is substantially reduced w.r.t.
Revised: Thus, although the mutliplier’s power (3) turns out to be the same as in the signed setting (1), the accumulator’s power (4) is substantially reduced w.r.t.

Original: the signed case (2).
Revised: the signed case (2).

Original: Converting a pre-trained network with ReLU activation functions to work with unsigned integers is simple.
Revised: Converting a pre-trained network with ReLU activation functions to work with unsigned integers is simple.

Original: Speciﬁcally, consider a layer performing y =
Revised: Speciﬁcally, consider a layer performing y =

Original: Wx + b .
Revised: Wx + b .

Original: The elements of x are non-negative because of the preceding ReLU 2 .
Revised: The elements of x are non-negative because of the preceding ReLU 2 .

Original: Therefore, we can split any such layer into two parallel layers as
Revised: Therefore, we can split any such layer into two parallel layers as

Original: [Equation]
Revised: [Equation]

Original: where W + = ReLU ( W ) , b + = ReLU ( b ) , W − =
Revised: where W + = ReLU ( W ) , b + = ReLU ( b ) , W − =

Original: ReLU ( − W ) , b − =
Revised: ReLU ( − W ) , b − =

Original: ReLU ( − b ) , and compute y =
Revised: ReLU ( − b ) , and compute y =

Original: y + − y − .
Revised: y + − y − .

Original: This way, all MACs are converted to unsigned ones in (5), and only a single subtraction per output element is needed in (6).
Revised: This way, all MACs are converted to unsigned ones in (5), and only a single subtraction per output element is needed in (6).

Original: This one subtraction is negligible w.r.t.
Revised: This one subtraction is negligible w.r.t.

Original: the MACs, whose number is usually in the thousands.
Revised: the MACs, whose number is usually in the thousands.

Original: Please see Fig. 11(b) in the Appendix for a schematic illustration.
Revised: Please see Fig. 11(b) in the Appendix for a schematic illustration.

Original: Figure 1 shows the effect that this approach has on the power consumption of several pretrained networks for ImageNet classiﬁcation.
Revised: Figure 1 shows the effect that this approach has on the power consumption of several pretrained networks for ImageNet classiﬁcation.

Original: Figures 1(a) and 1(b) show 4 bit and 2 bit quantized networks, respectively.
Revised: Figures 1(a) and 1(b) show 4 bit and 2 bit quantized networks, respectively.

Original: With a 32 bit accumulator, merely switching to unsigned arithmetic cuts 33% and 58% of the power consumption of these networks.
Revised: With a 32 bit accumulator, merely switching to unsigned arithmetic cuts 33% and 58% of the power consumption of these networks (see Appendix A.3.1 for other accumulator bit widths).

Original:  R EMOVING THE MULTIPLIER
Revised:  R EMOVING THE MULTIPLIER

Original: Having reduced the power consumed by the accumulator, we now turn to treat the multiplier.
Revised: Having reduced the power consumed by the accumulator, we now turn to treat the multiplier.

Original: Quantization methods often use different bit widths for the weights and activations.
Revised: Quantization methods often use different bit widths for the weights and activations.

Original: This ﬂexibility allows achieving good classiﬁcation accuracy with quite aggressive quantization of one of them (typically the weights), but a ﬁner quantization of the other.
Revised: This ﬂexibility allows achieving good classiﬁcation accuracy with quite aggressive quantization of one of them (typically the weights), but a ﬁner quantization of the other.

Original: An interesting question is whether this approach is beneﬁcial in terms of power consumption.
Revised: An interesting question is whether this approach is beneﬁcial in terms of power consumption.

Original: We repeated the experiment of Sec. 3, this time with the multiplier inputs having different bit widths, b w and b x .
Revised: We repeated the experiment of Sec. 3, this time with the multiplier inputs having different bit widths, b w and b x .

Original: We focused on the standard setting of signed numbers, which we drew uniformly from [ − 2 b w , 2 b w − 1 ) and [ − 2 b x , 2 b x − 1 ) .
Revised: We focused on the standard setting of signed numbers, which we drew uniformly from [ − 2 b w , 2 b w − 1 ) and [ − 2 b x , 2 b x − 1 ) .

Original: Interestingly, we found that the average number of bit ﬂips in the multiplier’s internal units is affected only by the larger among b w and b x .
Revised: Interestingly, we found that the average number of bit ﬂips in the multiplier’s internal units is affected only by the larger among b w and b x .

Original: Accounting also for the bit ﬂips at the inputs, we obtained that the multiplier’s total average power is
Revised: Accounting also for the bit ﬂips at the inputs, we obtained that the multiplier’s total average power is

Original: [Equation]
Revised: [Equation]

Original: We found this surprising behavior to be characteristic of both the Booth multiplier and the simple serial multiplier, and veriﬁed it also with accurate simulations on a 5 nm silicon process gate level synthesis (see Appendix Figs. 9,10).
Revised: We found this surprising behavior to be characteristic of both the Booth multiplier and the simple serial multiplier, and veriﬁed it also with accurate simulations on a 5 nm silicon process gate level synthesis (see Appendix Figs. 9,10).

Original: This leads us to our second important observation.
Revised: This leads us to our second important observation.

Original: Observation 2.
Revised: Observation 2.

Original: There is marginal beneﬁt in the common practice of decreasing the bit width of only the weights or only the activations, at least in terms of the power consumed by the multiplier.
Revised: There is marginal beneﬁt in the common practice of decreasing the bit width of only the weights or only the activations, at least in terms of the power consumed by the multiplier.

Original: It should be noted that in the case of unsigned numbers, where inputs are drawn uniformly from [0 , 2 b w − 1 ) and [0 , 2 b x − 1 ) , there exists some power save when reducing one of the bit widths, especially for the serial multiplier (see Appendix Fig. 10).
Revised: It should be noted that in the case of unsigned numbers, where inputs are drawn uniformly from [0 , 2 b w − 1 ) and [0 , 2 b x − 1 ) , there exists some power save when reducing one of the bit widths, especially for the serial multiplier (see Appendix Fig. 10).

Original: This highlights again the importance of unsigned arithmetic.
Revised: This highlights again the importance of unsigned arithmetic.

Original: However, in all our experiments we do not take this extra beneﬁt of our approach into account when computing power consumption, so that our reports are conservative.
Revised: However, in all our experiments we do not take this extra beneﬁt of our approach into account when computing power consumption, so that our reports are conservative.

Original: To beneﬁt from the ability to achieve high precision with drastic quantization of only the weights, we now explore a solution that allows removing the multiplier altogether.
Revised: To beneﬁt from the ability to achieve high precision with drastic quantization of only the weights, we now explore a solution that allows removing the multiplier altogether.

Original: Unlike other multiplier-free designs, our method allows converting any full-precision pre-trained model into a low-precision power-efﬁcient one without any changes to the architecture.
Revised: Unlike other multiplier-free designs, our method allows converting any full-precision pre-trained model into a low-precision power-efﬁcient one without any changes to the architecture.

Original: 5. P OWER AWARE WEIGHT QUANTIZATION
Revised: 5. P OWER AWARE WEIGHT QUANTIZATION

Original: Consider the computation
Revised: Consider the computation

Original: [Equation]
Revised: [Equation]

Original: which involves d MACs.
Revised: which involves d MACs.

Original: Here, { w i , x i } are the weights and activations of a convolution or a fullyconnected layer.
Revised: Here, { w i , x i } are the weights and activations of a convolution or a fullyconnected layer.

Original: Given { w i , x i } in full precision, our goal is to accurately approximate (8) in a power-efﬁcient manner.
Revised: Given { w i , x i } in full precision, our goal is to accurately approximate (8) in a power-efﬁcient manner.

Original: When quantizing the weights and activations we obtain the approximation
Revised: When quantizing the weights and activations we obtain the approximation

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the quantizers Q w ( · ) and Q x ( · ) map R to Z , and γ w and γ x are their quantization steps 3 .
Revised: where the quantizers Q w ( · ) and Q x ( · ) map R to Z , and γ w and γ x are their quantization steps 3 .

Original: To make the computation (9) power efﬁcient, we propose to implement multiplications via additions .
Revised: To make the computation (9) power efﬁcient, we propose to implement multiplications via additions .

Original: [Figure]
Revised: [Figure]

Original: Speciﬁcally, assume Q w ( w i ) is a non-negative integer (as in Sec. 4).
Revised: Speciﬁcally, assume Q w ( w i ) is a non-negative integer (as in Sec. 4).

Original: Then we can implement the term Q w ( w i ) · Q x ( x i ) as
Revised: Then we can implement the term Q w ( w i ) · Q x ( x i ) as

Original: [Equation]
Revised: [Equation]

Original: so that (9) is computed as
Revised: so that (9) is computed as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This is the basis for our power-aware neural network (PANN) design.
Revised: This is the basis for our power-aware neural network (PANN) design.

Original: Let w
Revised: Let w

Original: =
Revised: =

Original: w 1 , . . . , w d ) T and x = ( x 1 , . . .
Revised: w 1 , . . . , w d ) T and x = ( x 1 , . . .

Original: , x d ) T denote the full precision weights and activations, and denote their quantized versions by w q = ( Q w ( w 1 ) , . . .
Revised: , x d ) T denote the full precision weights and activations, and denote their quantized versions by w q = ( Q w ( w 1 ) , . . .

Original: , Q w ( w d )) T and x q = ( Q x ( x 1 ) , . . .
Revised: , Q w ( w d )) T and x q = ( Q x ( x 1 ) , . . .

Original: , Q x ( x d )) T , respectively.
Revised: , Q x ( x d )) T , respectively.

Original: Note that as opposed to conventional quantization methods, our approach does not necessitate that the quantized weights be conﬁned to any particular range of the form [0 , 2 b w ) .
Revised: Note that as opposed to conventional quantization methods, our approach does not necessitate that the quantized weights be conﬁned to any particular range of the form [0 , 2 b w ) .

Original: Indeed, what controls our approximation accuracy is not the largest possible entry in w q , but rather the number of additions per input element, which is given by (cid:107) w q (cid:107) 1 /d .
Revised: Indeed, what controls our approximation accuracy is not the largest possible entry in w q , but rather the number of additions per input element, which is given by (cid:107) w q (cid:107) 1 /d .

Original: Therefore, given a budget of R additions per input element, we propose to use a quantization step of γ w = (cid:107) w (cid:107) 1 / ( Rd ) in (9), so that
Revised: Therefore, given a budget of R additions per input element, we propose to use a quantization step of γ w = (cid:107) w (cid:107) 1 / ( Rd ) in (9), so that

Original: [Equation]
Revised: [Equation]

Original: This quantization ensures that the number of additions per input element is indeed as close as possible to the prescribed R .
Revised: This quantization ensures that the number of additions per input element is indeed as close as possible to the prescribed R .

Original: We remark that although we assumed unsigned weights, this quantization procedure can also be used for signed weights (after quantization, the positive and negative weights can be treated separately in order to save power, as in Sec. 4).
Revised: We remark that although we assumed unsigned weights, this quantization procedure can also be used for signed weights (after quantization, the positive and negative weights can be treated separately in order to save power, as in Sec. 4).

Original: 5. P OWER CONSUMPTION
Revised: 5. P OWER CONSUMPTION

Original: We emphasize that in PANN, we would not necessarily want to use the same bit width for the activations as in regular quantization.
Revised: We emphasize that in PANN, we would not necessarily want to use the same bit width for the activations as in regular quantization.

Original: We therefore denote the activation bit width in PANN by ˜
Revised: We therefore denote the activation bit width in PANN by ˜

Original: b x to distuingish it from the b x bits we would use with the regular quantizer.
Revised: b x to distuingish it from the b x bits we would use with the regular quantizer.

Original: To estimate the power consumed by our approach, note that we have approximately (cid:107) w (cid:107) 1 additions of ˜ b x bit numbers.
Revised: To estimate the power consumed by our approach, note that we have approximately (cid:107) w (cid:107) 1 additions of ˜ b x bit numbers.

Original: On average, each such addition leads to 0 .
Revised: On average, each such addition leads to 0 .

Original: 5˜ b x bit ﬂips at the accumulator’s output and 0 .
Revised: 5˜ b x bit ﬂips at the accumulator’s output and 0 .

Original: 5˜ b x bit ﬂips in the FF register (see Table 1).
Revised: 5˜ b x bit ﬂips in the FF register (see Table 1).

Original: The input to the accumulator, however, remains ﬁxed for Q w ( w i ) times when approximating the i th MAC and therefore changes a total of only d times throughout the entire computation in (11), each time with 0 .
Revised: The input to the accumulator, however, remains ﬁxed for Q w ( w i ) times when approximating the i th MAC and therefore changes a total of only d times throughout the entire computation in (11), each time with 0 .

Original: 5˜ b x bit ﬂips on average.
Revised: 5˜ b x bit ﬂips on average.

Original: Thus, overall, the average power per element consumed by PANN is
Revised: Thus, overall, the average power per element consumed by PANN is

Original: [Equation]
Revised: [Equation]

Original: This implies that to comply with a prescribed power budget, we can either increase the activation bit width ˜
Revised: This implies that to comply with a prescribed power budget, we can either increase the activation bit width ˜

Original: b x on the expense of the number of additions R , or vice versa.
Revised: b x on the expense of the number of additions R , or vice versa.

Original: Figure 3(a) depicts the combinations of ˜ b x and R that lead to the same power consumption as that of a b x bit unsigned MAC, P uMAC = 0 .
Revised: Figure 3(a) depicts the combinations of ˜ b x and R that lead to the same power consumption as that of a b x bit unsigned MAC, P uMAC = 0 .

Original: 5 b 2 x + 4 b x (see (3),(4)) for several values of b x (different colors).
Revised: 5 b 2 x + 4 b x (see (3),(4)), for several values of b x (different colors).

Original: When we traverse such an equal-power curve, we also change the quantization error.
Revised: When we traverse such an equal-power curve, we also change the quantization error.

Original: Thus, the question is whether there exist points along each curve, which lead to lower errors than those obtained with regular quantization at the bit-width corresponding to that curve.
Revised: Thus, the question is whether there exist points along each curve, which lead to lower errors than those obtained with regular quantization at the bit-width corresponding to that curve.

Original: 5. Q UANTIZATION ERROR
Revised: 5. Q UANTIZATION ERROR

Original: Let us compute the quantization error incurred by PANN and compare it to that of a regular uniform quantizer (RUQ).
Revised: Let us compute the quantization error incurred by PANN and compare it to that of a regular uniform quantizer (RUQ).

Original: Obviously, the error incurred by the approximation (9) is contributed by both the quantization of the weights and the quantization of the activations.
Revised: Obviously, the error incurred by the approximation (9) is contributed by both the quantization of the weights and the quantization of the activations.

Original: To see how their errors interact, let us assume that w and x are statistically independent random vectors, each with iid components.
Revised: To see how their errors interact, let us assume that w and x are statistically independent random vectors, each with iid components.

Original: In this case, if the quantization errors ε w = w − γ w Q ( w ) and ε x = x − γ x Q ( x ) satisfy E [ ε w | w ] =and E [ ε x | x ] = 0 , then we can show (see Appendix A.8) that the mean squared error (MSE) between the full precision operation (8) and its quantized version (9) is given by
Revised: In this case, if the quantization errors ε w = w − γ w Q ( w ) and ε x = x − γ x Q ( x ) satisfy E [ ε w | w ] =and E [ ε x | x ] = 0 , then we can show (see Appendix A.9) that the mean squared error (MSE) between the full precision operation (8) and its quantized version (9) is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here σ 2 w , σ 2 x , σ 2 ε w , and σ 2 ε x denote the second-order moments of the elements of w , x , ε w , and ε x , respectively, and the expression on the right results from neglecting second-order terms.
Revised: Here σ 2 w , σ 2 x , σ 2 ε w , and σ 2 ε x denote the second-order moments of the elements of w , x , ε w , and ε x , respectively, and the expression on the right results from neglecting second-order terms.

Original: Consider a simplistic scenario where the activations are uniformly distributed in [0 , M a ] (recall that activations are non-negative due to the preceding ReLU) and the weights are uniformly distributed in [ − 12 M w , 12 M w ] .
Revised: Consider a simplistic scenario where the activations are uniformly distributed in [0 , M a ] (recall that activations are non-negative due to the preceding ReLU) and the weights are uniformly distributed in [ − 12 M w , 12 M w ] .

Original: If we use a RUQ with b x bits to quantize the activations and a RUQ with b w bits to quantize the weights, then we have that
Revised: If we use a RUQ with b x bits to quantize the activations and a RUQ with b w bits to quantize the weights, then we have that

Original: [Equation]
Revised: [Equation]

Original: This is because the quantization errors are uniformly distributed as ε x ∼ U [ − 2 − ( b x − 1) , 2 − ( b x − 1) ] and ε w ∼ U
Revised: This is because the quantization errors are uniformly distributed as ε x ∼ U [ − 2 − ( b x − 1) , 2 − ( b x − 1) ] and ε w ∼ U

Original: [ − 2 − ( b w − 1) , 2 − ( b w − 1) ] .
Revised: [ − 2 − ( b w − 1) , 2 − ( b w − 1) ] .

Original: Substituting (15) into (14), we obtain that the MSE of a RUQ is
Revised: Substituting (15) into (14), we obtain that the MSE of a RUQ is

Original: [Equation]
Revised: [Equation]

Original: In PANN, we have that ( ε w ) i | w ∼ U [ − (cid:107)
Revised: In PANN, we have that ( ε w ) i | w ∼ U [ − (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 1 2 Rd , (cid:107)
Revised: (cid:107) 1 2 Rd , (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Rd ] , so that E [( ε w ) 2 i | w ] = (cid:107) w (cid:107) 21 12 Rd .
Revised: Rd ] , so that E [( ε w ) 2 i | w ] = (cid:107) w (cid:107) 21 12 Rd .

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where we used the fact that | w i | ∼ U [0 , 0 .
Revised: where we used the fact that | w i | ∼ U [0 , 0 .

Original: 5 M w ] .
Revised: 5 M w ] .

Original: Substituting this expression in (14), we ﬁnd that using PANN together with a ˜ b x bit RUQ for the activations, we achieve
Revised: Substituting this expression in (14), we ﬁnd that using PANN together with a ˜ b x bit RUQ for the activations, we achieve

Original: [Equation]
Revised: [Equation]

Original: To compare between PANN and RUQ, we need to ﬁx a power budget P .
Revised: To compare between PANN and RUQ, we need to ﬁx a power budget P .

Original: Given such a budget, (13) dictates that the number of additions in PANN should be set to R = P/ ˜
Revised: Given such a budget, (13) dictates that the number of additions in PANN should be set to R = P/ ˜

Original: b x − 0 .
Revised: b x − 0 .

Original: Substituting this into (19), we obtain that
Revised: Substituting this into (19), we obtain that

Original: [Equation]
Revised: [Equation]

Original: The optimal bit-width for the activations can therefore be found numerically by minimizing MSE PANN over ˜ b x ∈ Z + .
Revised: The optimal bit-width for the activations can therefore be found numerically by minimizing MSE PANN over ˜ b x ∈ Z + .

Original: This typically requires evaluating (19) for a small number of candidate bit widths, e.g. ,
Revised: This typically requires evaluating (19) for a small number of candidate bit widths, e.g. ,

Original: ˜ b x ∈ { 2 , . .
Revised: ˜ b x ∈ { 2 , . .

Original: See Appendix A.7 for a thorough analysis.
Revised: See Appendix A.8 for a thorough analysis.

Original: Figure 3(b) shows the ratio between MSE RUQ and MSE PANN (with the optimal ˜ b x ) as a function of the bit width of the RUQ, where PANN is tuned to the same power.
Revised: Figure 3(b) shows the ratio between MSE RUQ and MSE PANN (with the optimal ˜ b x ) as a function of the bit width of the RUQ, where PANN is tuned to the same power.

Original: For the RUQ, we use b x = b w as its power consumption (7) is anyway dominated by the larger of them.
Revised: For the RUQ, we use b x = b w as its power consumption (7) is anyway dominated by the larger of them.

Original: It can be seen that for low bit widths, PANN has a signiﬁcant advantage over RUQ (ratio larger than 1 ).
Revised: It can be seen that for low bit widths, PANN has a signiﬁcant advantage over RUQ (ratio larger than 1 ).

Original: In the Gaussian setting, which is closer to the distribution of DNN weights and activations, the range over which
Revised: In the Gaussian setting, which is closer to the distribution of DNN weights and activations, the range over which

Original: PANN outperforms RUQ is even larger.
Revised: PANN outperforms RUQ is even larger.

Original: As we show in Appendix A.7, this behavior is very similar to that observed in deep networks for image classiﬁcation.
Revised: As we show in Appendix A.8, this behavior is very similar to that observed in deep networks for image classiﬁcation.

Original: We emphasize that (19) is valid for uniformly distributed weights and activations, which is often not an accurate enough assumption for DNNs.
Revised: We emphasize that (19) is valid for uniformly distributed weights and activations, which is often not an accurate enough assumption for DNNs.

Original: Thus, in practice the best way to determine the optimal bit width ˜
Revised: Thus, in practice the best way to determine the optimal bit width ˜

Original: b x is by running the quantized network on a validation set, as summarized in Algorithm 1.
Revised: b x is by running the quantized network on a validation set, as summarized in Algorithm 1.

Original: for
Revised: for

Original: 10: set ˜ b x ← arg max ˜ b
Revised: 10: set ˜ b x ← arg max ˜ b

Original: x Acc (˜ b x ) , R ← P/ ˜ b x − 0 .
Revised: x Acc (˜ b x ) , R ← P/ ˜ b x − 0 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now examine PANN in DNN classiﬁcation experiments.
Revised: We now examine PANN in DNN classiﬁcation experiments.

Original: We start by examining its performance at post training, and then move on to employ it during training.
Revised: We start by examining its performance at post training, and then move on to employ it during training.

Original: Here we focus only on the effect of removing the multiplier (vertical arrows in Fig. 1).
Revised: Here we focus only on the effect of removing the multiplier (vertical arrows in Fig. 1).

Original: Namely, we assume all models have already been converted to unsigned arithmetic (recall this by itself reduces a lot of the power consumption).
Revised: Namely, we assume all models have already been converted to unsigned arithmetic (recall this by itself reduces a lot of the power consumption).

Original: PANN at post training We illustrate PANN’s performance in conjunction with a variety of post training quantization methods, including the data free approaches GDFQ (Shoukai et al., 2020) and ZeroQ (Cai et al., 2020), the small calibration set method ACIQ (Banner et al., 2019), and the optimization based approach BRECQ
Revised: PANN at post training We illustrate PANN’s performance in conjunction with a variety of post training quantization methods, including the data free approaches GDFQ (Shoukai et al., 2020) and ZeroQ (Cai et al., 2020), the small calibration set method ACIQ (Banner et al., 2019), and the optimization based approach BRECQ

Original: (Li et al., 2021), which is currently the state-of-the-art for post training quantization at low bit widths.
Revised: (Li et al., 2021), which is currently the state-of-the-art for post training quantization at low bit widths.

Original: Table 2 reports results with ResNet-50 on ImageNet (See
Revised: Table 2 reports results with ResNet-50 on ImageNet (See

Original: Appendix A.5.1 for results with other models).
Revised: Appendix A.5.1 for results with other models).

Original: For the baseline methods, we always use equal bit widths for the weights and activations.
Revised: For the baseline methods, we always use equal bit widths for the weights and activations.

Original: Each row also shows our PANN variant, which works at the precise same power budget, where we choose the optimal ˜ b x and R using Alg. 1.
Revised: Each row also shows our PANN variant, which works at the precise same power budget, where we choose the optimal ˜ b x and R using Alg. 1.

Original: As can be seen,
Revised: As can be seen,

Original: PANN exhibits only a minor degradation w.r.t.
Revised: PANN exhibits only a minor degradation w.r.t.

Original: the full-precision model, even when working at the power budget of 2 bit networks.
Revised: the full-precision model, even when working at the power budget of 2 bit networks.

Original: This is while all existing methods completely fail in this regime.
Revised: This is while all existing methods completely fail in this regime.

Original: PANN for quantization aware training To use PANN during training, we employ a straightthrough estimator for backpropagation through the quantizers.
Revised: PANN for quantization aware training To use PANN during training, we employ a straightthrough estimator for backpropagation through the quantizers.

Original: Table 3 compares our method to
Revised: Table 3 compares our method to

Original: LSQ (Esser et al., 2019), which is a state-of-the-art QAT approach, where in PANN we use LSQ for quantizing the activations.
Revised: LSQ (Esser et al., 2019), which is a state-of-the-art QAT approach, where in PANN we use LSQ for quantizing the activations.

Original: As can be seen, PANN outperforms LSQ for various models and power budgets.
Revised: As can be seen, PANN outperforms LSQ for various models and power budgets.

Original: In Table 4 we compare our method to the multiplication-free approaches AdderNet (Chen et al., 2020) and ShiftAddNet (You et al., 2020), which are also training-based techniques.
Revised: In Table 4 we compare our method to the multiplication-free approaches AdderNet (Chen et al., 2020) and ShiftAddNet (You et al., 2020), which are also training-based techniques.

Original: For each method, we report the addition factor , which is the ratio between its number of additions per layer and a regular layer.
Revised: For each method, we report the addition factor , which is the ratio between its number of additions per layer and a regular layer.

Original: For example, AdderNet uses no multiplications but twice as many additions, so that its addition factor is 2 .
Revised: For example, AdderNet uses no multiplications but twice as many additions, so that its addition factor is 2 .

Original: ShiftAddNet, on the other hand, uses one addition and one shift operation.
Revised: ShiftAddNet, on the other hand, uses one addition and one shift operation.

Original: According to You et al.
Revised: According to You et al.

Original: (2020), a shift operation costs between 0 .
Revised: (2020), a shift operation costs between 0 .

Original: 2 (on FPGA) and 0 .
Revised: 2 (on FPGA) and 0 .

Original: 8 (on a 45 nm
Revised: 8 (on a 45 nm

Original: ASIC) an addition operation.
Revised: ASIC) an addition operation.

Original: Therefore ShiftAddNet’s addition factor is between 1 . 2 and 1 . 8 , and for simplicity we regard it as 1 . 5 .
Revised: Therefore ShiftAddNet’s addition factor is between 1 . 2 and 1 . 8 , and for simplicity we regard it as 1 . 5 .

Original: In PANN, we can choose any addition factor R , and therefore examine
Revised: In PANN, we can choose any addition factor R , and therefore examine

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: [ % ] of models.

Revised: We

Original: [Table]
Revised: [Table]

Original: our method for R = 1 , 1 . 5 , 2 .
Revised: our method for R = 1 , 1 . 5 , 2 .

Original: We can see in the table that PANN outperforms both AdderNet and
Revised: We can see in the table that PANN outperforms both AdderNet and

Original: ShiftAddNet for all bit widths, even when using a smaller addition factor.
Revised: ShiftAddNet for all bit widths, even when using a smaller addition factor.

Original: Please see more QAT comparisons in Appendix A.5.
Revised: Please see more QAT comparisons in Appendix A.5.

Original: Runtime memory footprint and latency of PANN We now analyze the effect of PANN on other inference aspects besides power.
Revised: Runtime memory footprint and latency of PANN We now analyze the effect of PANN on other inference aspects besides power.

Original: One important aspect is runtime memory footprint .
Revised: One important aspect is runtime memory footprint .

Original: When working with batches of image, the runtime memory consumption is dominated by the activations (Mishra et al., 2017) (see discussion on the memory footprint of the weights in Appendix A.6).
Revised: When working with batches of image, the runtime memory consumption is dominated by the activations (Mishra et al., 2017) (see discussion on the memory footprint of the weights in Appendix A.7).

Original: The optimal number of bits ˜ b
Revised: The optimal number of bits ˜ b

Original: x we use for the activations is typically larger than the bit
Revised: x we use for the activations is typically larger than the bit

Original: width b x used in regular quantization.
Revised: width b x used in regular quantization.

Original: The second column of Table 2 reports the factor ˜
Revised: The second column of Table 2 reports the factor ˜

Original: b x /b x by which the runtime memory of PANN exceeds that of the baseline model.
Revised: b x /b x by which the runtime memory of PANN exceeds that of the baseline model.

Original: As can be seen, this factor never exceeds 3.
Revised: As can be seen, this factor never exceeds 3.

Original: In the comparisons with the multiplier-free methods (Table 4), we keep the same bit width for the activations and therefore there is no change in the memory footprint.
Revised: In the comparisons with the multiplier-free methods (Table 4), we keep the same bit width for the activations and therefore there is no change in the memory footprint.

Original: A second important factor is latency .
Revised: A second important factor is latency .

Original: Recall we remove the multiplier and remain only with the accumulator.
Revised: Recall we remove the multiplier and remain only with the accumulator.

Original: Since addition is faster than multiplication, one could potentially use a higher clock-rate and thus gain speed.
Revised: Since addition is faster than multiplication, one could potentially use a higher clock-rate and thus gain speed.

Original: However, if we conservatively assume the original clock-rate, then the latency is increased by R (each multiplication is replaced by R additions).
Revised: However, if we conservatively assume the original clock-rate, then the latency is increased by R (each multiplication is replaced by R additions).

Original: As can be seen in Table 2, the increase in latency is quite small at the lower power budgets.
Revised: As can be seen in Table 2, the increase in latency is quite small at the lower power budgets.

Original: For the multiplier-free methods, we obtain improvement in accuracy even for
Revised: For the multiplier-free methods, we obtain improvement in accuracy even for

Original: R = 1 .
Revised: R = 1 .

Original: In that case, our latency is smaller than that of AdderNet ( 2 × ) and ShiftAddNet ( 1 . 5 × ).
Revised: In that case, our latency is smaller than that of AdderNet ( 2 × ) and ShiftAddNet ( 1 . 5 × ).

Original: Please refer to Appendix A.6 for more analyses.
Revised: Please refer to Appendix A.7 for more analyses.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented an approach for reducing the power consumption of DNNs.
Revised: We presented an approach for reducing the power consumption of DNNs.

Original: Our technique relies on a detailed analysis of the power consumption of each arithmetic module in the network, and makes use of two key principles: switching to unsigned arithemtic, and employing a new weight quantization method that allows removing the multiplier.
Revised: Our technique relies on a detailed analysis of the power consumption of each arithmetic module in the network, and makes use of two key principles: switching to unsigned arithemtic, and employing a new weight quantization method that allows removing the multiplier.

Original: Our method substantially improves over existing approaches, both at post-training and when used during training, and allows achieving a signiﬁcantly higher accuracy at any given power consumption budget.
Revised: Our method substantially improves over existing approaches, both at post-training and when used during training, and allows achieving a signiﬁcantly higher accuracy at any given power consumption budget.

Original: A A PPENDIX A. P OWER SIMULATION ON 5 NM PROCESS
Revised: A A PPENDIX A. P OWER SIMULATION ON 5 NM PROCESS

Original: Using the Synopsys DesignWare Library 4 , we built a Verilog RTL (Register Transfer Logic) simulation which instantiates signed multipliers and signed adders of 2 - 8 bit widths.
Revised: Using the Synopsys DesignWare Library 4 , we built a Verilog RTL (Register Transfer Logic) simulation which instantiates signed multipliers and signed adders of 2 - 8 bit widths.

Original: For the multipliers, we used a Radix-4 Booth-encoder implementation and for the adders, we used the Ripple Carry implementation.
Revised: For the multipliers, we used a Radix-4 Booth-encoder implementation and for the adders, we used the Ripple Carry implementation.

Original: We synthesized these adders and multipliers, with a 5 nm cells library at a clock frequency of 1 . 6 GHz .
Revised: We synthesized these adders and multipliers, with a 5 nm cells library at a clock frequency of 1 . 6 GHz .

Original: In order to analyze the power consumption of each module individually, we used a hierarchical gate-level synthesis where each module is a different utility that does not share logic with any other module.
Revised: In order to analyze the power consumption of each module individually, we used a hierarchical gate-level synthesis where each module is a different utility that does not share logic with any other module.

Original: The synthesis result is the gate-level netlist with the logic gates, which is the actual implementation of the multipliers and adders.
Revised: The synthesis result is the gate-level netlist with the logic gates, which is the actual implementation of the multipliers and adders.

Original: Then, using Synopsys PrimeTime PX 5 (PTPX), which accurately reﬂects ASIC power consumption, we ran a simulation with uniformly distributed random inputs on the gate-level netlist and measured the power of multiplication and addition instructions.
Revised: Then, using Synopsys PrimeTime PX 5 (PTPX), which accurately reﬂects ASIC power consumption, we ran a simulation with uniformly distributed random inputs on the gate-level netlist and measured the power of multiplication and addition instructions.

Original: It should be noted that, as opposed to papers experimenting with FPGAs, our primary interest is in modern integrated chips (ASICs), like CPUs and GPUs.
Revised: It should be noted that, as opposed to papers experimenting with FPGAs, our primary interest is in modern integrated chips (ASICs), like CPUs and GPUs.

Original: Gate-level simulations of the type we use here are the only practical (and most accurate) way to estimate ASIC power consumption.
Revised: Gate-level simulations of the type we use here are the only practical (and most accurate) way to estimate ASIC power consumption.

Original: This is because even if we had fabricated this netlist as part of a real ASIC (which would cost millions of dollars), it would still be impossible to measure the power of a small portion of the chip accurately.
Revised: This is because even if we had fabricated this netlist as part of a real ASIC (which would cost millions of dollars), it would still be impossible to measure the power of a small portion of the chip accurately.

Original: In ﬁgures 4(a) and 4(b) we depict the average power consumed by a multiplication of two b -bit numbers and by an addition of two b -bit numbers, respectively.
Revised: In ﬁgures 4(a) and 4(b) we depict the average power consumed by a multiplication of two b -bit numbers and by an addition of two b -bit numbers, respectively.

Original: We can see that these power measurements agree with our Python simulation, which we discuss in the next section.
Revised: We can see that these power measurements agree with our Python simulation, which we discuss in the next section.

Original: A remark is in place about the slight deviation between the 5 nm simulation and our Python simulation seen in Fig. 4(a).
Revised: A remark is in place about the slight deviation between the 5 nm simulation and our Python simulation seen in Fig. 4(a).

Original: This deviation implies that the advantage of PANN over a regular DNN is slightly more substantial than that reported in the paper.
Revised: This deviation implies that the advantage of PANN over a regular DNN is slightly more substantial than that reported in the paper.

Original: Speciﬁcally, since our theoretical model slightly underestimates the power consumed by a multiplier at high bit-widths, we actually underestimate the beneﬁt of our method, which avoids multiplications.
Revised: Speciﬁcally, since our theoretical model slightly underestimates the power consumed by a multiplier at high bit-widths, we actually underestimate the beneﬁt of our method, which avoids multiplications.

Original: The conﬁguration ﬁles will be published online to enable easy reproduction of the results.
Revised: The conﬁguration ﬁles will be published online to enable easy reproduction of the results.

Revised: In Table 5, we report the amount in [%] of the dynamic power and the static power as measured using our 5 nm simulation.

Revised: [Table]

Original: A. P OWER SIMULATION IN P YTHON
Revised: A. P OWER SIMULATION IN P YTHON

Original: Using Python, we implemented a simple serial adder and two types of multipliers: a simple serial multiplier and a Radix-2 Booth encoding multiplier.
Revised: Using Python, we implemented a simple serial adder and two types of multipliers: a simple serial multiplier and a Radix-2 Booth encoding multiplier.

Original: A serial multiplier follows the long multiplication concept in which each bit of the multiplicand multiplies the multiplier word.
Revised: A serial multiplier follows the long multiplication concept in which each bit of the multiplicand multiplies the multiplier word.

Original: This results in a word of length b + 1 bits at most, called a partial product.
Revised: This results in a word of length b + 1 bits at most, called a partial product.

Original: Going over all bits in the multiplicand results in b partial products that need to be summed.
Revised: Going over all bits in the multiplicand results in b partial products that need to be summed.

Original: The Booth encoder is more efﬁcient in terms of the number of partial products that need to be summed.
Revised: The Booth encoder is more efﬁcient in terms of the number of partial products that need to be summed.

Original: It comprises an encoder that follows a lookup table and directs whether to perform shift, addition or subtraction, basing on consecutive pairs of bits of the multiplicand.
Revised: It comprises an encoder that follows a lookup table and directs whether to perform shift, addition or subtraction, basing on consecutive pairs of bits of the multiplicand.

Original: For example, suppose we want to multiply a number x by 15, which is 1111 in binary representation.
Revised: For example, suppose we want to multiply a number x by 15, which is 1111 in binary representation.

Original: The serial multiplier performs the computation x × (cid:0) 2 3 + 2 2 + 2 1 + 2 0 (cid:1) while the
Revised: The serial multiplier performs the computation x × (cid:0) 2 3 + 2 2 + 2 1 + 2 0 (cid:1) while the

Original: Booth encoder multiplier computes x × (cid:0) 2 4 − 2 0 (cid:1) , and saves two sums.
Revised: Booth encoder multiplier computes x × (cid:0) 2 4 − 2 0 (cid:1) , and saves two sums.

Original: Those partial products are summed by half and full adders that are the major area and power consumers of a multiplier.
Revised: Those partial products are summed by half and full adders that are the major area and power consumers of a multiplier.

Original: We are focusing on the dynamic power, which is the dominant source of power consumption and linearly depends on the switching activity.
Revised: We are focusing on the dynamic power, which is a prominent source of power consumption and linearly depends on the switching activity.

Original: Therefore, in order to estimate the power, we measured the average number of bit toggles per instruction ( e.g. , multiplication and addition).
Revised: Therefore, in order to estimate the power, we measured the average number of bit toggles per instruction ( e.g. , multiplication and addition).

Original: We counted the toggles at the inputs of each half or full 1-bit adder component, both for the b -bit multiplier and for the b -bit adder.
Revised: We counted the toggles at the inputs of each half or full 1-bit adder component, both for the b -bit multiplier and for the b -bit adder.

Original: Please see an illustrative example in Fig.
Revised: Please see an illustrative example in Fig.

Original: In ﬁgures 7,8 we show the average number of toggles per instruction that were counted in the multiplier and in the adder using signed and unsigned numbers, respectively.
Revised: In ﬁgures 7,8 we show the average number of toggles per instruction that were counted in the multiplier and in the adder using signed and unsigned numbers, respectively.

Original: We ran our simulation with data drawn from a uniform distribution and a Gaussian one.
Revised: We ran our simulation with data drawn from a uniform distribution and a Gaussian one.

Original: We took the uniform distribution to be over the range [ − 2 b − 1 , 2 b − 1 ) .
Revised: We took the uniform distribution to be over the range [ − 2 b − 1 , 2 b − 1 ) .

Original: As for the Gaussian, we ﬁrst drew N full precision numbers from N (0 , 1) .
Revised: As for the Gaussian, we ﬁrst drew N full precision numbers from N (0 , 1) .

Original: Then, we divided them by their maximum (in absolute value), multiplied by 2 b − 1 , and rounded to the closest integer.
Revised: Then, we divided them by their maximum (in absolute value), multiplied by 2 b − 1 , and rounded to the closest integer.

Original: We clipped the values to the range [ − 2 b − 1 , 2 b − 1 ) in order to eliminate outliers (speciﬁcally the number 2 b − 1 ).
Revised: We clipped the values to the range [ − 2 b − 1 , 2 b − 1 ) in order to eliminate outliers (speciﬁcally the number 2 b − 1 ).

Original: In all our experiments, we took N = 36000 .
Revised: In all our experiments, we took N = 36000 .

Original: Please see an example histogram for the Gaussian distributed numbers in Fig. 5(b), where b = 8 .
Revised: Please see an example histogram for the Gaussian distributed numbers in Fig. 5(b), where b = 8 .

Original: A. O BSERVATION - 
Revised: A. O BSERVATION -

Original: When working with an accumulator having a large bit width B ( e.g. B = 32 ), a dominant source of power consumption is the bit toggling in its inputs, which is 0 .
Revised: When working with an accumulator having a large bit width B ( e.g. B = 32 ), a dominant source of power consumption is the bit toggling in its inputs, which is 0 .

Original: 5 B per instruction on average.
Revised: 5 B per instruction on average.

Original: This comes from the 2’s complement representation.
Revised: This comes from the 2’s complement representation.

Original: Hence, a signiﬁcant amount of power can be saved when switching to unsigned numbers.
Revised: Hence, a signiﬁcant amount of power can be saved when switching to unsigned numbers.

Original: For example, in the right plot of Fig. 8 we show that the power in the accumulator inputs is reduced from 16 (assuming a 32 -bit accumulator) to 0 .
Revised: For example, in the right plot of Fig. 8 we show that the power in the accumulator inputs is reduced from 16 (assuming a 32 -bit accumulator) to 0 .

Original: 5 b acc where b acc is the bit width at the input to the accumulator.
Revised: 5 b acc where b acc is the bit width at the input to the accumulator.

Original:  As for the multiplier, switching to unsigned values turns out to have a negligible effect in terms of power consumption (left plot of Fig.
Revised: [Figure] [Figure] As for the multiplier, switching to unsigned values turns out to have a negligible effect in terms of power consumption (left plot of Fig.

Original: In Fig.
Revised: In Fig.

Original: 5(a)
Revised: 5(a)

Original: we show the ratio between the power consumed by multiplication of unsigned numbers and multiplication of signed numbers, as measured in our
Revised: we show the ratio between the power consumed by multiplication of unsigned numbers and multiplication of signed numbers, as measured in our

Original: Python simulation and in the 5 nm silicon process.
Revised: Python simulation and in the 5 nm silicon process.

Original: As can be seen, this ratio is close to 1 for all bit widths.
Revised: As can be seen, this ratio is close to 1 for all bit [Figure] 1-bit [Figure]

Original: {+adder 1-bit full adder 1-bit half adder 11111 11-bit full adder 1-bit full adder 1-bit full adder 1-bit half adder 11110 
+}
Revised: {+adder 1-bit full adder 1-bit half adder 11111 11-bit full adder 1-bit full adder 1-bit full adder 1-bit half adder 11110 
+}

Original:  Therefore, we adopt the same power model for the unsigned multiplier case as in the signed setting.
Revised: [Figure] label sum” refers to the consumed due to (row 4 in also toggles widths. Therefore, we adopt the same power model for the unsigned multiplier case as in the signed setting.

Original: A.3. S WITCHING TO UNSIGNED ARITHMETIC
Revised: A.3. S WITCHING TO UNSIGNED ARITHMETIC

Original: Figure 11(a) compares the average power consumption of a signed MAC to that of an unsigned
Revised: Figure 11(a) compares the average power consumption of a signed MAC to that of an unsigned

Original: MAC for a 32 bit accumulator.
Revised: MAC for a 32 bit accumulator.

Original: Speciﬁcally, we are dividing P umult + P uacc by P mult + P acc .
Revised: Speciﬁcally, we are dividing P umult + P uacc by P mult + P acc .

Original: In this setting, it can be seen for example that when working with b = 4 bits for the weights and activations, unsigned MACs are 33% cheaper in power.
Revised: In this setting, it can be seen for example that when working with b = 4 bits for the weights and activations, unsigned MACs are 33% cheaper in power.

Original: The approach we suggest for switching a linear layer ( e.g. , convolution, fully connected) to work with unsigned arithmetic is illustrated schematically in
Revised: The approach we suggest for switching a linear layer ( e.g. , convolution, fully connected) to work with unsigned arithmetic is illustrated schematically in

Original: Fig. 11(b).
Revised: Fig. 11(b).

Revised: Accumulator bit width The bit width of the accumulator is commonly chosen to be 32.

Revised: One of the main reasons for that is this allows ﬂexibility in changing the bit widths of the activations and weights

Revised: ( e.g. , from 4-bit to 8-bit and vice versa).

Revised: Nevertheless, if we are not concerned with ﬂexibility, then when quantizing the activations and weights to less than 8 bits, we can use an accumulator with less than 32 bits.

Revised: The required accumulator bit width B can be calculated by

Original: [Figure]
Revised: [Equation]

Revised: (20) where k is the convolution kernel size, and C in is the number of input channels.

Revised: In Table 6 we analyze the case of ResNet networks.

Revised: We choose the layer with the largest value of k 2 C in , which is 3x3x (Table 1 in He et al.

Revised: We calculate the required bit width for the accumulator ( e.g. , B ) when the activations and weights are quantized to 2 - 6 bits.

Revised: In addition, we calculate the power save in [%] when switching to unsigned arithmetic.

Revised: As can be seen, even with smaller accumulator bit widths, switching to unsigned arithmetic leads to a signiﬁcant saving in power.

Revised: This is also visually

Original: [Figure]
Revised: [Figure]

Revised: Assuming the elements of its input x are non-negative (due to the preceding ReLU) this makes all

Revised: MACs unsigned and thus substantially reduces illustrated in Fig.

Revised: where we repeat the experiment of Fig.

Revised: 1 bit with a 17 bit accumulator for the 2-bit networks, and with a 21 bit accumulator for the 4-bit networks.

Original: A. O BSERVATION - 
Revised: A. O BSERVATION -

Original: We now analyze the case where the inputs of the multiplier have different bit widths, b x and b w .
Revised: We now analyze the case where the inputs of the multiplier have different bit widths, b x and b w .

Original: In
Revised: In

Original: Fig.
Revised: Fig.

Original: 9 we show the average bit toggles in signed and unsigned Booth encoder multiplication, for b w ≤ b x .
Revised: 9 we show the average bit toggles in signed and unsigned Booth encoder multiplication, for b w ≤ b x .

Original: In Fig.
Revised: In Fig.

Original: 10 we show the same analysis for the simple serial multiplier.
Revised: 10 we show the same analysis for the simple serial multiplier.

Original: We observe that when

Original: [Figure]

Original: [-adder 1-bit full adder 1-bit full adder 1-bit half adder 11111 11-bit full adder 1-bit full adder 1-bit full adder 1-bit half adder 11110 
-]
Revised: [-adder 1-bit full adder 1-bit full adder 1-bit half adder 11111 11-bit full adder 1-bit full adder 1-bit full adder 1-bit half adder 11110 
-]

Original: [Figure]

Original: The label “acc sum” refers to the power consumed due to the toggles at the output of the accumulator (row 4 in Table 1) which also toggles the bits in the FF.

Original: Again, the simulation agrees with our model.

Original:  working with signed numbers (the common setting), the power is mostly affected by the larger bit width ( b x in this case).
Revised: We observe that when working with signed numbers (the common setting), the power is mostly affected by the larger bit width ( b x in this case).

Original: In the case of unsigned numbers, there is some power save when reducing one of the bit widths.
Revised: In the case of unsigned numbers, there is some power save when reducing one of the bit widths.

Original: In other words, Eq.
Revised: In other words, Eq.

Original: (7) in the paper is accurate for the popular signed case and behaves as an upper bound for the unsigned case.
Revised: (7) in the paper is accurate for the popular signed case and behaves as an upper bound for the unsigned case.

Original: The difference between Eq.
Revised: The difference between Eq.

Original: (7) and the actual power consumption in the unsigned setting is more dominant for the simple serial multiplier.
Revised: (7) and the actual power consumption in the unsigned setting is more dominant for the simple serial multiplier.

Revised: Therefore, in certain settings,

Revised: 2- BIT 3- BIT

Revised: 4- BIT 5- BIT

Revised: 6- BIT

Revised: R EQUIRED BIT WIDTH B    25 P OWER SAVE FOR A B BIT ACCUMULATOR 39% 28% 21% 16% 13% P OWER SAVE FOR A 32 BIT ACCUMULATOR 58% 44% 33% 25% 19%

Original: Therefore, in certain settings, there is an additional beneﬁt of switching to unsigned arithmetic, which we did not report in the experiments in the paper (i.e. the horizontal arrows in Fig.
Revised: [Figure] we there is an additional beneﬁt of switching to unsigned arithmetic, which we did not report in the experiments in the paper (i.e. the horizontal arrows in Fig.

Original: 1 should actually be slightly longer in some cases).
Revised: 1 should actually be slightly longer in some cases).

Original: Yet, this effect is relatively small compared to the reduction in bit ﬂips in the accumulator.
Revised: Yet, this effect is relatively small compared to the reduction in bit ﬂips in the accumulator.

Original: We validated our observation on the 5 -nm silicon process setup.
Revised: We validated our observation on the 5 -nm silicon process setup.

Original: We used 8 × 8 multiplier and measured the power when one of the inputs was drawn uniformly from [0 , 2 7 ) and the other from [0 , 2 3 ) .
Revised: We used 8 × 8 multiplier and measured the power when one of the inputs was drawn uniformly from [0 , 2 7 ) and the other from [0 , 2 3 ) .

Original: We got 95% of the power that was measured when both inputs were drawn from [0 , 2 7 ) .
Revised: We got 95% of the power that was measured when both inputs were drawn from [0 , 2 7 ) .

Original: In case of signed values, when one of the inputs was drawn uniformly from [ − 2 7 , 2 7 ) and the other from [ − 2 3 , 2 3 ) we observed 100% of the power that was measured when both inputs were drawn from [ − 2 7 , 2 7 ) .
Revised: In case of signed values, when one of the inputs was drawn uniformly from [ − 2 7 , 2 7 ) and the other from [ − 2 3 , 2 3 ) we observed 100% of the power that was measured when both inputs were drawn from [ − 2 7 , 2 7 ) .

Original: Note that in order to avoid changing the multiplier’s architecture, when switching to unsigned numbers, we use only half the range allowed by the bit width b x , i.e. [0 , 2 b x − 1 ) .
Revised: Note that in order to avoid changing the multiplier’s architecture, when switching to unsigned numbers, we use only half the range allowed by the bit width b x , i.e. [0 , 2 b x − 1 ) .

Original: Therefore, we obtain a representation with half of the 2 b x levels of the signed case (note that we also need to replace − 2 b x − 1 with − 2 b x − 1 + 1 ).
Revised: Therefore, we obtain a representation with half of the 2 b x levels of the signed case (note that we also need to replace − 2 b x − 1 with − 2 b x − 1 + 1 ).

Original: If we permit architectural changes, then a better way to represent unsigned numbers would be to replace the signed multiplier by a ( b x + 1) ×
Revised: If we permit architectural changes, then a better way to represent unsigned numbers would be to replace the signed multiplier by a ( b x + 1) ×

Original: ( b x + 1) multiplier that can support unsigned and signed multiplications but consumes much more power, or to work with a b x × b x unsigned multiplier that allow representation in the full interval of [0 , 2 b x ) and then will follow Eq.
Revised: ( b x + 1) multiplier that can support unsigned and signed multiplications but consumes much more power, or to work with a b x × b x unsigned multiplier that allow representation in the full interval of [0 , 2 b x ) and then will follow Eq.

Original: (7) again.
Revised: (7) again.

Original: [Figure]

Original: Assuming the elements of its input x are non-negative (due to the preceding ReLU) this makes all

Original: MACs unsigned and thus substantially reduces power consumption.

Original: A. A DDITIONAL RESULTS A.5. P OST TRAINING QUANTIZATION
Revised: A. A DDITIONAL RESULTS A.5. P OST TRAINING QUANTIZATION

Original: Similarly to Table  we now examine PANN’s performance on additional networks.
Revised: Similarly to Table 2, we now examine PANN’s performance on additional networks.

Original: We use the same methods for activations quantization as in Table 2: ACIQ 6 (Banner et al., 2019), ZeroQ 7 (Cai et al., 2020), GDFQ 8 (Shoukai et al., 2020) and BRECQ 9 (Li et al., 2021).
Revised: We use the same methods for activations quantization as in Table 2: ACIQ 6 (Banner et al., 2019), ZeroQ 7 (Cai et al., 2020), GDFQ 8 (Shoukai et al., 2020) and BRECQ 9 (Li et al., 2021).

Original: We also add Dynamic
Revised: We also add Dynamic

Original: Quantization, which quantizes the activation and weights on the ﬂy at inference time, according to their dynamic ranges.
Revised: Quantization, which quantizes the activation and weights on the ﬂy at inference time, according to their dynamic ranges.

Original: The results are reported in Tables 5-7.
Revised: The results are reported in Tables 7-9.

Original: We ﬁrst change all networks to work with unsigned arithmetic and measure the classiﬁcation accuracy, which serves as a baseline (see left side of each method ‘Base.’).
Revised: We ﬁrst change all networks to work with unsigned arithmetic and measure the classiﬁcation accuracy, which serves as a baseline (see left side of each method ‘Base.’).

Original: As mentioned, this step already saves a signiﬁcant amount of power, without any change in classiﬁcation accuracy, compared to signed MAC arithmetic.
Revised: As mentioned, this step already saves a signiﬁcant amount of power, without any change in classiﬁcation accuracy, compared to signed MAC arithmetic.

Original: We use the unsigned MAC power consumption as the power budget P and follow Alg. 1.
Revised: We use the unsigned MAC power consumption as the power budget P and follow Alg. 1.

Original: We report the classiﬁcation accuracy on the right side of each columns (see ‘Our’).
Revised: We report the classiﬁcation accuracy on the right side of each columns (see ‘Our’).

Original: [Table]
Revised: [Table]

Original: leftmost column). The bit width determines the power P , which we specify in parentheses in units of Giga bit-ﬂips. The power is calculated as
Revised: B ASE

Original: P umult + P uacc
Revised: [Table] O [Table]

Original: (Eqs.
Revised: B ASE .

Original: (3),(4)) times the number of MACs in the network. ( 1 . 82 × 10 9 in ResNet-18). In each row, our variant is tuned to work at the same power budget, for which we choose the optimal ˜ b x and R using Alg. 1.
Revised: O

Original: [Table]
Revised: [Table]

Original: P , which we specify in parentheses in units of Giga bit-ﬂips. The power is calculated as
Revised: B ASE .

Original: P umult + P uacc
Revised: O UR [Table]

Original: (Eqs.
Revised: ASE .

Original: (3),(4)) times the number of MACs in the network.
Revised: O

Original: 33 × 10 9 in MobileNet-V2).
Revised: [Table]

Original:  In each row, our variant is tuned to work at the same power budget, for which we choose the optimal ˜ b x and R using Alg. 1.
Revised: [Table] [Table] In ﬁgures

Original: In ﬁgures 12-13, we demonstrate PANN at post training for different networks under power constrains of 4-bit and 2-bit unsigned MAC. 
Revised: [Table] we demonstrate PANN at post training for different networks under power constrains of 4-bit and 2-bit unsigned MAC. In each ﬁgure [Table]

Original: In each ﬁgure, we start by running the speciﬁed approach to  quantize the bits and the activation to 4 or 2 bits (see caption).
Revised: we start by running the speciﬁed approach to [Table] quantize the bits and the activation to 4 or 2 bits (see caption).

Original: We measure the power in bit ﬂips.
Revised: We measure the power in bit ﬂips.

Original: Speciﬁcally, the power of each signed MAC is calculated by P mult + P acc (Eqs.
Revised: Speciﬁcally, the power of each signed MAC is calculated by P mult + P acc (Eqs.

Original: times the number of MACs in the network.
Revised: times the number of MACs in the network.

Original: Then, we switch to unsigned arithmetic ( ← ).
Revised: Then, we switch to unsigned arithmetic ( ← ).

Original: In this case the power is calculated by P umult + P uacc (Eqs.
Revised: In this case the power is calculated by P umult + P uacc (Eqs.

Original: (3),(4)) times the number of MACs in the network.
Revised: (3),(4)) times the number of MACs in the network.

Original: For simplicity, let us denote the total power as P u .
Revised: For simplicity, let us denote the total power as P u .

Original: As can be seen, this stage save power but does not change the accuracy (points marked as ‘Unsigned’).
Revised: As can be seen, this stage save power but does not change the accuracy (points marked as ‘Unsigned’).

Original: Constraining to the same power budget P u , we apply PANN ( ↑ ).
Revised: Constraining to the same power budget P u , we apply PANN ( ↑ ).

Original: Using Alg. 1, we calculate the optimal activation bit width and the corresponding addition factor (points marked as ‘PANN’).
Revised: Using Alg. 1, we calculate the optimal activation bit width and the corresponding addition factor (points marked as ‘PANN’).

Original: We can see a dramatic improvement in classiﬁcation accuracy without any change in the power consumption.
Revised: We can see a dramatic improvement in classiﬁcation accuracy without any change in the power consumption.

Revised: [Figure]

Original: A.5. Q UANTIZATION AWARE TRAINING
Revised: A.5. Q UANTIZATION AWARE TRAINING

Original: In Table 8, we report the classiﬁcation accuracy of different networks on ImageNet,when we use
Revised: In Table 10, we report the classiﬁcation accuracy of different networks on ImageNet,

Original: PANN during training. Each row deﬁnes a speciﬁc power budget, corresponding to 2-bit, 3-bit and [Table]
Revised: when we

Original: [Table]
Revised: [Figure]

Original: PANN

Original: PANN

Original: Unsigned

Original: PANN

Original: PANN

Original: Unsigned ACIQ Unsigned ACIQ Unsigned
Revised: Using remove

Original: ACIQ (a) Improving upon ACIQ (4-bit MAC)

Original: GDFQ

Original: PANN

Original: GDFQ Unsigned

Original: PANN

Original: PANN

Original: PANN

Original: Unsigned

Original: Unsigned

Original: GDFQ

Original: GDFQ (b) Improving upon GDFQ (4-bit MAC)

Original: 4-bit unsigned MACs (weights and activations have equal bit widths).
Revised: 4-bit unsigned MACs (weights and activations have equal bit widths).

Original: We compare our results to
Revised: We compare our results to

Original: LSQ (Esser et al., 2019), whose accuracy is reported in parentheses.
Revised: LSQ (Esser et al., 2019), whose accuracy is reported in parentheses.

Original: Note that the total number of bit ﬂips differs between the networks, because each has a different number of MACS in its forward pass.
Revised: Note that the total number of bit ﬂips differs between the networks, because each has a different number of MACS in its forward pass.

Original: Therefore, instead of specifying results as a function of the total number of bit ﬂips, we report results as a function of the bit-width.
Revised: Therefore, instead of specifying results as a function of the total number of bit ﬂips, we report results as a function of the bit-width.

Original: Each bit-width deﬁnes a power budget for which we tune PANN (Alg. 1), where we use LSQ to quantize the activations.
Revised: Each bit-width deﬁnes a power budget for which we tune PANN (Alg. 1), where we use LSQ to quantize the activations.

Original: We can see that PANN outperforms LSQ at all power budgets.
Revised: We can see that PANN outperforms LSQ at all power budgets.

Original: A.5. A DDITIONAL COMPARISONS TO MULTIPLICATION - FREE METHODS

Original: In tables 9-10 we report additional comparisons with the recent multiplication free methods

Original: ShiftAddNet (You et al., 2020) and AdderNet (Chen et al., 2020), this time on the CIFAR100 and

Original: MHEALTH Banos et al.

Original: (2014) datasets.

Original: Here again PANN is used during training, like the competing methods.

Original: [Figure]

Original: Power (bit-width)
Revised: Power (bit-width)

Original: ResNetResNet- ResNetResNet-
Revised: ResNetResNet- ResNetResNet-

Original: VGG-16bn
Revised: VGG-16bn

Original: FP 70.
Revised: FP 70.

Original: [Table]
Revised: [Table]

Revised: A.5. A DDITIONAL COMPARISONS TO MULTIPLICATION - FREE METHODS

Original: [Table]
Revised: In tables 11-12 we report additional comparisons with the recent multiplication free methods ShiftAddNet (You et al., 2020) and AdderNet (Chen et al., 2020), this time on the CIFAR100 and MHEALTH Banos et al. (2014) datasets.

Revised: Here again PANN is used during training, like the competing methods.

Revised: A. H YPER PARAMETERS FOR PANN IN QAT A.6. LSQ

Revised: In all experiments we used the SGD optimizer with momentum of 0.9 and weight decay of 10 − 4 .

Revised: We used the softmax cross entropy loss.

Revised: Unlike the original paper, we started the training from

Original: [Table] 
Revised: [Table] [Table]

Revised: pre-trained networks and with a smaller initial learning rate of 10 − 3 .

Revised: Please refer to Table 13 for

Revised: PANN details, scheduling and number of epochs used for the training.

Original: [Table]
Revised: [Equation] P ˜ b x R INIT .

Revised: LR

Revised: LR SCHEDULE

Revised: B s

Revised: EPOCHS

Revised: R ES N ETLSQ (2/2)  2.

Revised: 1 e −× 0 .

Original: {+EVERY 
+}
Revised: {+EVERY 
+}

Revised: EPOCHS

Revised: R ES N ETLSQ (3/3)  2.

Revised: 1 e −× 0 .

Original: {+EVERY 
+}
Revised: {+EVERY 
+}

Revised: EPOCHS

Revised: R ES N ETLSQ (4/4)  3.

Revised: 1 e −× 0 .

Original: {+EVERY 
+}
Revised: {+EVERY 
+}

Revised: EPOCHS

Original: [Table]
Revised: R ES N ETLSQ (2/2)  2. 1 e −× 0 . EVERY 20 EPOCHS R ES N ETLSQ (3/3)  2.

Revised: 1 e −× 0 .

Revised: EVERY 20 EPOCHS

Original: [Table]
Revised: R ES N ETLSQ (4/4)  3. 1 e −× 0 . EVERY 20 EPOCHS R ES N ETLSQ (2/2)  2.

Revised: 1 e −× 0 .

Revised: EVERY 20 EPOCHS

Original: [Table]
Revised: R ES N ETLSQ (3/3)  2. 1 e −× 0 . EVERY 20 EPOCHS R ES N ETLSQ (4/4)  3.

Revised: 1 e −× 0 .

Revised: EVERY 20 EPOCHS

Original: [Table]
Revised: R ES N ETLSQ (2/2)  2. 1 e −× 0 . EVERY 20 EPOCHS R ES N ETLSQ (3/3)  2.

Revised: 1 e −× 0 .

Revised: EVERY 20 EPOCHS

Revised: R ES N ETLSQ (4/4)  3.

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: , we follow Alg. 1 to calculate the optimal activation bit width and addition factor.
Revised: In all experiments we followed the training regime described in (You et al., 2020). Speciﬁcally, for CIFAR10 or CIFAR100 we used a batch size of 256, and 160 epochs.

Revised: The initial learning rate was 0.1 and then divided by 10 at the 80-th and the 120-th epoch.

Revised: We used the SGD optimizer with momentum of 0.9 and weight decay of 1 e − 4 .

Revised: For the MHEALTH dataset, we used only 40 epochs to train.

Revised: The initial learning rate was 0.01 and then divided by 10 at the 20-th and the 30-th epochs.

Revised: Similarly to the CIFAR experiments, we used SGD optimizer with momentum of 0.9 and weight decay of 1 e − 4 .

Original: A. H ARDWARE - ACCURACY TRADE - OFF
Revised: A. H ARDWARE - ACCURACY TRADE - OFF

Original: When operating on a single image at inference time (rather than on a large batch), the memory footprint of the weights is not negligible anymore
Revised: When operating on a single image at inference time (rather than on a large batch), the memory footprint of the weights is not negligible anymore

Original: (Mishra et al., 2017).
Revised: (Mishra et al., 2017).

Original: Therefore, we need to also account for the bit-width required for storing the quantized weights.
Revised: Therefore, we need to also account for the bit-width required for storing the quantized weights.

Original: In Table 11 we report the optimal activation bit width and addition factor for each power constraint in a certain setting.
Revised: In Table 14 we report the optimal activation bit width and addition factor for each power constraint in a certain setting.

Original: Speciﬁcally, we use ZeroQ to quantize the activations of a pre-trained full-precision ResNet-50.
Revised: Speciﬁcally, we use ZeroQ to quantize the activations of a pre-trained full-precision ResNet-50.

Original: We then measure the maximal addition factor per neuron, which deﬁnes the bit width b R required to store the weights.
Revised: We then measure the maximal addition factor per neuron, which deﬁnes the bit width b R required to store the weights.

Original: We can observe that overall, the increase in the runtime memory footprint of the weights is relatively low, especially in the low power regimes.
Revised: We can observe that overall, the increase in the runtime memory footprint of the weights is relatively low, especially in the low power regimes.

Original: Up to now, we have only shown results with the bit width ˜
Revised: Up to now, we have only shown results with the bit width ˜

Original: b x (and corresponding additions factor R ) that is optimal in terms of classiﬁcation accuracy.
Revised: b x (and corresponding additions factor R ) that is optimal in terms of classiﬁcation accuracy.

Original: However, for a given power budget P , choosing ˜ b
Revised: However, for a given power budget P , choosing ˜ b

Original: x
Revised: x

Original: and R ) can be done while also accounting for other factors, like latency and memory footprint.
Revised: and R ) can be done while also accounting for other factors, like latency and memory footprint.

Original: We illustrate this in Table 12 for the case of a power constraing corresponding to 2 -bit MAC.
Revised: We illustrate this in Table 15 for the case of a power constraing corresponding to 2 -bit MAC.

Original: Here, we report results for all options for ˜ b
Revised: Here, we report results for all options for ˜ b

Original: x
Revised: x

Original: and R that conform to that power budget.
Revised: and R that conform to that power budget.

Original: While ˜ b x =and R = 1 . 16 is optimal in terms of classiﬁcation accuracy, the user can choose other options, e.g. , according to latency or memory constraints.[Table] [Table] A. Q UANTIZATION ERROR ANALYSIS
Revised: While ˜ b x =and R = 1 . 16 is optimal in terms of classiﬁcation accuracy, the user can choose other options, e.g. , according to latency or memory constraints.

Original: Figure 14 (ﬁrst row) depicts MSE PANN (Eq.

Original: as a function of ˜ b x for several power budgets P .

Original: First, it can be seen that our theoretical analysis agrees well with simulations.

Original: Second, it shows that the optimal ˜ b x (where the minimum MSE is attained

Original: ) increases with the power budget.

Original: This implies that at higher power budgets, it is preferable to increase the bit width of the activations on the expense of reducing the number of additions R .

Original: The second row of the ﬁgure illustrates that the qualitative conclusions drawn from our uniform distribution analysis also hold when the weights are Gaussian and the activations are Gaussian numbers after a ReLU function (here we used the ACIQ quantizer (Banner et al., 2019)).

Original: In the third row, we show that similar behaviors characterise the error rates of a ResNet-18 model for ImageNet classiﬁcation when using PANN to quantize its weights and ACIQ to quantize the activations.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: footprint and latency increase for all different values of ˜ b x and R that lead to the same power of a 2 -bit unsigned MAC (blue curve in Figure 3(a)). For each setting we measure the classiﬁcation accuracy of ResNet50 on ImageNet. Here we use ACIQ (Banner et al., 2019) for quantizing the activations.
Revised: Bit Width

Original: The baseline (pure ACIQ) accuracy is 0.20% (Table 2, third column, last row). A. P ROOF OF E Q . (14) Let w and x be statistically independent random vectors, each with iid components. Recall that w q and x q are obtained by applying a scalar function on each of the elements of w and x , respectively.
Revised: Bit Width

Original: [-Therefore w q and x q also have iid components. We assume that w = w q + ε w and x = x q + ε x , where E [ ε w | w ] = 
 and E [ ε x | x ] = 0 . Then we have that-]
Revised: [-Therefore w q and x q also have iid components. We assume that w = w q + ε w and x = x q + ε x , where E [ ε w | w ] = 
 and E [ ε x | x ] = 0 . Then we have that-]Bit Width

Original: [Equation]

Original: (20) and therefore,

Original: [Equation]

Original: Bit Width
Revised: Bit Width

Original: Bit Width

Original: Bit Width

Original: Bit Width

Original: G a u ss i a n
Revised: G a u ss i a n

Original: U n i f
Revised: U n i f

Original: o r m
Revised: o r m

Original: I m a g e N e t
Revised: I m a g e N e t

Original: Bit Width Bit Width
Revised: Bit Width Bit Width

Original: Bit Width Bit Width
Revised: Bit Width Bit Width

Original: Bit Width Bit Width
Revised: Bit Width Bit Width

Original: Bit Width Bit Width
Revised: Bit Width Bit Width

Original: Power of unsigned 2-bit MAC
Revised: Power of unsigned 2-bit MAC

Original: Power of unsigned 3-bit MAC
Revised: Power of unsigned 3-bit MAC

Original: Power of unsigned 4-bit MAC
Revised: Power of unsigned 4-bit MAC

Original: Power of unsigned 5-bit MAC
Revised: Power of unsigned 5-bit MAC

Original: M S E M S E
Revised: M S E M S E

Original: E rr o r
Revised: E rr o r

Original: M S E M S E
Revised: M S E M S E

Original: E rr o r
Revised: E rr o r

Original: M S E M S E
Revised: M S E M S E

Original: E rr o r
Revised: E rr o r

Original: M S E M S E
Revised: M S E M S E

Original: E rr o r
Revised: E rr o r

Revised: A. Q UANTIZATION ERROR ANALYSIS

Revised: Figure 15 (ﬁrst row) depicts MSE PANN (Eq.

Revised: as a function of ˜ b x for several power budgets P .

Revised: First, it can be seen that our theoretical analysis agrees well with simulations.

Revised: Second, it shows that the optimal ˜ b x (where the minimum MSE is attained) increases with the power budget.

Revised: This implies that at higher power budgets, it is preferable to increase the bit width of the activations on the expense of reducing the number of additions R .

Revised: The second row of the ﬁgure illustrates that the qualitative conclusions drawn from our uniform distribution analysis also hold when the weights are Gaussian and the activations are Gaussian numbers after a ReLU function (here we used the ACIQ behaviors characterise rates of a ResNet-18 model for ImageNet classiﬁcation when using PANN to quantize its weights and ACIQ to quantize the

Revised: A. P ROOF OF E Q .

Revised: [Equation]

Revised: Let w and x be statistically independent random vectors, each with iid components.

Revised: Recall that w q and x q are obtained by applying a scalar function on each of the elements of w and x , respectively.

Revised: Therefore w q and x q also have iid components.

Original: {+We assume that w = w q + ε w and x = x q + ε x , where E [ ε w | w ] = 
+}
Revised: {+We assume that w = w q + ε w and x = x q + ε x , where E [ ε w | w ] = 
+}

Revised: and E [ ε x | x ] = 0 .

Revised: Then we have that

Revised: [Equation]

Revised: 21) and therefore,

Revised: [Equation]

Original: We now turn to show that the last three terms equal zero.
Revised: We now turn to show that the last three terms equal zero.

Original: For the ﬁrst of those , we have
Revised: For the ﬁrst of those terms, we have

Original: [Equation]
Revised: [Equation]

Original: (22) where in the second line we used the law of total expectations, in the fourth line we used the fact that the pair { w , ε w } is statistically independent of the pair { x , ε x } , and in the ﬁfth line we used the fact that E [ ε x x T ]
Revised: where in the second line we used the law of total expectations, in the fourth line we used the fact that the pair { w , ε w } is statistically independent of the pair { x , ε x } , and in the ﬁfth line we used the fact that E [ ε x x T ]

Original: =
Revised: =

Original: E [ E [ ε x x T | x ]] =
Revised: E [ E [ ε x x T | x ]] =

Original: E [ E [ ε x | x ] x T ] = 0 because of our assumption
Revised: E [ E [ ε x | x ] x T ] = 0 because of our assumption

Original: that E [ ε x | x ] = 0 .
Revised: that E [ ε x | x ] = 0 .

Original: For the second among the last three terms in (21), we have that
Revised: For the second among the last three terms in (22), we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where we used the fact that the pair { w , ε w } is independent of ε x , and E [ wε T w ]
Revised: where we used the fact that the pair { w , ε w } is independent of ε x , and E [ wε T w ]

Original: =
Revised: =

Original: E [ E [ wε T w | w ]] =
Revised: E [ E [ wε T w | w ]] =

Original: E [ w E [ ε w | w ] T ] = 0 because of our assumption that E [ ε w | w ] = 0 .
Revised: E [ w E [ ε w | w ] T ] = 0 because of our assumption that E [ ε w | w ] = 0 .

Original: For the the last term in (21), we have that
Revised: For the the last term in (22), we have that

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that the pair { x , ε x } is independent of ε w
Revised: where we used the fact that the pair { x , ε x } is independent of ε w

Original: [Equation]
Revised: [Equation]

Original: and E [ xε T x ] = 0 , as in (22).
Revised: and E [ xε T x ] = 0 , as in (23).

Original: We thus remain only with the ﬁrst three terms of (21) [Equation] so that
Revised: We thus remain only with the ﬁrst three terms of (22), so that

Original: [Equation]
Revised: [Equation]

Original: where σ 2 w , σ 2 x , σ 2 ε w , and σ 2 ε x denote the second-order moments of the elements of w , x , ε w , and ε x , respectively, and I is the d × d identity matrix.
Revised: where σ 2 w , σ 2 x , σ 2 ε w , and σ 2 ε x denote the second-order moments of the elements of w , x , ε w , and ε x , respectively, and I is the d × d identity matrix.

Original: Here, in the ﬁfth equality we used the fact that w is independent of ε x , x is independent of ε w , and ε w is independent of ε x .
Revised: Here, in the ﬁfth equality we used the fact that w is independent of ε x , x is independent of ε w , and ε w is independent of ε x .

Original: In the sixth equality we used the fact that ε x and ε w are iid vectors with zero mean, since E [ ε x ] =
Revised: In the sixth equality we used the fact that ε x and ε w are iid vectors with zero mean, since E [ ε x ] =

Original: E [ E [ ε x | x ]] = 0 and E [ ε w ] =
Revised: E [ E [ ε x | x ]] = 0 and E [ ε w ] =

Original: E [ E [ ε w | w ]] = 0 .
Revised: E [ E [ ε w | w ]] = 0 .

Original: This completes the proof of Eq.
Revised: This completes the proof of Eq.

Original: (14) in the main text.
Revised: (14) in the main text.