Original: P RECISION G ATING : I MPROVING N EURAL N ETWORK E FFICIENCY WITH D YNAMIC D UAL -P RECISION A CTIVATIONS
Revised: P RECISION G ATING : I MPROVING N EURAL N ETWORK E FFICIENCY WITH D YNAMIC D UAL -P RECISION A CTIVATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose precision gating (PG), an end-to-end trainable dual-precision quantization technique for deep neural networks.
Revised: We propose precision gating (PG), an end-to-end trainable dual-precision quantization technique for deep neural networks.

Original: PG computes most features in a low precision and only a small proportion of important features in a higher precision.
Revised: PG computes most features in a low precision and only a small proportion of important features in a higher precision.

Original: Precision gating is very lightweight and widely applicable to many neural network architectures.
Revised: Precision gating is very lightweight and widely applicable to many neural network architectures.

Original: Experimental results show that precision gating can greatly reduce the average bitwidth of computations in both CNNs and LSTMs with negligible accuracy loss.
Revised: Experimental results show that precision gating can greatly reduce the average bitwidth of computations in both CNNs and LSTMs with negligible accuracy loss.

Original: Compared to state-of-the-art counterparts, PG achieves the same or better accuracy with 2.4 × less compute on ImageNet.
Revised: Compared to state-of-the-art counterparts, PG achieves the same or better accuracy with 2.4 × less compute on ImageNet.

Original: Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement in perplexity per word with 2.8 × computational cost reduction on LSTM on the Penn Tree Bank dataset.
Revised: Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement in perplexity per word with 2.8 × computational cost reduction on LSTM on the Penn Tree Bank dataset.

Original: Precision gating has the potential to greatly reduce the execution costs of DNNs on both commodity and dedicated hardware accelerators.
Revised: Precision gating has the potential to greatly reduce the execution costs of DNNs on both commodity and dedicated hardware accelerators.

Original: We implement the sampled dense-dense matrix multiplication kernel in PG on CPU, which achieves up to 8.3 × wall clock speedup over the dense baseline.
Revised: We implement the sampled dense-dense matrix multiplication kernel in PG on CPU, which achieves up to 8.3 × wall clock speedup over the dense baseline.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, deep neural networks (DNNs) have demonstrated excellent performance on many computer vision and language modeling tasks including image classiﬁcation, semantic segmentation, face recognition, machine translation and image captioning (Krizhevsky et al., 2012;
Revised: In recent years, deep neural networks (DNNs) have demonstrated excellent performance on many computer vision and language modeling tasks including image classiﬁcation, semantic segmentation, face recognition, machine translation and image captioning (Krizhevsky et al., 2012;

Original: He et al., 2015; Ronneberger et al., 2015; Chen et al., 2016; Zhao et al., 2018; Schroff et al., 2015; Luong et al., 2015; Vaswani et al., 2017).
Revised: He et al., 2015; Ronneberger et al., 2015; Chen et al., 2016; Zhao et al., 2018; Schroff et al., 2015; Luong et al., 2015; Vaswani et al., 2017).

Original: One visible trend in DNN design is that as researchers strive for better accuracy, both the model size and the number of DNN layers have increased over time (Xu et al., 2018).
Revised: One visible trend in DNN design is that as researchers strive for better accuracy, both the model size and the number of DNN layers have increased over time (Xu et al., 2018).

Original: At the same time, there is a growing demand to push deep learning technology to edge devices such as mobile phones, Internet of Things (IoT) devices, and lightweight drones (Wu et al., 2019).
Revised: At the same time, there is a growing demand to push deep learning technology to edge devices such as mobile phones, Internet of Things (IoT) devices, and lightweight drones (Wu et al., 2019).

Original: The limited computational, memory, and energy budgets on these devices create major challenges for the deployment of large DNN models on the edge.
Revised: The limited computational, memory, and energy budgets on these devices create major challenges for the deployment of large DNN models on the edge.

Original: Neural network quantization is an important technique for improving the hardware efﬁciency of
Revised: Neural network quantization is an important technique for improving the hardware efﬁciency of

Original: DNN execution.
Revised: DNN execution.

Original: Circuits operating at a lower precision are smaller, faster, and consume less energy.
Revised: Circuits operating at a lower precision are smaller, faster, and consume less energy.

Original: Numerous studies have shown that full-precision ﬂoating-point computation is not necessary for
Revised: Numerous studies have shown that full-precision ﬂoating-point computation is not necessary for

Original: DNN inference — quantized ﬁxed-point models produce competitive results with a small or zero loss in prediction accuracy (Lin et al., 2016;
Revised: DNN inference — quantized ﬁxed-point models produce competitive results with a small or zero loss in prediction accuracy (Lin et al., 2016;

Original: He et al., 2016; Zhou et al., 2016; 2017).
Revised: He et al., 2016; Zhou et al., 2016; 2017).

Original: In some cases, quantization may even improve model generalization by acting as a form of regularization.
Revised: In some cases, quantization may even improve model generalization by acting as a form of regularization.

Original: Existing studies mainly focus on static quantization, in which the precision of each weight and activation is ﬁxed at run time (Hubara et al., 2017;
Revised: Existing studies mainly focus on static quantization, in which the precision of each weight and activation is ﬁxed at run time (Hubara et al., 2017;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: Along this line of work, researchers have explored tuning the bitwidth per layer (Wu et al., 2018;
Revised: Along this line of work, researchers have explored tuning the bitwidth per layer (Wu et al., 2018;

Original: Wang et al., 2018a; Dong et al., 2019) as well as various types of quantization functions (Wang et al., 2018b; Courbariaux et al., 2016; Li et al., 2016;
Revised: Wang et al., 2018a; Dong et al., 2019) as well as various types of quantization functions (Wang et al., 2018b; Courbariaux et al., 2016; Li et al., 2016;

Original: Zhou et al., 2016).
Revised: Zhou et al., 2016).

Original: However, static DNN quantization methods cannot exploit input-dependent characteristics, where certain features during inference may contribute less to the classiﬁcation result than others and can be computed in lower precision.
Revised: However, static DNN quantization methods cannot exploit input-dependent characteristics, where certain features during inference may contribute less to the classiﬁcation result than others and can be computed in lower precision.

Original: For example, in computer vision tasks the pixels representing the object of interest are typically more important than the background pixels.
Revised: For example, in computer vision tasks the pixels representing the object of interest are typically more important than the background pixels.

Original: In this paper, we propose to reduce the inefﬁciency of a statically quantized DNN via precision gating (PG), which computes most features with low-precision arithmetic and only updates few im- portant features to high precision.
Revised: In this paper, we propose to reduce the inefﬁciency of a statically quantized DNN via precision gating (PG), which computes most features with low-precision arithmetic and only updates few im- portant features to high precision.

Original: More concretely, PG ﬁrst executes a DNN layer in low precision and identiﬁes the output features with large magnitude as important features.
Revised: More concretely, PG ﬁrst executes a DNN layer in low precision and identiﬁes the output features with large magnitude as important features.

Original: It then computes a sparse update to increase the precision of those important output features.
Revised: It then computes a sparse update to increase the precision of those important output features.

Original: Intuitively, small values make very little contribution to the DNNs output; approximating them in low-precision is reasonable.
Revised: Intuitively, small values make very little contribution to the DNN’s output; approximating them in low-precision is reasonable.

Original: Precision gating enables dual-precision DNN execution at the granularity of each individual output feature, and therefore greatly reducing the average bitwidth and computational cost of DNNs.
Revised: Precision gating enables dual-precision DNN execution at the granularity of each individual output feature, and therefore greatly reducing the average bitwidth and computational cost of DNNs.

Original: To show the visualization of PG, we refer readers to Appendix A.1.
Revised: To show the visualization of PG, we refer readers to Appendix A.1.

Original: We further introduce a differentiable gating logic which makes the technique applicable to a greater variety of network models.
Revised: We further introduce a differentiable gating logic which makes the technique applicable to a greater variety of network models.

Original: Experimental results show that precision gating achieves signiﬁcant speed-up and accuracy improvement on both CNNs and LSTMs.
Revised: Experimental results show that precision gating achieves signiﬁcant speed-up and accuracy improvement on both CNNs and LSTMs.

Original: Compared to the baseline CNN counterparts, PG obtains 3.5% and 0.6% higher classiﬁcation accuracy with up to 4.5 × and 2.4 × less computation cost for CIFAR10 and ImageNet, respectively.
Revised: Compared to the baseline CNN counterparts, PG obtains 3.5% and 0.6% higher classiﬁcation accuracy with up to 4.5 × and 2.4 × less computation cost for CIFAR10 and ImageNet, respectively.

Original: On LSTM, compared to 8-bit uniform quantization PG boosts perplexity per word (PPW) by 1.2% with 2.8 × less compute on the Penn
Revised: On LSTM, compared to 8-bit uniform quantization PG boosts perplexity per word (PPW) by 1.2% with 2.8 × less compute on the Penn

Original: Tree Bank (PTB) dataset.
Revised: Tree Bank (PTB) dataset.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: We propose precision gating, which to the best of our knowledge is the ﬁrst end-to-end trainable method that enables dual-precision execution of DNNs.
Revised: We propose precision gating, which to the best of our knowledge is the ﬁrst end-to-end trainable method that enables dual-precision execution of DNNs.

Original: Precision gating is also applicable to a wide variety of network architectures (including both CNNs and RNNs).
Revised: Precision gating is also applicable to a wide variety of network architectures (including both CNNs and RNNs).

Original: Precision gating enables DNN computation with lower average bitwidth than other state-ofthe-art quantization methods.
Revised: Precision gating enables DNN computation with lower average bitwidth than other state-ofthe-art quantization methods.

Original: Combined with its lightweight gating logic, PG demonstrates the potential to reduce DNN execution costs in both commodity and dedicated hardware.
Revised: Combined with its lightweight gating logic, PG demonstrates the potential to reduce DNN execution costs in both commodity and dedicated hardware.

Original: Unlike prior works that focus only on inference, precision gating achieves the same sparsity during back-propagation as forward propagation, which dramatically reduces the computational cost for both passes.
Revised: Unlike prior works that focus only on inference, precision gating achieves the same sparsity during back-propagation as forward propagation, which dramatically reduces the computational cost for both passes.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Quantizing activations.
Revised: Quantizing activations.

Original: Prior studies show that weights can be quantized to low bitwidth without hurting much the accuracy (Zhu et al., 2017), but quantizing activations to low bitwidth typically incurs a large accuracy degradation.
Revised: Prior studies show that weights can be quantized to low bitwidth without hurting much the accuracy (Zhu et al., 2017), but quantizing activations to low bitwidth typically incurs a large accuracy degradation.

Original: The problem is that large outliers in the activations make the quantization grid sparse under low precision, thus increasing the error.
Revised: The problem is that large outliers in the activations make the quantization grid sparse under low precision, thus increasing the error.

Original: To address the problem,
Revised: To address the problem,

Original: Choi et al.
Revised: Choi et al.

Original: propose PACT to reduce the dynamic range of the activations through clipping the outliers by a learnable threshold, which provides a better quantization scheme for ultra low bitwidth.
Revised: propose PACT to reduce the dynamic range of the activations through clipping the outliers by a learnable threshold, which provides a better quantization scheme for ultra low bitwidth.

Original: PACT is reported to be able to quantize activations to arbitrary bitwidth with better accuracy than other works.
Revised: PACT is reported to be able to quantize activations to arbitrary bitwidth with better accuracy than other works.

Original: We incorporate PG with PACT to address the same large outlier problem.
Revised: We incorporate PG with PACT to address the same large outlier problem.

Original: Moreover,
Revised: Moreover,

Original: PG is a dynamic quantization scheme whereas PACT is a static apporach.
Revised: PG is a dynamic quantization scheme whereas PACT is a static apporach.

Original: Prediction-based execution.
Revised: Prediction-based execution.

Original: To lower the computation cost of a convolutional layer, some prior works explored the method of predicting ReLU-induced zeros or Max-pooling-induced compute redundancy in CNNs.
Revised: To lower the computation cost of a convolutional layer, some prior works explored the method of predicting ReLU-induced zeros or Max-pooling-induced compute redundancy in CNNs.

Original: For example, Lin et al.
Revised: For example, Lin et al.

Original: ; Song et al.
Revised: ; Song et al.

Original: (2018) propose zero-prediction to utilize a few most-signiﬁcant bits in multipliers to identify the sign of the output elements.
Revised: (2018) propose zero-prediction to utilize a few most-signiﬁcant bits in multipliers to identify the sign of the output elements.

Original: The rationale behind is that negative outputs will be suppressed by the ReLU anyway.
Revised: The rationale behind is that negative outputs will be suppressed by the ReLU anyway.

Original: The limitations of this method are that it only applies to ReLU activations and only when the ReLU must follow directly after the weight layer.
Revised: The limitations of this method are that it only applies to ReLU activations and only when the ReLU must follow directly after the weight layer.

Original: As a result, the method cannot be applied to modern CNNs with a batch norm layer (Ioffe & Szegedy, 2015) before the ReLU or RNNs (which use sigmoid or tanh as the activation function).
Revised: As a result, the method cannot be applied to modern CNNs with a batch norm layer (Ioffe & Szegedy, 2015) before the ReLU or RNNs (which use sigmoid or tanh as the activation function).

Original: Cao et al.
Revised: Cao et al.

Original: (2019) propose SeerNet in which for each convolutional layer it ﬁrst executes a duplicated quantized version of that layer to predict the output sparsity induced by the following ReLU or Max-pooling.
Revised: (2019) propose SeerNet in which for each convolutional layer it ﬁrst executes a duplicated quantized version of that layer to predict the output sparsity induced by the following ReLU or Max-pooling.

Original: It then computes the sparse original full-precision (32bit ﬂoating-point) convolutional layer to obtain the outputs.
Revised: It then computes the sparse original full-precision (32bit ﬂoating-point) convolutional layer to obtain the outputs.

Original: Results of the quantized layer are not reused.
Revised: Results of the quantized layer are not reused.

Original: Though the original layer is executed sparsely, practically the extra quantized layer induces higher execution cost.
Revised: Though the original layer is executed sparsely, practically the extra quantized layer induces higher execution cost.

Original: Feature level precision tuning.
Revised: Feature level precision tuning.

Original: There is also prior work that uses different precision at the feature level.
Revised: There is also prior work that uses different precision at the feature level.

Original: Park et al.
Revised: Park et al.

Original: propose value-aware quantization where the majority of data are computed at reduced precision while a small number of outliers are handled at high precision.
Revised: propose value-aware quantization where the majority of data are computed at reduced precision while a small number of outliers are handled at high precision.

Original: Our approach is signiﬁcantly different because we apply dual precision for every feature, not only for the outliers.
Revised: Our approach is signiﬁcantly different because we apply dual precision for every feature, not only for the outliers.

Original: Hua et al.
Revised: Hua et al.

Original: (2018) propose channel gating to dynamically turn off a subset of channels that contribute little to the classiﬁcation result.
Revised: (2018) propose channel gating to dynamically turn off a subset of channels that contribute little to the classiﬁcation result.

Original: Precision gating is orthogonal to this pruning technique as channel gating executes the whole network in the same precision.
Revised: Precision gating is orthogonal to this pruning technique as channel gating executes the whole network in the same precision.

Original: Input Activations   ℎ  
Revised: Input Activations   ℎ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ℎ
Revised: ℎ

Original: [Equation]
Revised: [Equation]

Original: 
Revised:

Original: [Equation]
Revised: [Equation]

Original: (a) - +  ℎ  ℎ
Revised: (a) - +  ℎ  ℎ

Original: H i gh - p r e c
Revised: H i gh - p r e c

Original: i s i on
Revised: i s i on

Original: Precision Gating on CNN
Revised: Precision Gating on CNN

Original: Precision Gating on RNN - +
Revised: Precision Gating on RNN - +

Original: H i gh - p r e c
Revised: H i gh - p r e c

Original: i s i on Δ Δ
Revised: i s i on Δ Δ

Original: Lo w - p r e c i s i on
Revised: Lo w - p r e c i s i on

Original: Lo w - p r e c i s i on (b)
Revised: Lo w - p r e c i s i on (b)

Original:  P RECISION G ATING
Revised:  P RECISION G ATING

Original: In this section, we ﬁrst describe the basic mechanism of precision gating.
Revised: In this section, we ﬁrst describe the basic mechanism of precision gating.

Original: Then we discuss how to design the gating logic to accelerate both forward and backward passes.
Revised: Then we discuss how to design the gating logic to accelerate both forward and backward passes.

Original: Finally, we incorporate PG with outlier clipping to reduce the quantization error.
Revised: Finally, we incorporate PG with outlier clipping to reduce the quantization error.

Original: 3. B ASIC F ORMULATION
Revised: 3. B ASIC F ORMULATION

Original: Deﬁne a linear layer (i.e. convolutional or fully-connected layers) in a neural network as O =
Revised: Deﬁne a linear layer (i.e. convolutional or fully-connected layers) in a neural network as O =

Original: I ∗ W , where O , I , and W are the output, input, and weights, respectively.
Revised: I ∗ W , where O , I , and W are the output, input, and weights, respectively.

Original: Let I be represented in a B -bit ﬁxed-point format.
Revised: Let I be represented in a B -bit ﬁxed-point format.

Original: As shown in Figure 1a, precision gating partitions I into I hb , the most-signiﬁcant
Revised: As shown in Figure 1a, precision gating partitions I into I hb , the most-signiﬁcant

Original: B hb bits, and I lb , the remaining B lb bits, where B = B hb + B lb .
Revised: B hb bits, and I lb , the remaining B lb bits, where B = B hb + B lb .

Original: More formally, we can write:
Revised: More formally, we can write:

Original: [Equation]
Revised: [Equation]

Original: Here << is the shift operator.
Revised: Here << is the shift operator.

Original: We can then reformulate a single B -bit linear layer into two lowerprecision computations as:
Revised: We can then reformulate a single B -bit linear layer into two lowerprecision computations as:

Original: [Equation]
Revised: [Equation]

Original: O hb is the partial sum obtained by using the MSBs of input feature ( I hb ) whereas O lb represents the remaining sum computed with I lb .
Revised: O hb is the partial sum obtained by using the MSBs of input feature ( I hb ) whereas O lb represents the remaining sum computed with I lb .

Original: Precision gating works in two phases.
Revised: Precision gating works in two phases.

Original: In the prediction phase , precision gating performs the computation O hb = W ∗ ( I hb << B lb ) .
Revised: In the prediction phase , precision gating performs the computation O hb = W ∗ ( I hb << B lb ) .

Original: Elements of O hb greater than a gating threshold ∆ are considered important.
Revised: Elements of O hb greater than a gating threshold ∆ are considered important.

Original: In the update phase , precision gating computes O lb =
Revised: In the update phase , precision gating computes O lb =

Original: W ∗ I lb only for the important elements and adds it to O hb .
Revised: W ∗ I lb only for the important elements and adds it to O hb .

Original: Precision gating’s execution ﬂow is shown in Figure 2: unimportant output features take only the upper path while important ones are computed as the sum of both paths.
Revised: Precision gating’s execution ﬂow is shown in Figure 2: unimportant output features take only the upper path while important ones are computed as the sum of both paths.

Original: Precision gating can be summarized as:
Revised: Precision gating can be summarized as:

Original: [Equation]
Revised: [Equation]

Original: In essence, precision gating computes important features with B -bit arithmetic and unimportant features with only B hb bits.
Revised: In essence, precision gating computes important features with B -bit arithmetic and unimportant features with only B hb bits.

Original: The importance of each element in the output O is predicted by comparing the magnitude of its partial sum O hb to ∆ .
Revised: The importance of each element in the output O is predicted by comparing the magnitude of its partial sum O hb to ∆ .

Original: Let p be the percentage of important activations over all features.
Revised: Let p be the percentage of important activations over all features.

Original: PG saves (1 − p ) · B lb B fraction of the compute in the original DNN model.
Revised: PG saves (1 − p ) · B lb B fraction of the compute in the original DNN model.

Original: Figure 1b illustrates which values are executed in high-precision.
Revised: Figure 1b illustrates which values are executed in high-precision.

Original: PG thus achieves dual-precision execution using a lightweight gating mechanism, and reduces the average bitwidth of computations in a DNN.
Revised: PG thus achieves dual-precision execution using a lightweight gating mechanism, and reduces the average bitwidth of computations in a DNN.

Original: 3. E FFICIENT L EARNABLE G ATING L OGIC
Revised: 3. E FFICIENT L EARNABLE G ATING L OGIC

Original: In order to automatically determine model and dataset speciﬁc threshold ∆ , we propose an efﬁcient end-to-end trainable gating logic.
Revised: In order to automatically determine model and dataset speciﬁc threshold ∆ , we propose an efﬁcient end-to-end trainable gating logic.

Original: The gating threshold ∆ is speciﬁc to each output channel in DNN
Revised: The gating threshold ∆ is speciﬁc to each output channel in DNN

Original: [Figure]
Revised: [Figure]

Original: weights
Revised: weights

Original: [Figure]
Revised: [Figure]

Original: layers and learned alongside the DNN weights during training.
Revised: layers and learned alongside the DNN weights during training.

Original: A larger ∆ indicates that more output features are computed in low-precision, resulting in greater computational savings but possibly at the expense of reduced model accuracy.
Revised: A larger ∆ indicates that more output features are computed in low-precision, resulting in greater computational savings but possibly at the expense of reduced model accuracy.

Original: We formulate the problem of optimizing ∆ as minimizing the original model loss L along with an L2 penalty term:
Revised: We formulate the problem of optimizing ∆ as minimizing the original model loss L along with an L2 penalty term:

Original: [Equation]
Revised: [Equation]

Original: Here y is the ground truth label, σ is the penalty factor, and δ is the gating target , a target value for the learnable threshold.
Revised: Here y is the ground truth label, σ is the penalty factor, and δ is the gating target , a target value for the learnable threshold.

Original: The penalty factor and gating target are hyperparameters which allow a user to emphasize high computation savings (large σ or δ ) or accuracy preservation (small σ or δ ).
Revised: The penalty factor and gating target are hyperparameters which allow a user to emphasize high computation savings (large σ or δ ) or accuracy preservation (small σ or δ ).

Original: Training a model with precision gating can be done on commodity GPUs using existing deep learning frameworks.
Revised: Training a model with precision gating can be done on commodity GPUs using existing deep learning frameworks.

Original: We implement PG on GPU as the equation O = O hb + mask (cid:12)
Revised: We implement PG on GPU as the equation O = O hb + mask (cid:12)

Original: O lb , where mask =
Revised: O lb , where mask =

Original: 1 O hb > ∆ is a binary decision mask and (cid:12) represents element-wise multiplication.
Revised: 1 O hb > ∆ is a binary decision mask and (cid:12) represents element-wise multiplication.

Original: During forward propagation, most elements in mask are 0.
Revised: During forward propagation, most elements in mask are 0.

Original: PG therefore saves hardware execution cost by only computing a sparse O lb in the update phase.
Revised: PG therefore saves hardware execution cost by only computing a sparse O lb in the update phase.

Original: In dedicated hardware, MSBs and LSBs are wired separately.
Revised: In dedicated hardware, MSBs and LSBs are wired separately.

Original: The prediction phase is controlled by a multiplexer, and only computes LSB convolutions while O hb exceeds the threshold, thus achieving the savings.
Revised: The prediction phase is controlled by a multiplexer, and only computes LSB convolutions while O hb exceeds the threshold, thus achieving the savings.

Original: The mask is computed using a binary decision function (i.e. step function), which has a gradient of zero almost everywhere.
Revised: The mask is computed using a binary decision function (i.e. step function), which has a gradient of zero almost everywhere.

Original: To let gradients ﬂow through mask to ∆ , we use a sigmoid on the backward pass to approximate the step.
Revised: To let gradients ﬂow through mask to ∆ , we use a sigmoid on the backward pass to approximate the step.

Original: Speciﬁcally, we deﬁne mask
Revised: Speciﬁcally, we deﬁne mask

Original: = sigmoid ( α ( O hb − ∆)) only on the backward pass following Hua et al.
Revised: = sigmoid ( α ( O hb − ∆)) only on the backward pass following Hua et al.

Original: Here α changes the slope of the sigmoid, thus controlling the magnitude of the gradients.
Revised: Here α changes the slope of the sigmoid, thus controlling the magnitude of the gradients.

Original: 3. S PARSE B ACK -P ROPAGATION
Revised: 3. S PARSE B ACK -P ROPAGATION

Original: A sparse update phase only reduces the computational cost during the inference (forward propagation).
Revised: A sparse update phase only reduces the computational cost during the inference (forward propagation).

Original: We further propose to save the compute during the back-propagation by modifying the forward function of the PG block.
Revised: We further propose to save the compute during the back-propagation by modifying the forward function of the PG block.

Original: Speciﬁcally, we square the mask element-wise.
Revised: Speciﬁcally, we square the mask element-wise.

Original: [Equation]
Revised: [Equation]

Original: Given that mask is a binary tensor, mask 2 in Eq.
Revised: Given that mask is a binary tensor, mask 2 in Eq.

Original: (5) preserves the same value as mask .
Revised: (5) preserves the same value as mask .

Original: Thus the forward pass remains unchanged.
Revised: Thus the forward pass remains unchanged.

Original: During the back-propagation, an additional mask term is introduced in computing the gradient of O with respect to the gating threshold ∆ in Eq.
Revised: During the back-propagation, an additional mask term is introduced in computing the gradient of O with respect to the gating threshold ∆ in Eq.

Original: (6) because of the element-wise square.
Revised: (6) because of the element-wise square.

Original: Consequently, the update of ∆ only requires the result of mask (cid:12) O lb which has already been computed during the forward pass.
Revised: Consequently, the update of ∆ only requires the result of mask (cid:12) O lb which has already been computed during the forward pass.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The gradient of O with respect to the weights W in Eq.
Revised: The gradient of O with respect to the weights W in Eq.

Original: (7) employs the same sparse mask (cid:12)
Revised: (7) employs the same sparse mask (cid:12)

Original: O lb as the update of ∆ .
Revised: O lb as the update of ∆ .

Original: Therefore, precision gating can reduce the computational cost of both forwardprogagtion (inference) and back-propagation by the same factor.
Revised: Therefore, precision gating can reduce the computational cost of both forwardprogagtion (inference) and back-propagation by the same factor.

Original: [Equation]
Revised: [Equation]

Original: 3. O UTLIER C LIPPING
Revised: 3. O UTLIER C LIPPING

Original: Precision gating identiﬁes important features using low-precision computation.
Revised: Precision gating identiﬁes important features using low-precision computation.

Original: One difﬁculty with this is that DNN activations are distributed in a bell curve, with most values close to zero and a few large outliers.
Revised: One difﬁculty with this is that DNN activations are distributed in a bell curve, with most values close to zero and a few large outliers.

Original: The top row of Figure 3(a) shows some activations in blue dots, including a single outlier.
Revised: The top row of Figure 3(a) shows some activations in blue dots, including a single outlier.

Original: If we quantize each value to 4 bits (second column) and use 2 most-signiﬁcant bits in the prediction phase (third column), we see that almost all values are rounded to zero.
Revised: If we quantize each value to 4 bits (second column) and use 2 most-signiﬁcant bits in the prediction phase (third column), we see that almost all values are rounded to zero.

Original: In this case, PG can only distinguish the importance between the single outlier and the rest of the values no matter what ∆ is.
Revised: In this case, PG can only distinguish the importance between the single outlier and the rest of the values no matter what ∆ is.

Original: Thus the presence of large outliers greatly reduces precision gating’s effectiveness.
Revised: Thus the presence of large outliers greatly reduces precision gating’s effectiveness.

Original: To address this, we combine precision gating with PACT (Choi et al., 2018), which clips each layer’s outputs using a learnable clip threshold.
Revised: To address this, we combine precision gating with PACT (Choi et al., 2018), which clips each layer’s outputs using a learnable clip threshold.

Original: The bottom row of Figure 3 shows how clipping limits the dynamic range of activations, making values more uniformly distributed along the quantization grid.
Revised: The bottom row of Figure 3 shows how clipping limits the dynamic range of activations, making values more uniformly distributed along the quantization grid.

Original: Now the 2 most-signiﬁcant bits can effectively separate out different groups of values based on magnitude.
Revised: Now the 2 most-signiﬁcant bits can effectively separate out different groups of values based on magnitude.

Original: We apply PACT to precision gating in CNNs, which typically use an unbounded activation function (ReLU).
Revised: We apply PACT to precision gating in CNNs, which typically use an unbounded activation function (ReLU).

Original: RNNs typically employ a bounded activation function (tanh, sigmoid), making PACT unnecessary.
Revised: RNNs typically employ a bounded activation function (tanh, sigmoid), making PACT unnecessary.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test precision gating using ResNet-18 (He et al., 2015) and ShiftNet-20 (Wu et al., 2017) on
Revised: We test precision gating using ResNet-18 (He et al., 2015) and ShiftNet-20 (Wu et al., 2017) on

Original: CIFAR-10 (Krizhevsky & Hinton, 2009), and ShufﬂeNet V2 (Ma et al., 2018) on ImageNet (Deng et al., 2009).
Revised: CIFAR-10 (Krizhevsky & Hinton, 2009), and ShufﬂeNet V2 (Ma et al., 2018) on ImageNet (Deng et al., 2009).

Original: ResNet is a very popular CNN for image classiﬁcation, and the other two lightweight architectures are designed speciﬁcally for edge devices.
Revised: ResNet is a very popular CNN for image classiﬁcation, and the other two lightweight architectures are designed speciﬁcally for edge devices.

Original: We set the expansion rate of ShiftNet to be six and choose the 0.5 × variant of ShfﬂeNet V2 for all experiments.
Revised: We set the expansion rate of ShiftNet to be six and choose the 0.5 × variant of ShfﬂeNet V2 for all experiments.

Original: On CIFAR-10, the batch size is 128, and the models are trained for 200 epochs.
Revised: On CIFAR-10, the batch size is 128, and the models are trained for 200 epochs.

Original: The initial learning rate is 0.1 and decayed at epoch 100, 150, 200 by a factor of 0.1 (multiply learning rate by 0.1).
Revised: The initial learning rate is 0.1 and decayed at epoch 100, 150, 200 by a factor of 0.1 (multiply learning rate by 0.1).

Original: On ImageNet, the batch size isand the models are trained for 120 epochs.
Revised: On ImageNet, the batch size isand the models are trained for 120 epochs.

Original: The learning rate decayed linearly from an initial value of 0.5 to 0.
Revised: The learning rate decayed linearly from an initial value of 0.5 to 0.

Original: We also test an LSTM (Hochreiter & Schmidhuber, 1997) on the Penn Tree Bank (PTB) (Marcus et al., 1993) corpus.
Revised: We also test an LSTM (Hochreiter & Schmidhuber, 1997) on the Penn Tree Bank (PTB) (Marcus et al., 1993) corpus.

Original: The model accuracy is measured by perplexity per word (PPW), and lower is better.
Revised: The model accuracy is measured by perplexity per word (PPW), and lower is better.

Original: Following the conﬁguration used by He et al.
Revised: Following the conﬁguration used by He et al.

Original: (2016) and Hubara et al.
Revised: (2016) and Hubara et al.

Original: (2017), the number of hidden units in the LSTM cell is set to 300, and the number of layers is set to 1.
Revised: (2017), the number of hidden units in the LSTM cell is set to 300, and the number of layers is set to 1.

Original: We follow the same training setting as described in He et al.
Revised: We follow the same training setting as described in He et al.

Original: (2016), except the learning rate decayed by a factor of 0.1 at epoch 50 and 90.
Revised: (2016), except the learning rate decayed by a factor of 0.1 at epoch 50 and 90.

Original: All experiments are run on Tensorﬂow (Abadi et al., 2016) with NVIDIA
Revised: All experiments are run on Tensorﬂow (Abadi et al., 2016) with NVIDIA

Original: GeForce 1080Ti GPUs.
Revised: GeForce 1080Ti GPUs.

Original: We report the top-1 accuracy for all experiments.
Revised: We report the top-1 accuracy for all experiments.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We replace the conv layers in CNNs and FC layers in LSTM with the proposed PG block.
Revised: We replace the conv layers in CNNs and FC layers in LSTM with the proposed PG block.

Original: Moreover, the following hyperparameters in PG need to be tuned appropriately to achieve low average bitwidth with high accuracy.
Revised: Moreover, the following hyperparameters in PG need to be tuned appropriately to achieve low average bitwidth with high accuracy.

Original: • The full bitwidth B – this represents the bitwidth for high-precision computation in PG.
Revised: • The full bitwidth B – this represents the bitwidth for high-precision computation in PG.

Original: B is set to 5 or less on CIFAR-10, 5 or 6 on ImageNet, and 3 or 4 on PTB.
Revised: B is set to 5 or less on CIFAR-10, 5 or 6 on ImageNet, and 3 or 4 on PTB.

Original: • The prediction bitwidth B hb – this represents the bitwidth for low-precision computation.
Revised: • The prediction bitwidth B hb – this represents the bitwidth for low-precision computation.

Original: • The penalty factor σ – this is the scaling factor of the L2 loss for gating thresholds ∆ .
Revised: • The penalty factor σ – this is the scaling factor of the L2 loss for gating thresholds ∆ .

Original: • The gating target δ – the target gating threshold.
Revised: • The gating target δ – the target gating threshold.

Original: We use a variety of values δ ∈ [ − 1 . 0 , 5 . 0] in our experiments.
Revised: We use a variety of values δ ∈ [ − 1 . 0 , 5 . 0] in our experiments.

Original: • The coefﬁcient α in the backward pass – α controls the magnitude of gradients ﬂowing to ∆ .
Revised: • The coefﬁcient α in the backward pass – α controls the magnitude of gradients ﬂowing to ∆ .

Original: We set α to be 5 across all experiments.
Revised: We set α to be 5 across all experiments.

Original: 4. CNN R ESULTS
Revised: 4. CNN R ESULTS

Original: To compare hardware execution efﬁciency across different techniques, we compute the average bitwidth ( B avg ) of all features in a DNN model:
Revised: To compare hardware execution efﬁciency across different techniques, we compute the average bitwidth ( B avg ) of all features in a DNN model:

Original: [Equation]
Revised: [Equation]

Original: where p is the percentage of high-precision activations (i.e. number of important features divided by total features).
Revised: where p is the percentage of high-precision activations (i.e. number of important features divided by total features).

Original: The computational cost of DNNs is proportional to the average bitwidth.
Revised: The computational cost of DNNs is proportional to the average bitwidth.

Original: Our results on CNNs are presented in Table 1 and Table 2, where sp. stands for sparsity.
Revised: Our results on CNNs are presented in Table 1 and Table 2, where sp. stands for sparsity.

Original: We ﬁrst compare PG against two widely adopted quantization schemes — uniform quantization (UQ) and PACT (Choi et al., 2018).
Revised: We ﬁrst compare PG against two widely adopted quantization schemes — uniform quantization (UQ) and PACT (Choi et al., 2018).

Original: In Table 1, the 3rd and 4th columns show the average bitwidth and model accuracy of PG whereas the 5th–8th columns list the bitwidth and corresponding model accuracy of UQ and PACT.
Revised: In Table 1, the 3rd and 4th columns show the average bitwidth and model accuracy of PG whereas the 5th–8th columns list the bitwidth and corresponding model accuracy of UQ and PACT.

Original: At each row PG achieves better accuracy with fewer average bitwidth (Bits vs. B avg ).
Revised: At each row PG achieves better accuracy with fewer average bitwidth (Bits vs. B avg ).

Original: Speciﬁcally, PG achieves 6.7% and 1.6% higher accuracy with 1.25 × computational cost reduction , and 6.6% and 0.5% higher accuracy with 1.54 × computational cost reduction than 2-bit UQ and PACT on ShiftNet-20 and ResNet-18 for CIFAR-10 (3rd & 6th rows), respectively.
Revised: Speciﬁcally, PG achieves 6.7% and 1.6% higher accuracy with 1.25 × computational cost reduction , and 6.6% and 0.5% higher accuracy with 1.54 × computational cost reduction than 2-bit UQ and PACT on ShiftNet-20 and ResNet-18 for CIFAR-10 (3rd & 6th rows), respectively.

Original: We observe the same trend on ShufﬂeNet for ImageNet where PG improves the accuracy of 5-bit UQ and PACT by 1.0% and 1.4% with 1.22 × computational cost reduction (9th row).
Revised: We observe the same trend on ShufﬂeNet for ImageNet where PG improves the accuracy of 5-bit UQ and PACT by 1.0% and 1.4% with 1.22 × computational cost reduction (9th row).

Original: It is also worth noting that PG can recover the accuracy of the ﬂoating-point ShiftNet-20, ResNet-18, and ShufﬂeNet with 3.9 , 3.2 , and 4.7 average bitwidth.
Revised: It is also worth noting that PG can recover the accuracy of the ﬂoating-point ShiftNet-20, ResNet-18, and ShufﬂeNet with 3.9 , 3.2 , and 4.7 average bitwidth.

Original: This demonstrates that PG, using a learnable threshold, can identify unimportant features and reduce bitwidth without compromising accuracy.
Revised: This demonstrates that PG, using a learnable threshold, can identify unimportant features and reduce bitwidth without compromising accuracy.

Original: We then compare PG with Fix-Threshold, which is an extension of zero-prediction (Lin et al., 2017;
Revised: We then compare PG with Fix-Threshold, which is an extension of zero-prediction (Lin et al., 2017;

Original: Song et al., 2018).
Revised: Song et al., 2018).

Original: Lin et al.
Revised: Lin et al.

Original: (2017) and Song et al.
Revised: (2017) and Song et al.

Original: (2018) explicitly predict ReLU-induced zeros during inference.
Revised: (2018) explicitly predict ReLU-induced zeros during inference.

Original: To have a fair comparison, we extend their technique to predict an arbitrary ﬁxed threshold to achieve the same or fewer B avg as PG, reported as Fix-Threshold in Table 1.
Revised: To have a fair comparison, we extend their technique to predict an arbitrary ﬁxed threshold to achieve the same or fewer B avg as PG, reported as Fix-Threshold in Table 1.

Original: For
Revised: For

Original: CIFAR-10, we observe that PG achieves 20.2% and 18.7% higher accuracy with 1.75 × and 1.38 × computational cost reduction than Fix-Threshold on ShiftNet-20 and ResNet-18 (3rd & 6th rows), respectively.
Revised: CIFAR-10, we observe that PG achieves 20.2% and 18.7% higher accuracy with 1.75 × and 1.38 × computational cost reduction than Fix-Threshold on ShiftNet-20 and ResNet-18 (3rd & 6th rows), respectively.

Original: The gap in accuracy becomes even larger on ShufﬂeNet V2 for ImageNet dataset.
Revised: The gap in accuracy becomes even larger on ShufﬂeNet V2 for ImageNet dataset.

Original: With the same or lower average bitwidth, the accuracy of PG is at least 26% higher than Fix-Threshold.
Revised: With the same or lower average bitwidth, the accuracy of PG is at least 26% higher than Fix-Threshold.

Original: In conclusion, PG consistently outperforms Fix-Threshold at any bitwidth across all networks and datasets (7th – 9th rows).
Revised: In conclusion, PG consistently outperforms Fix-Threshold at any bitwidth across all networks and datasets (7th – 9th rows).

Original: The reason is that PG learns to adjust the gating thresholds to the clipped activation distribution so that it predicts with less zero bits.
Revised: The reason is that PG learns to adjust the gating thresholds to the clipped activation distribution so that it predicts with less zero bits.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We further compare PG with SeerNet (Cao et al., 2019) in Table 2.
Revised: We further compare PG with SeerNet (Cao et al., 2019) in Table 2.

Original: SeerNet adds an extra quantized conv layer to predict a sparse mask for executing the original ﬂoating-point layer during inference.
Revised: SeerNet adds an extra quantized conv layer to predict a sparse mask for executing the original ﬂoating-point layer during inference.

Original: As the code of SeerNet is unavailable, we implement the network in Tensorﬂow by ourselves and boost its accuracy by retraining the network.
Revised: As the code of SeerNet is unavailable, we implement the network in Tensorﬂow by ourselves and boost its accuracy by retraining the network.

Original: We reduce the average bitwidth of SeerNet while keeping a comparable accuracy as PG.
Revised: We reduce the average bitwidth of SeerNet while keeping a comparable accuracy as PG.

Original: At similar model prediction accuracy, PG achieves 3.5% higher accuracy with 3.3 × and 4.5 × less compute on ResNet-18 and ShiftNet for CIFAR-10, respectively.
Revised: At similar model prediction accuracy, PG achieves 3.5% higher accuracy with 3.3 × and 4.5 × less compute on ResNet-18 and ShiftNet for CIFAR-10, respectively.

Original: For ImageNet, PG on ShufﬂeNet also achieves 0.4% higher accuracy with 2.4 × computational cost reduction than SeerNet.
Revised: For ImageNet, PG on ShufﬂeNet also achieves 0.4% higher accuracy with 2.4 × computational cost reduction than SeerNet.

Original: PG outpeforms SeerNet as SeerNet uses the extra quantized layer for prediction only and its outputs are sparse thus inaccurate, while PG reuses the low-precision results and passes them to outputs anyway.
Revised: PG outpeforms SeerNet as SeerNet uses the extra quantized layer for prediction only and its outputs are sparse thus inaccurate, while PG reuses the low-precision results and passes them to outputs anyway.

Original: To quantify the impact of sparse back-propagation described in Section 3.3, we run PG with and without sparse back-propagation on CNNs.
Revised: To quantify the impact of sparse back-propagation described in Section 3.3, we run PG with and without sparse back-propagation on CNNs.

Original: Table 3 compares the sparsity in the update phase of PG with and without back-propagation under the same model accuracy and average bitwidth.
Revised: Table 3 compares the sparsity in the update phase of PG with and without back-propagation under the same model accuracy and average bitwidth.

Original: Surprisingly, we ﬁnd that the sparsity in the update phase of PG with sparse back-propagation is consistently higher than that of PG without sparse back-propagation across all models and datasets.
Revised: Surprisingly, we ﬁnd that the sparsity in the update phase of PG with sparse back-propagation is consistently higher than that of PG without sparse back-propagation across all models and datasets.

Original: For both CIFAR-10 and ImageNet, the sparsity increment is between 6% and 21%.
Revised: For both CIFAR-10 and ImageNet, the sparsity increment is between 6% and 21%.

Original: We hypothesize that sparse back-propagation zeros out the gradients ﬂowing to non-activated LSB convolutions in the update phase, which leads to higher sparsity.
Revised: We hypothesize that sparse back-propagation zeros out the gradients ﬂowing to non-activated LSB convolutions in the update phase, which leads to higher sparsity.

Original: Overall, compared to other prediction based quantization schemes, our method has measurably up to 21.6% higher accuracy than the baseline with lower computational cost or up to 4.5 × less compute with the same or better accuracy for CIFAR-10.
Revised: Overall, compared to other prediction based quantization schemes, our method has measurably up to 21.6% higher accuracy than the baseline with lower computational cost or up to 4.5 × less compute with the same or better accuracy for CIFAR-10.

Original: For ImageNet ours has a maximum of 30.3% higher accuracy with similar computational cost or 2.4 × less computational cost with 0.4% higher accuracy.
Revised: For ImageNet ours has a maximum of 30.3% higher accuracy with similar computational cost or 2.4 × less computational cost with 0.4% higher accuracy.

Original: The results compared to quantization baselines are visualized in Figure 4, which plots accuracy vs. average bitwidth — uniform quantization (squares), PACT (triangles), and PG (circles) are shown on separate curves.
Revised: The results compared to quantization baselines are visualized in Figure 4, which plots accuracy vs. average bitwidth — uniform quantization (squares), PACT (triangles), and PG (circles) are shown on separate curves.

Original: Results closer to the upper-left corner are better.
Revised: Results closer to the upper-left corner are better.

Original: 4. LSTM R ESULTS
Revised: 4. LSTM R ESULTS

Original: Besides CNNs, PG works great on RNNs as well.
Revised: Besides CNNs, PG works great on RNNs as well.

Original: The results on LSTM with PG for the PTB corpus are reported in Table 4.
Revised: The results on LSTM with PG for the PTB corpus are reported in Table 4.

Original: Here, only the uniform quantization is used as the baseline because PACT,
Revised: Here, only the uniform quantization is used as the baseline because PACT,

Original: Fix-Threshold, and SeerNet do not work for sigmoid or tanh activation functions.
Revised: Fix-Threshold, and SeerNet do not work for sigmoid or tanh activation functions.

Original: Although Hubara et al.
Revised: Although Hubara et al.

Original: (2017) claim that quantizing both weights and activations to 4 bits does not lower PPW, we observe a signiﬁcant PPW degradation as B decreases from 8 to 4 bits in our implementation.
Revised: (2017) claim that quantizing both weights and activations to 4 bits does not lower PPW, we observe a signiﬁcant PPW degradation as B decreases from 8 to 4 bits in our implementation.

Original: The
Revised: The

Original: LSTM with 8-bit activations is therefore considered as the full accuracy model.
Revised: LSTM with 8-bit activations is therefore considered as the full accuracy model.

Original: We observe the same phenomena as in the CNN results that PG enables the 3-bit LSTM cell to improve the PPW by 1.2% and reduce the computational cost by 2.7 × compared to 8-bit uniform quantization.
Revised: We observe the same phenomena as in the CNN results that PG enables the 3-bit LSTM cell to improve the PPW by 1.2% and reduce the computational cost by 2.7 × compared to 8-bit uniform quantization.

Original: 4. K ERNEL S PEEDUP
Revised: 4. K ERNEL S PEEDUP

Original: The sparse update phase of PG can be implemented efﬁciently with a new kernel called sampled dense-dense matrix multiplication (SDDMM).
Revised: The sparse update phase of PG can be implemented efﬁciently with a new kernel called sampled dense-dense matrix multiplication (SDDMM).

Original: A convolutional layer with PG is then factorized into a regular low-precision convolution bounded with a low-precision SDDMM.
Revised: A convolutional layer with PG is then factorized into a regular low-precision convolution bounded with a low-precision SDDMM.

Original: To evaluate the actual speedup of PG, we implement the SDDMM kernel in Python leveraging a high performance JIT compiler Numba (Lam et al., 2015) and test it on the ResNet-18 model for CIFAR-10.
Revised: To evaluate the actual speedup of PG, we implement the SDDMM kernel in Python leveraging a high performance JIT compiler Numba (Lam et al., 2015) and test it on the ResNet-18 model for CIFAR-10.

Original: Table 6 shows the layer-wise sparsity and the kernel speedup compared to the dense baseline on Intel Xeon Silver 4114 CPU (2.20GHz).
Revised: Table 6 shows the layer-wise sparsity and the kernel speedup compared to the dense baseline on Intel Xeon Silver 4114 CPU (2.20GHz).

Original: With the large sparsity (from 76% to 99%) in each layer induced by PG, the SDDMM kernel achieves up to 8.3 × wall clock speedup over the general dense matrix-matrix multiplication (GEMM) kernel.
Revised: With the large sparsity (from 76% to 99%) in each layer induced by PG, the SDDMM kernel achieves up to 8.3 × wall clock speedup over the general dense matrix-matrix multiplication (GEMM) kernel.

Original: The signiﬁcant wall clock speedup shows an enormous potential of deploying PG on commodity hardware.
Revised: The signiﬁcant wall clock speedup shows an enormous potential of deploying PG on commodity hardware.

Original: [Table]
Revised: [Table]

Original: For a GPU implementation, we face the challenge in replacing the GEMM kernel with SDDMM kernel.
Revised: For a GPU implementation, we face the challenge in replacing the GEMM kernel with SDDMM kernel.

Original: Popular deep learning framework such as Tensorﬂow does not incorporate SDDMM as an built-in operator.
Revised: Popular deep learning framework such as Tensorﬂow does not incorporate SDDMM as an built-in operator.

Original: However, Nisa et al.
Revised: However, Nisa et al.

Original: (2018) demonstrate that a highly optimized SDDMM kernel with a similar sparsity shown in Table 6 achieves about 4 × speedup over a GEMM kernel.
Revised: (2018) demonstrate that a highly optimized SDDMM kernel with a similar sparsity shown in Table 6 achieves about 4 × speedup over a GEMM kernel.

Original: This shows strong evidence that PG has a potential to obtain high speedup on GPUs as well.
Revised: This shows strong evidence that PG has a potential to obtain high speedup on GPUs as well.

Original: Additionally, our approach is compatible with the specialized accelerator architectures proposed by Lin et al.
Revised: Additionally, our approach is compatible with the specialized accelerator architectures proposed by Lin et al.

Original: (2017) and Song et al.
Revised: (2017) and Song et al.

Original: Due to the enormous sparsity, DNNs with PG are estimated to get at least 3 × speedup and 5 × energy efﬁciency on these dedicated hardware accelerators.
Revised: Due to the enormous sparsity, DNNs with PG are estimated to get at least 3 × speedup and 5 × energy efﬁciency on these dedicated hardware accelerators.

Original: We leave the deployment of the SDDMM kernel on GPU and dedicated hardware to future work.
Revised: We leave the deployment of the SDDMM kernel on GPU and dedicated hardware to future work.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose precision gating, a dynamic dual-precision quantization method that can reduce the computation and storage costs of DNNs.
Revised: We propose precision gating, a dynamic dual-precision quantization method that can reduce the computation and storage costs of DNNs.

Original: PG assigns higher precision to important features and lower precision to the remaining features at run-time.
Revised: PG assigns higher precision to important features and lower precision to the remaining features at run-time.

Original: The technique is end-to-end trainable allowing individual models to learn to distinguish salient and non-salient features.
Revised: The technique is end-to-end trainable allowing individual models to learn to distinguish salient and non-salient features.

Original: Experimental results show that PG outperforms state-of-the-art quantization and prediction approaches by a large margin on both CNN and RNN benchmarks on datasets such as Imagenet and Penn Tree Bank.
Revised: Experimental results show that PG outperforms state-of-the-art quantization and prediction approaches by a large margin on both CNN and RNN benchmarks on datasets such as Imagenet and Penn Tree Bank.

Original: A A PPENDIX A. F EATURE V ISUALIZATION
Revised: A A PPENDIX A. F EATURE V ISUALIZATION

Original: In precision gating we expect that the model will learn to compute features whose prediction values exceeding the threshold in high-precision while keeping others computed at low precision.
Revised: In precision gating we expect that the model will learn to compute features whose prediction values exceeding the threshold in high-precision while keeping others computed at low precision.

Original: In the image recognition task we expect that high-precision features are mostly in the region where an object lies.
Revised: In the image recognition task we expect that high-precision features are mostly in the region where an object lies.

Original: To provide evidence that the precision gating can effectively learn to identity those regions, in Figure we visualize the decision maps extracted from the ﬁnal convolutional layer that is modiﬁed to support precision gating in the ResNet-18 model.
Revised: To provide evidence that the precision gating can effectively learn to identity those regions, in Figure we visualize the decision maps extracted from the ﬁnal convolutional layer that is modiﬁed to support precision gating in the ResNet-18 model.

Original: A decision map is a gray scale 2D image that has the same spatial size as the output feature map in the same convolutional layer.
Revised: A decision map is a gray scale 2D image that has the same spatial size as the output feature map in the same convolutional layer.

Original: The brighter a pixel is in the decision map, the more probably the same spatial location in the output feature map will be computed in high precision.
Revised: The brighter a pixel is in the decision map, the more probably the same spatial location in the output feature map will be computed in high precision.

Original: The ﬁrst row contains the original input images in
Revised: The ﬁrst row contains the original input images in

Original: CIFAR-10, and their corresponding decision maps are shown in the second row.
Revised: CIFAR-10, and their corresponding decision maps are shown in the second row.

Original: We can see clearly that in each decision map, the locations of bright pixels roughly align with the object in the original image, which is exactly what we expect.
Revised: We can see clearly that in each decision map, the locations of bright pixels roughly align with the object in the original image, which is exactly what we expect.

Revised: A. A DDITIONAL R ESULTS

Revised: We provide more supplementary results in this section.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: and ShufﬂeNet V2 0.5 × on ImageNet. ”

Revised: fp” is ﬂoating-point accuracy.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: (fp

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: (fp top-5 99.8%) 2/

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ShufﬂeNet V

Revised: [Table]

Revised: .5 × 6/

Revised: [Table]

Revised: (fp top-1 59.0%)

Revised: [Table]

Revised: [Table]

Revised: (fp

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]