Original: P RECISION G ATING : I MPROVING N EURAL N ETWORK E FFICIENCY WITH D YNAMIC D UAL -P RECISION A CTIVATIONS
Revised: P RECISION G ATING : I MPROVING N EURAL N ETWORK E FFICIENCY WITH D YNAMIC D UAL -P RECISION A CTIVATIONS

Original: Anonymous authors
Revised: @cornell edu rz252@cornell edu wh399@ edu

Original: Paper under double-blind review
Revised: Zhang

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose precision gating (PG), an end-to-end trainable  dual-precision quantization technique for deep neural networks.
Revised: We propose precision gating (PG), an end-to-end trainable dynamic dual-precision quantization technique for deep neural networks.

Original: PG computes most features in a low precision and only a small proportion of important features in a higher precision.
Revised: PG computes most features in a low precision and only a small proportion of important features in a higher precision to preserve accuracy.

Original: Precision gating isvery lightweight and widely applicable to many neural network architectures. Experimental results show that precision gating can greatly reduce the average bitwidth of computations in both CNNs and LSTMs with negligible accuracy loss. 
Revised: The proposed approach is applicable to a variety of DNN architectures and signiﬁcantly reduces the computational cost of DNN execution with almost no accuracy loss. Our experiments indicate that PG achieves excellent results on CNNs, including statically compressed mobile-friendly networks such as ShufﬂeNet.

Original: Compared to  state-of-the-art counterparts, PG achieves the same or better accuracy with 2.4 × less compute on ImageNet.
Revised: Compared to the state-of-the-art prediction-based quantization schemes, PG achieves the same or higher accuracy with 2.4 × less compute on ImageNet.

Revised: PG furthermore applies to RNNs.

Original: Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement in perplexity per word with 2.8 × computational cost reduction on LSTM on the Penn Tree Bank dataset.
Revised: Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement in perplexity per word with 2.7 × computational cost reduction on LSTM on the Penn Tree Bank dataset.

Original: Precision gating has the potential to greatly reduce the execution costs of DNNs on both commodity and dedicated hardware accelerators.

Original: We implement the sampled dense-dense matrix multiplication kernel in PG on CPU, which achieves up to 8.3 × wall clock speedup over the dense baseline.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, deep neural networks (DNNs) have demonstrated excellent performance on many computer vision and language modeling tasks including image classiﬁcation, semantic segmentation, face recognition, machine translation and image captioning (Krizhevsky et al., 2012;
Revised: In recent years, deep neural networks (DNNs) have demonstrated excellent performance on many computer vision and language modeling tasks such as image classiﬁcation, semantic segmentation, face recognition, machine translation, and image captioning (Krizhevsky et al., 2012;

Original: He et al., 2015; Ronneberger et al., 2015; Chen et al., 2016; Zhao et al., 2018; Schroff et al., 2015; Luong et al., 2015; Vaswani et al., 2017).
Revised: He et al., 2016a; Ronneberger et al., 2015; Chen et al., 2016; Zhao et al., 2018; Schroff et al., 2015; Luong et al., 2015; Vaswani et al., 2017).

Original: One visible trend in DNN design is that as researchers strive for better accuracy, both the model size and the number of DNN layers have  increased over time (Xu et al., 2018).
Revised: One evident trend in DNN design is that as researchers strive for better accuracy, both the model size and the number of DNN layers have drastically increased over time (Xu et al., 2018).

Original: At the same time, there is a growing demand to push deep learning technology to edge devices such as mobile phones, Internet of Things (IoT) devices, andlightweight drones (Wu et al., 2019).
Revised: At the same time, there is a growing demand to deploy deep learning technology in edge devices such as mobile phones, VR/AR glasses, and drones (Wu et al., 2019).

Original: The limited computational, memory, and energy budgets on these devices create major challenges for the deployment of large DNN models on the edge.
Revised: The limited computational, memory, and energy budgets on these devices impose major challenges for the deployment of large DNN models at the edge.

Original: Neural network quantization is an important technique for improving the hardware efﬁciency of DNN execution.
Revised: DNN quantization is an important technique for improving the hardware efﬁciency of DNN execution (Zhao et al., 2017).

Original: Circuits operating at a lower precision are smaller, faster, and consume less energy.

Original: Numerous studies have shown that full-precision ﬂoating-point computation is not necessary for DNN inference — quantized ﬁxed-point models produce competitive results with a small or zero loss in prediction accuracy (Lin et al., 2016;
Revised: Numerous studies have shown that full-precision ﬂoating-point computation is not necessary for DNN inference — quantized ﬁxed-point models produce competitive results with a small or zero loss in prediction accuracy (Lin et al., 2016;

Original: He et al., 2016; Zhou et al., 2016; 2017).
Revised: He et al., 2016b; Zhou et al., 2016; 2017).

Original: In some cases, quantization may even improve model generalization by acting as a form of regularization.
Revised: In some cases, quantization may even improve model generalization by acting as a form of regularization.

Original: Existing studies mainly focus on static quantization, in which the precision of each weight and activation is ﬁxed at run time (Hubara et al., 2017;
Revised: Existing studies mainly focus on static quantization, in which the precision of each weight and activation is ﬁxed prior to inference (Hubara et al., 2017;

Original: He et al., 2016).
Revised: He et al., 2016b).

Original: Along this line of work, researchers have explored tuning the bitwidth per layer (Wu et al., 2018; Wang et al., 2018a; Dong et al., 2019) as well as various types of quantization functions (Wang et al., 2018b; Courbariaux et al., 2016; Li et al., 2016; Zhou et al., 2016).
Revised: Along this line of work, researchers have explored tuning the bitwidth per layer (Wu et al., 2018b; Wang et al., 2019; Dong et al., 2019) as well as various types of quantization functions (Wang et al., 2018; Courbariaux et al., 2016; Li et al., 2016; Zhou et al., 2016).

Original: However, static DNN quantization methods cannot exploit input-dependent characteristics, where certain features  during inference may contribute less to the classiﬁcation result than others and can be computed in lower precision.
Revised: However, static DNN quantization methods cannot exploit input-dependent characteristics, where certain features can be computed in a lower precision during inference as they contribute less to the classiﬁcation result for the given input.

Original: For example, in computer vision tasks the pixels representing the object of interest are typically more important than the background pixels.
Revised: For example, in computer vision tasks, the pixels representing the object of interest are typically more important than the background pixels.

Original: In this paper, wepropose to reduce the inefﬁciency of a statically quantized DNN via precision gating (PG), which computes most features with low-precision arithmetic  and only updates few im- portant features to  high precision.
Revised: In this paper, we reduce the inefﬁciency of a statically quantized DNN via precision gating (PG), which computes most features with low-precision arithmetic operations and only updates few important features to a high precision.

Original: More concretely, PG ﬁrst executes a DNN layer in  low precision and identiﬁes the output features with large magnitude as important features.
Revised: More concretely, PG ﬁrst executes a DNN layer in a low precision and identiﬁes the output features with large magnitude as important features.

Original: It then computes a sparse update to increase the precision of those important output features.
Revised: It then computes a sparse update to increase the precision of those important output features.

Original: Intuitively, small values make very little contribution to the DNN’s output;  approximating them in low-precision is reasonable.
Revised: Intuitively, small values make a small contribution to the DNN’s output; thus approximating them in a low precision is reasonable.

Original: Precision gating enables dual-precision DNN execution at the granularity of each individual output feature, and therefore greatly reducing the average bitwidth and computational cost of DNNs.
Revised: Precision gating enables dual-precision DNN execution at the granularity of each individual output feature, and therefore greatly reducing the average bitwidth and computational cost of the DNN.

Original: To show the visualization of PG, we refer readers to Appendix A.1.

Original: We further introduce a differentiable gating logic which makes the technique applicable to a greater variety of network models.
Revised: We further introduce a differentiable gating function which makes PG applicable to a rich variety of network models.

Original: Experimental results show that precision gating achieves signiﬁcant speed-up and accuracy improvement on both CNNs and LSTMs.
Revised: Experimental results show that PG achieves signiﬁcant compute reduction and accuracy improvement on both CNNs and LSTMs.

Original: Compared to the baseline CNN counterparts, PG obtains 3.5% and 0.6% higher classiﬁcation accuracy with up to 4.5 × and 2.4 × less computation cost for CIFAR10 and ImageNet, respectively.
Revised: Compared to the baseline CNN counterparts, PG obtains 3.5% and 0.6% higher classiﬁcation accuracy with up to 4.5 × and 2.4 × less computational cost for CIFARand ImageNet, respectively.

Original: On LSTM, compared to 8-bit uniform quantization PG boosts perplexity per word (PPW) by 1.2% with 2.8 × less compute on the Penn Tree Bank (PTB) dataset.
Revised: On LSTM, compared to 8-bit uniform quantization PG boosts perplexity per word (PPW) by 1.2% with 2.8 × less compute on the Penn Tree Bank (PTB) dataset.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: We propose precision gating, which to the best of our knowledge is the ﬁrst end-to-end trainable method that enables dual-precision execution of DNNs.
Revised: We propose precision gating (PG), which to the best of our knowledge is the ﬁrst end-toend trainable method that enables dual-precision execution of DNNs.

Original: Precision gating isalso applicable to a wide variety of network architectures (including both CNNs and RNNs).
Revised: PG is applicable to a wide variety of CNN and LSTM models.

Original: Precision gating enables DNN computation with lower average bitwidth than other state-ofthe-art quantization methods.
Revised: PG enables DNN computation with lower average bitwidth than other state-of-the-art quantization methods.

Original: Combined with its lightweight gating logic, PG demonstrates the potential to reduce DNN execution costs in both commodity and dedicated hardware. 
Revised: By employing a low-cost gating scheme, PG has the potential to reduce DNN execution costs in both commodity and dedicated hardware. PG achieves

Original: Unlike prior works that focus only on inference, precision gating achieves the same sparsity during back-propagation as forward propagation, which dramatically reduces the computational cost for both passes. 
Revised: the same sparsity during back-propagation as forward propagation, which dramatically reduces the computational cost for both passes. This is in stark contrast to prior dynamic DNN optimization methods that focus only on reducing the inference cost.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Quantizing activations.
Revised: Quantizing activations.

Original: Prior studies show that weights can be quantized to low bitwidth without hurting muchthe accuracy (Zhu et al., 2017), but quantizing activations to low bitwidth  typically incurs a large accuracy degradation.
Revised: Prior studies show that weights can be quantized to low bitwidth without compromising much accuracy (Zhu et al., 2017); however, quantizing activations with a low bitwidth (e.g., 4 bits) typically incurs a nontrivial accuracy degradation (Mishra et al., 2018; Zhou et al., 2016).

Original: The problem is that large outliers in the activations make the quantization grid  sparse under  low precision, thus increasing the error.
Revised: This is partially caused by large activation and weight outliers, which stretch the quantization grid too wide and too sparse under a low precision, thus increasing the error (Park et al., 2018; Zhao et al., 2019).

Original: To address the problem, Choi et al.
Revised: To address this problem, Choi et al.

Original: propose PACT to reduce the dynamic range of the activations through clipping the outliers by a learnable threshold, which provides a better quantization scheme for ultra lowbitwidth. PACT is reported to be able to quantize activations to arbitrary bitwidth with better accuracy than other works.
Revised: propose PACT to reduce the dynamic range of the activations through clipping the outliers using a learnable threshold. PACT provides a more effective quantization scheme under a very low bitwidth (e.g., 4 bits).

Original: We incorporate PG with PACT  to address thesame large outlier problem. Moreover, PG is a dynamic quantization scheme whereas PACT is a static apporach.
Revised: In this work we incorporate the PACT method in the training ﬂow of PG to handle the large outliers.

Original: Prediction-based execution.
Revised: Prediction-based execution.

Original: To lower the computation cost of a convolutional layer, some prior works  exploredthe method of predicting ReLU-induced zeros or Max-pooling-induced compute redundancy in CNNs.
Revised: Prior works have explored predicting ReLU-induced zeros and maxpooling compute redundancy to lower the computational cost of CNNs.

Original: For example, Lin et al.
Revised: For example, Lin et al.

Original: ; Song et al.
Revised: (2017); Song et al.

Original: (2018) propose zero-prediction to utilize a few  most-signiﬁcant bits in multipliers to identify the sign of the output elements.
Revised: (2018) propose zero-prediction to utilize a few of the most-signiﬁcant bits in the input activations to predict the sign of the output activation.

Original: The rationale behind is that negative outputs will be suppressed by the ReLU anyway.
Revised: Zero-prediction removes redundancy by exploiting the fact that negative outputs will be suppressed by the ReLU anyway.

Original: The limitations of this methodare that it only applies to ReLU activations and only when the ReLU must follow directly after the weight layer.
Revised: Yet, this method only applies to ReLU activations and only when a linear layer is directly followed by ReLU.

Original: As a result, the method cannot be applied tomodern CNNs with a batch norm layer (Ioffe & Szegedy, 2015) before the ReLU or RNNs (which use sigmoid or tanh as the activation function).
Revised: Hence such method does not apply to RNNs that use sigmoid or tanh as the activation function and many modern CNN networks that use batch normalization (Ioffe & Szegedy, 2015) before ReLU.

Original: Cao et al.
Revised: Hua et al.

Revised: (2019a;b) propose channel gating to dynamically turn off a subset of channels that contribute little to the model prediction result.

Revised: Precision gating is orthogonal to this pruning technique as channel gating executes the whole network at the same precision.

Revised: More recently, Cao et al.

Original: (2019) propose SeerNet in which for each convolutional layer it ﬁrst executes aduplicated quantized version of that layer  to predict the output sparsity induced by thefollowing ReLU or Max-pooling.
Revised: (2019) propose SeerNet , which also executes a CNN model in dual precision. For each convolutional layer, SeerNet ﬁrst executes a quantized version of the layer and uses the results to predict the output sparsity induced by the ReLU or the computational sparsity induced by the max-pooling layer.

Original: It then computes thesparse original  full-precision (32bit ﬂoating-point) convolutional layer to obtain the outputs.
Revised: For those activations that are not suppressed according to the prediction, SeerNet computes the original convolution in full-precision (32-bit ﬂoat). One key dif- [Figure] [Figure]

Original: Results of the quantized layer are not reused.
Revised: [Figure] << [Figure] ference between PG and SeerNet is that PG reuses the result of the low-precision compute as a partial product when it performs the high-precision multiplication in the update phase.

Original: Though the original layer is executed sparsely, practically theextra quantized layer induces higher execution cost.
Revised: In contrast, the full-precision compute in SeerNet does not reuse the output from the quantized layer, which incurs a higher execution cost.

Original: Feature level precision tuning.
Revised: Feature-level precision tuning.

Original: There is also prior work that uses  different precision at the feature level.
Revised: There is also prior work that uses a different precision to handle the outlier quantization.

Original: Park et al.
Revised: Park et al.

Original: propose value-aware quantization where the majority of data are computed at reduced precision while a small number of outliers are handled at high precision.
Revised: propose value-aware quantization where the majority of data are computed at reduced precision while a small number of outliers are handled at high precision.

Original: Our approach is signiﬁcantly different because we apply dual precision for every feature, not only for the outliers.
Revised: Our approach is signiﬁcantly different because we allow dual precision for every feature, not only for the outliers.

Original: Hua et al.

Original: (2018) propose channel gating to dynamically turn off a subset of channels that contribute little to the classiﬁcation result.

Original: Precision gating is orthogonal to this pruning technique as channel gating executes the whole network in the same precision.

Original: Input Activations   ℎ

Original: [Equation]

Original: =

Original: [Equation]

Original: ℎ

Original: [Equation]

Original:

Original: [Equation]

Original: (a) - +  ℎ  ℎ

Original: H i gh - p r e c

Original: i s i on

Original: Precision Gating on CNN

Original: Precision Gating on RNN - +

Original: H i gh - p r e c

Original: i s i on Δ Δ

Original: Lo w - p r e c i s i on

Original: Lo w - p r e c i s i on (b)

Original:  P RECISION G ATING 
Revised:  P RECISION G ATING (PG)

Original: In this section, we ﬁrst describe the basic mechanism of precision gating.
Revised: In this section we ﬁrst describe the basic mechanism of PG.

Original: Then we discuss how to design the gating logic to accelerate both forward and backward passes.
Revised: We then discuss how to design the gating scheme to accelerate both forward and backward passes.

Original: Finally, we incorporate PG with outlier clipping to reduce the quantization error.
Revised: Finally, we consider incorporating outlier clipping to reduce the quantization error.

Original: 3. B ASIC F ORMULATION
Revised: 3. B ASIC F ORMULATION

Original: Deﬁne a linear layer(i.e. convolutional or fully-connected layers) in a neural network  as O =
Revised: We ﬁrst deﬁne a linear layer in a neural network (either convolutional or fully-connected) as O =

Original: I ∗ W , where O , I , and W are the output, input, and weights, respectively.
Revised: I ∗ W , where O , I , and W are the output, input, and weights, respectively.

Original: Let I be represented in a B -bit ﬁxed-point format. As shown in Figure 1a, precision gating partitions I into  I hb , the  most-signiﬁcant 
Revised: Suppose I is represented in a B -bit ﬁxed-point format, which is shown in Figure 1. PG partitions I into (1) I hb , the B hb most-signiﬁcant bits (MSBs), and (2)

Original: B hb bits, and I lb , the remaining B lb bits, where B = B hb + B lb .
Revised: I lb , the remaining B lb least-signiﬁcant bits (LSBs). Here B = B hb + B lb .

Original: More formally, we can write:
Revised: More formally, we can write:

Original: [Equation]
Revised: [Equation]

Original: Here << is the  shift operator.
Revised: Here << denotes the left shift operator.

Original: We can then reformulate a single B -bit linear layer into two lowerprecision computations as:
Revised: We can then reformulate a single B -bit linear layer into two lower-precision computations as:

Original: [Equation]
Revised: [Equation]

Original: O hb is the partial sum obtained by using the MSBs of input feature ( I hb ) whereas O lb represents the remaining sum computed with I lb .
Revised: O hb is the partial product obtained by using the MSBs of input feature ( I hb ) whereas O lb represents the remaining partial product computed with I lb .

Original: Precision gating works in two phases.
Revised: Precision gating works in two phases.

Original: In the prediction phase , precision gating performs the computation O hb = W ∗ ( I hb << B lb ) .
Revised: In the prediction phase , PG performs the computation O hb = W ∗ ( I hb << B lb ) .

Original: Elements of O hb greater than a  gating threshold ∆ are considered important.
Revised: Output features of O hb greater than a learnable gating threshold ∆ are considered important.

Original: In the update phase , precision gating computes O lb=
Revised: In the update phase , PG computes O lb

Original:  W ∗ I lb only for the important elements and adds it to O hb .
Revised: = W ∗ I lb only for the important features and adds it to O hb .

Original: Precision gating’s execution ﬂow  is shown in Figure 2: unimportant output featurestake only  the upper path while important ones are computed as the sum of both paths.
Revised: The overall execution ﬂow of PG is illustrated in Figure 2, where unimportant output features only take the upper path while important ones are computed as the sum of both paths.

Original: Precision gating can be summarized as:
Revised: More precisely, this can be summarized as follows:

Original: [Equation]
Revised: [Equation]

Original: In essence, precision gating computes important features with B -bit arithmetic and unimportant features withonly Bhb bits.
Revised: In essence, PG intends to compute the majority of (unimportant) features with B hb bits and only a small set of important features with B bits.

Original: The importance of each element in the output O is predicted by comparing the magnitude of its partial sum O hb to ∆ .
Revised: The importance of each element in the output O is calculated by comparing the magnitude of its partial sum O hb to ∆ .

Original: Let p be the percentage of important activations over all features.
Revised: Let Sp be the percentage of [Figure] unimportant activations over all features.

Original: PG saves (1 − p ) · B lb B fraction of the compute in the original DNN model.
Revised: PG saves Sp · B lb B fraction of the compute in the original DNN model.

Original: Figure 1b illustrates which values are executed in high-precision. PG thus achieves dual-precision execution using a lightweight gating mechanism, and reduces the average bitwidth of computations in a DNN.
Revised: PG thus achieves dual-precision execution using a lightweight gating mechanism which only adds a comparison operation. The goal is to minimize the average bitwidth of the multiply-add operations in the DNN execution.

Original: 3. E FFICIENT L EARNABLE G ATING L OGIC
Revised: 3. E FFICIENT L EARNABLE G ATING S CHEME

Original: In order to automatically determine model and dataset speciﬁc threshold ∆ , we propose an efﬁcient end-to-end trainable gating logic.

Original: The gating threshold ∆ is speciﬁc to each output channel in  DNN [Figure] weights
Revised: PG automatically learns a gating threshold ∆ c,l during training for each output channel in each DNN layer.

Original: [Figure] layers and learned alongside the DNN weights during training. A larger ∆  indicates that more output features are computed in low-precision, resulting in greater computational savings but possibly at the expense of reduced model accuracy. 
Revised: A larger ∆ c,l indicates that more output features are computed in low-precision, resulting in greater computational savings but possibly at the expense of reduced model accuracy. Deﬁne ∆ as the vector containing each gating threshold ∆ c,l .

Original: We formulate the problem of optimizing ∆ as minimizing the original model loss L along with an L2 penalty term:
Revised: We formulate the problem of optimizing ∆ as minimizing the original model loss L along with an L2 penalty term:

Original: [Equation]
Revised: [Equation]

Original: Here y is the ground truth label, σ is the penalty factor, and δ is the gating target , a target value for the learnable threshold.
Revised: Here y is the ground truth label, σ is the penalty factor, and δ is the gating target , a target value for the learnable threshold.

Original: The penalty factor and gating target are hyperparameters which allow a user to emphasize high computation savings (large σ or δ ) or accuracy preservation (small σ or δ ).
Revised: The penalty factor and gating target are hyperparameters which allow a user to emphasize high computation savings (large σ or δ ) or accuracy preservation (small σ or δ ).

Original: Training a model with precision gating can be done on commodity GPUs using existing deep learning frameworks.
Revised: Training a model with precision gating can be performed on commodity GPUs using existing deep learning frameworks.

Original: We implement PG on GPU as the equation O = O hb + mask (cid:12)
Revised: We implement PG on GPU as the equation O = O hb + mask (cid:12)

Original: O lb , where mask =
Revised: O lb , where mask =

Original: 1 O hb > ∆ is a binary decision mask and (cid:12) represents element-wise multiplication.
Revised: 1 O hb > ∆ is a binary decision mask and (cid:12) represents element-wise multiplication.

Original: During forward propagation, most elements in mask are 0.
Revised: During forward propagation, most elements in mask are 0.

Original: PG therefore saves hardware execution cost by only computing a sparse O lb in the update phase.
Revised: PG therefore saves hardware execution cost by only computing a sparse O lb in the update phase.

Original: In dedicated hardware, MSBs and LSBs are wired separately. The prediction phase is controlled by a multiplexer, and only computes LSB convolutions while O hb exceeds the threshold, thus achieving the savings.
Revised: If PG is implemented in a dedicated hardware accelerator, MSBs and LSBs will be wired separately; The prediction phase can be controlled by a multiplexer, and only computes LSB convolutions while O hb exceeds the threshold, thus achieving savings in both compute cycles and energy.

Original: The mask is computed using a binary decision function (i.e. step function), which has a gradient of zero almost everywhere.
Revised: The mask is computed using a binary decision function (i.e., a step function), which has a gradient of zero almost everywhere.

Original: To let gradients ﬂow through mask to ∆ , we use a sigmoid on the backward pass to approximate the step.
Revised: To let gradients ﬂow through mask to ∆ , we use a sigmoid on the backward pass to approximate the step function.

Original: Speciﬁcally, we deﬁne mask
Revised: Speciﬁcally, we deﬁne mask

Original: = sigmoid ( α ( O hb − ∆)) only on the backward pass following Hua et al. 
Revised: = sigmoid ( α ( O hb − ∆)) only on the backward pass as in the previous work Hua et al. (2019b).

Original: Here α changes the slope of the sigmoid, thus controlling the magnitude of the gradients. 
Revised: Here α changes the slope of the sigmoid, thus controlling the magnitude of the gradients. [Figure]

Original: 3. S PARSE B ACK -P ROPAGATION
Revised: 3. S PARSE B ACK -P ROPAGATION

Original: A sparse update phase only reduces the computational cost during the inference (forward propagation).
Revised: A sparse update phase only reduces the computational cost during the inference (forward propagation).

Original: We further propose to save the compute during the back-propagation by modifying the forward function of the PG block.
Revised: We further propose to save the compute during the back-propagation by modifying the forward function of the PG block.

Original: Speciﬁcally, we square the mask element-wise.
Revised: Speciﬁcally, we square the mask element-wise.

Original: [Equation]
Revised: [Equation]

Original: Given that mask is a binary tensor, mask 2 in Eq.
Revised: Given that mask is a binary tensor, mask 2 in Eq.

Original: (5) preserves the same value as mask .
Revised: (5) preserves the same value as mask .

Original: Thus the forward pass remains unchanged.
Revised: Thus the forward pass remains unchanged.

Original: During the back-propagation, an additional mask term is introduced in computing the gradient of O with respect to the gating threshold ∆ in Eq.
Revised: During the back-propagation, an additional mask term is introduced in computing the gradient of O with respect to the gating threshold ∆ in Eq.

Original: (6) because of the element-wise square.
Revised: (6) because of the element-wise square.

Original: Consequently, the update of ∆ only requires the result of mask (cid:12) O lb which has already been computed during the forward pass.
Revised: Consequently, the update of ∆ only requires the result of mask (cid:12) O lb which has already been computed during the forward pass.

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: The gradient of O with respect to the weights W in Eq.
Revised: The gradient of O with respect to the weights W in Eq.

Original: (7) employs the same sparse mask (cid:12)
Revised: (7) employs the same sparse mask (cid:12)

Original: O lb as the update of ∆ .
Revised: O lb as the update of ∆ .

Original: Therefore, precision gating can reduce the computational cost of both forwardprogagtion (inference) and back-propagation by the same factor.
Revised: Therefore, precision gating can reduce the computational cost of both forwardprogagtion (inference) and back-propagation by the same factor.

Original: [Equation]
Revised: [Equation]

Original: 3. O UTLIER C LIPPING
Revised: 3. O UTLIER C LIPPING

Original: Precision gating identiﬁes important features using low-precision computation.
Revised: PG predicts important features using low-precision computation.

Original: One difﬁculty with this is that DNN activations are distributed in a bell curve, with most values close to zero and a few large outliers.
Revised: One difﬁculty with this is that DNN activations are distributed in a bell curve, with most values close to zero and a few large outliers.

Original: The top row of Figure 3(a) shows some activations in blue dots, including a single outlier.
Revised: The top row of Figure 3(a) shows some activations as blue dots, including a single outlier.

Original: If we quantize each value to 4 bits (second column) and use 2 most-signiﬁcant bits in the prediction phase (third column), we see that almost all values are rounded to zero.
Revised: If we quantize each value to 4 bits (second column) and use 2 most-signiﬁcant bits in the prediction phase (third column), we see that almost all values are rounded to zero.

Original: In this case, PG can only distinguish the importance between the single outlier and the rest of the values no matter what ∆ is.
Revised: In this case, PG can only distinguish the importance between the single outlier and the rest of the values no matter what ∆ is.

Original: Thus the presence of large outliers greatly reduces precision gating’s effectiveness.
Revised: Thus, the presence of large outliers greatly reduces the effectiveness of PG.

Original: To address this, we combine precision gating with PACT (Choi et al., 2018), which clips each layer’s outputs using a learnable clip threshold.
Revised: To address this, we combine PG with PACT (Choi et al., 2018), which clips each layer’s outputs using a learnable clip threshold.

Original: The bottom row of Figure 3 shows how clipping limits the dynamic range of activations, making values more uniformly distributed along the quantization grid.
Revised: The bottom row of Figure 3 shows how clipping limits the dynamic range of activations, making values more uniformly distributed along the quantization grid.

Original: Now the 2 most-signiﬁcant bits can effectively separate out different groups of values based on magnitude.
Revised: Now the 2 most-signiﬁcant bits can effectively separate out different groups of values based on magnitude.

Original: We apply PACT to precision gating in CNNs, which typically use an unbounded activation function (ReLU).
Revised: We apply PACT to PG in CNNs, which commonly use an unbounded activation function such as ReLU.

Original: RNNs typically employ a bounded activation function (tanh, sigmoid), making PACT unnecessary.
Revised: RNNs, on the other hand, typically employ a bounded activation function (e.g., tanh, sigmoid) that often makes PACT unnecessary.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test precision gating using ResNet-18 (He et al., 2015) and ShiftNet-20 (Wu et al., 2017) on CIFAR-10 (Krizhevsky & Hinton, 2009), and ShufﬂeNet V2 (Ma et al., 2018) on ImageNet (Deng et al., 2009).
Revised: We evaluate PG using ResNet-20 (He et al., 2016a) and ShiftNet-20 (Wu et al., 2018a) on CIFAR10 (Krizhevsky & Hinton, 2009), and ShufﬂeNet V2 (Ma et al., 2018) on ImageNet (Deng et al., 2009).

Original: ResNet is a very popular CNN  for image classiﬁcation, and the other two lightweight architectures are  designed speciﬁcally for  edge devices.
Revised: ResNet is a very popular CNN architecture for image classiﬁcation. ShiftNet and ShufﬂeNet are more compact architectures designed speciﬁcally for mobile and edge devices.

Original: We set the expansion rate of ShiftNet to be six and choose the 0.5 × variant of ShfﬂeNet V2 for all experiments. 
Revised: We set the expansion rate of ShiftNet to be 6 and choose the 0.5 × variant of ShfﬂeNet V2 for all experiments. On

Original: On CIFAR-10, the batch size is 128, and the models are trained for 200 epochs.
Revised: CIFAR-10, the batch size is 128, and the models are trained for 200 epochs.

Original: The initial learning rate is 0.1 and decayed at epoch 100, 150, 200 by a factor of 0.1 (multiply learning rate by 0.1).
Revised: The initial learning rate is 0.1 and decays at epoch 100, 150, 200 by a factor of 0.1 (i.e., multiply learning rate by 0.1).

Original: On ImageNet, the batch size isand the models are trained for 120 epochs.
Revised: On ImageNet, the batch size is 512 and the models are trained for 120 epochs.

Original: The learning rate decayed linearly from an initial value of 0.5 to 0.
Revised: The learning rate decays linearly from an initial value of 0.5 to 0.

Original: We also test an LSTM  (Hochreiter & Schmidhuber, 1997) on the Penn Tree Bank (PTB) (Marcus et al., 1993) corpus.
Revised: We also test an LSTM model (Hochreiter & Schmidhuber, 1997) on the Penn Tree Bank (PTB) (Marcus et al., 1993) corpus.

Original: The model accuracy is measured by perplexity per word (PPW), and lower  is better.
Revised: The model accuracy is measured by perplexity per word (PPW), where a lower PPW is better.

Original: Following the conﬁguration used by He et al.
Revised: Following the conﬁguration used by He et al.

Original: (2016) and Hubara et al.
Revised: (2016b) and Hubara et al.

Original: (2017), the number of hidden units in the LSTM cell is set to 300, and the number of layers is set to 1.
Revised: (2017), the number of hidden units in the LSTM cell is set to 300, and the number of layers is set to 1.

Original: We follow the same training setting as described in He et al.
Revised: We follow the same training setting as described in He et al.

Original: (2016), except  the learning rate decayed by a factor of 0.1 at epoch 50 and 90.
Revised: (2016b), except that the learning rate decays by a factor of 0.1 at epoch 50 and 90.

Original: All experiments are run on Tensorﬂow (Abadi et al., 2016) with NVIDIA GeForce 1080Ti GPUs.
Revised: All experiments are conducted on Tensorﬂow (Abadi et al., 2016) with NVIDIA GeForce 1080Ti GPUs.

Original: We report the top-1 accuracy for all experiments.
Revised: We report the top-1 accuracy for all experiments.

Original: [Table] [Table] We replace the conv layers in CNNs and FC layers in LSTM with the proposed PG block.
Revised: We replace the convolutional layers in CNNs and the dense layers in LSTM with the proposed PG block.

Original: Moreover, the following hyperparameters in PG need to be tuned appropriately to achieve  low average bitwidth with  high accuracy.
Revised: Moreover, the following hyperparameters in PG need to be tuned appropriately to achieve a low average bitwidth with a high accuracy

Original: • The full bitwidth B – this represents the bitwidth for high-precision computation in PG.
Revised: • The full bitwidth B – this represents the bitwidth for high-precision computation in PG.

Original: B is set to 5 or less on CIFAR-10, 5 or 6 on ImageNet, and 3 or 4 on PTB.
Revised: B is set to 5 or less on CIFAR-10, 5 or 6 on ImageNet, and 3 or 4 on PTB.

Original: • The prediction bitwidth B hb – this represents the bitwidth for low-precision computation.
Revised: • The prediction bitwidth B hb – this represents the bitwidth for low-precision computation.

Original: • The penalty factor σ – this is the scaling factor of the L2 loss for gating thresholds ∆ .
Revised: • The penalty factor σ – this is the scaling factor of the L2 loss for gating thresholds ∆ .

Original: • The gating target δ – the target gating threshold.
Revised: • The gating target δ – the target gating threshold.

Original: We use a variety of values δ ∈ [ − 1 . 0 , 5 . 0] in our experiments.
Revised: We use a variety of values δ ∈ [ − 1 . 0 , 5 . 0] in our experiments.

Original: • The coefﬁcient α in the backward pass – α controls the magnitude of gradients ﬂowing to ∆ .
Revised: • The coefﬁcient α in the backward pass – α controls the magnitude of gradients ﬂowing to ∆ .

Original: We set α to be 5 across all experiments.
Revised: We set α to be 5 across all experiments.

Original: 4. CNN R ESULTS
Revised: [Table] (fp 89.4%) 3/ 71. ResNet4/ 78. CIFAR3/ 90.

Revised: (fp 91.7%) 2/ 71.

Revised: ShufﬂeNet 6/ 57.

Original:  To compare  hardware execution efﬁciency across different techniques, we compute the average bitwidth ( B avg ) of all features in a DNN model:
Revised: ImageNet 6/ 62. (fp 59.0%) 5/ 41. 4. CNN R ESULTS To compare the hardware execution efﬁciency across different techniques, we compute the average bitwidth ( B avg ) of all features in a DNN model:

Original: [Equation]
Revised: [Equation]

Original: where p is the percentage of high-precision activations (i.e. number of important features divided by total features).
Revised: Here Sp denotes sparsity in terms of the percentage of low-precision activations (i.e., number of unimportant features divided by total features).

Original: The computational cost of DNNs is proportional to the average bitwidth.
Revised: The computational cost of DNNs is proportional to the average bitwidth.

Original: Our results on CNNs are presented in Table 1 and Table 2, where sp. stands for sparsity.
Revised: Our results on CNNs are presented in Tables 1 and 2.

Original: We ﬁrst compare PG against two widely adopted quantization schemes — uniform quantization (UQ) and PACT (Choi et al., 2018).
Revised: We ﬁrst compare PG against two widely adopted quantization schemes — uniform quantization (UQ) and PACT (Choi et al., 2018).

Original: In Table 1,the 3rd and 4th columns  show the average bitwidth and model accuracy of PG whereas the 5th–8th columns  list the bitwidth and corresponding model accuracy of UQ and PACT.
Revised: In Table 1, columns 3 and 4 show the average bitwidth and model accuracy of PG, respectively; columns 5-7 list the bitwidth and corresponding model accuracy of UQ and PACT, respectively.

Original: At each row PG achieves  better accuracy with fewer average bitwidth (Bits vs. B avg ).
Revised: At each row, PG achieves a better accuracy with a lower average bitwidth ( B avg vs. Bits).

Original: Speciﬁcally, PG achieves 6.7% and 1.6% higher accuracy with 1.25 × computational cost reduction , and 6.6% and 0.5% higher accuracy with 1.54 × computational cost reduction than 2-bit UQ and PACT on ShiftNet-20 and ResNet-18 for CIFAR-10 (3rd & 6th rows), respectively.
Revised: Speciﬁcally, PG achieves 6.7% and 1.6% higher accuracy with 1.25 × computational cost reduction, and 6.6% and 0.5% higher accuracy with 1.54 × computational cost reduction than 2-bit UQ and PACT on ShiftNet-20 and ResNet-20 for CIFAR-10 (rows 3 and 6), respectively.

Original: We observe the same trend on ShufﬂeNet for ImageNet where PG improves the accuracy of 5-bit UQ and PACT by 1.0% and 1.4% with 1.22 × computational cost reduction (9th row).
Revised: We observe the same trend on ShufﬂeNet for ImageNet, where PG improves the accuracy of 5-bit UQ and PACT by 1.0% and 1.4% with 1.22 × computational cost reduction (row 9).

Original: It is also worth noting that PG can recover the accuracy of the ﬂoating-point ShiftNet-20, ResNet-18, and ShufﬂeNet  with 3.9 , 3.2 , and 4.7 average bitwidth.
Revised: It is also worth noting that PG can recover the accuracy of the ﬂoating-point ShiftNet-20, ResNet-20, and ShufﬂeNet V2 0.5 × with 3.9 , 3.2 , and 4.7 average bitwidth, respectively.

Original: This demonstrates that PG, using a learnable threshold, can identify unimportant features and reduce  bitwidth without compromising accuracy.
Revised: This demonstrates that PG, using a learnable threshold, can predict unimportant features and reduce their bitwidth without compromising accuracy.

Revised: The results compared to quantization baselines are visualized in Figure 4, which plots accuracy vs. average bitwidth — uniform quantization (squares), PACT (triangles), and

Revised: PG (circles) are shown on separate curves.

Revised: Results closer to the upper-left corner are better.

Original: We then compare PG with Fix-Threshold, which is an extension of  zero-prediction (Lin et al., 2017; Song et al., 2018).
Revised: We then compare PG with Fix-Threshold, which is an extension of the zero-prediction (Lin et al., 2017; Song et al., 2018).

Original: Lin et al.
Revised: Lin et al.

Original: (2017) and Song et al.
Revised: ; Song et al.

Original: (2018) explicitly predict ReLU-induced zeros during inference.
Revised: (2018) explicitly predict ReLU-induced zeros during inference.

Original: To have a fair comparison, we extend their technique to predict an arbitrary ﬁxed threshold to achieve the same or fewer B avg as PG, reported as Fix-Threshold in Table 1.
Revised: To have a fair comparison, we extend their technique to predict an arbitrary ﬁxed threshold to achieve the same or lower B avg as PG, reported as Fix-Threshold in Table 1.

Original: For CIFAR-10, we observe that PG achieves 20.2% and 18.7% higher accuracy with 1.75 × and 1.38 × computational cost reduction than Fix-Threshold on ShiftNet-20 and ResNet-18 (3rd & 6th rows), respectively.
Revised: For CIFAR-10, we observe that PG achieves 20.2% and 18.7% higher accuracy with 1.75 × and 1.38 × computational cost reduction than Fix-Threshold on ShiftNet-20 and ResNet-20 (rows 3 and 6), respectively.

Original: The gap in accuracy becomes even larger on ShufﬂeNet V2 for ImageNet dataset.
Revised: The gap in accuracy becomes even larger on ShufﬂeNet V2 for ImageNet dataset.

Original: With the same or  lower average bitwidth, the accuracy of PG is at least 26% higher than Fix-Threshold.
Revised: With the same or a lower average bitwidth, the accuracy of PG is at least 26% higher than FixThreshold (rows 7-9).

Original: In conclusion, PG consistently outperforms Fix-Threshold at any bitwidth across all networks and datasets (7th – 9th rows).

Original: The reason is that PG learns to adjust the gating thresholds to the clipped activation distribution so that it predicts with less zero bits.
Revised: In conclusion, PG consistently outperforms Fix-Threshold because PG uses a learnable gate function which can adjust its threshold to the clipped activation distribution.

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Figure] We further compare PG with SeerNet (Cao et al., 2019) in Table 2.
Revised: We further compare PG with SeerNet (Cao et al., 2019) in Table 2.

Original:  SeerNet adds an extra quantized conv layer to predicta sparse mask for executing the original ﬂoating-point layer during inference.
Revised: For each convolutional layer, SeerNet executes a quantized version of the same layer to predict the output sparsity.

Revised: It then computes in full precision the output activations that are not suppressed according to the prediction.

Original: As the code of SeerNet is  unavailable, we implement the network in Tensorﬂowby ourselves and boost its accuracy by retraining the network.
Revised: Since the code of SeerNet is currently unavailable, we implement the network in Tensorﬂow and boost its accuracy by retraining the network.

Original: We reduce the average bitwidth of SeerNet while keeping a comparable accuracy as PG.
Revised: We reduce the average bitwidth of SeerNet while keeping a comparable accuracy as PG.

Original: At similar model prediction accuracy, PG achieves 3.5% higher accuracy with3.3 × and 4.5 × less compute on ResNet-18 and ShiftNet for CIFAR-10, respectively.
Revised: For CIFAR-10, PG reduces the computational cost of ResNet3.3 × more than SeerNet at the same prediction accuracy. Meanwhile for the hardware-friendly ShiftNet-20, PG achieves 3.2% higher accuracy with 4.5 × less compute than SeerNet.

Original: For ImageNet, PG on ShufﬂeNet also achieves 0.4% higher accuracy with 2.4 × computational cost reduction than SeerNet.
Revised: For ImageNet, PG on ShufﬂeNet also achieves 0.4% higher accuracy with 2.4 × computational cost reduction than SeerNet.

Original: PG outpeforms SeerNet as SeerNet uses theextra quantized layer for prediction only and its outputs are sparse thus inaccurate, while PG  reuses thelow-precision results and passes them to outputs anyway.
Revised: It is worth noting that SeerNet does not reuse the outputs from the quantized layer, which may incur a nontrivial overhead in execution time. In contrast, when PG invokes the update phase, it reuses the outputs of the low-precision computation from the prediction phase. [Table]

Revised: In order to evaluate the efﬁcacy of the learnable thresholds, we further sweep a series of manually thresholds on the ResNet-20 for CIFAR-10.

Revised: Table 3 shows the results where we use a dual-precision mode of B / B hb = 3 / 2 .

Revised: As the threshold decreases from 3 to -4, the average bitwidth in the update phase consistently increases.

Revised: This is expected because we compute more output features in a high precision.

Original:  To quantify the impact of sparse back-propagation described in Section 3.3, we run PG with and without sparse back-propagation on CNNs.
Revised: The model prediction accuracy therefore increases. Compared to these manually set thresholds, PG achieves a much improved model accuracy (91.2%) with a higher sparsity (90.1%). [Table] To quantify the impact of sparse back-propagation described in Section 3.3, we run PG with and without sparse back-propagation on CNNs.

Original: Table 3 compares the sparsity in the update phase of PG with and without back-propagation under the same model accuracy and average bitwidth.
Revised: Table 4 compares the sparsity in the update phase of PG with and without back-propagation under the same model accuracy and average bitwidth.

Original: Surprisingly, we ﬁnd that the sparsity in the update phase of PG with sparse back-propagation is consistently higher than that of PG without sparse back-propagation across all models and datasets.
Revised: Interestingly, we ﬁnd that the sparsity in the update phase of PG with sparse back-propagation is consistently higher than that of PG without sparse back-propagation across the tested models and datasets.

Original: For both CIFAR-10 and ImageNet, the sparsity increment is between 6% and 21%.
Revised: For both CIFAR-10 and ImageNet, the sparsity increases by between 6% and 21%.

Original: We hypothesize that sparse back-propagation zeros out the gradients ﬂowing to non-activated LSB convolutions in the update phase, which leads to  higher sparsity.
Revised: We hypothesize that sparse back-propagation zeros out the gradients ﬂowing to non-activated LSB convolutions in the update phase, which leads to a higher sparsity.

Original: Overall, compared to other prediction based quantization schemes, our method has measurably up to 21.6% higher accuracy than the baseline with lower computational cost or up to 4.5 × less compute with the same or better accuracy for CIFAR-10.

Original: For ImageNet ours has a maximum of 30.3% higher accuracy with similar computational cost or 2.4 × less computational cost with 0.4% higher accuracy.

Original: The results compared to quantization baselines are visualized in Figure 4, which plots accuracy vs. average bitwidth — uniform quantization (squares), PACT (triangles), and PG (circles) are shown on separate curves.

Original: Results closer to the upper-left corner are better.

Original: 4. LSTM R ESULTS
Revised: 4. LSTM R ESULTS

Original: Besides CNNs, PG  works great on RNNs as well.
Revised: PG also works well on RNNs.

Original: The results  on  LSTM with PG for the PTB corpus are reported in Table 4.
Revised: Table 5 reports our results applying PG on an LSTM model for the PTB corpus.

Original: Here,  only  the uniform quantization is used as the baseline because PACT, Fix-Threshold, and SeerNet do not work for sigmoid or tanh activation functions.
Revised: Here, we only compare with the uniform quantization since PACT, Fix-Threshold, and SeerNet do not work for sigmoid or tanh activation functions.

Original: Although Hubara et al.
Revised: Although Hubara et al.

Original: (2017) claim that quantizing both weights and activations to 4 bits does not lower PPW, we observe asigniﬁcant PPW degradation as B decreases from 8 to 4 bits in our implementation.The
Revised: (2017) claim that quantizing both weights and activations to 4 bits does not lower PPW, we observe a PPW degradation when B decreases from 8 to 4 bits in our implementation.

Original:  LSTM with 8-bit activations is therefore considered as the full accuracy model.
Revised: The LSTM with 8-bit activations is therefore considered as the full-accuracy model.

Revised: We observe the same trend as in the

Original: We observe the same phenomena as in the CNN results that PG enables the 3-bit LSTM cell to improve the PPW by 1.2% and reduce the computational cost by 2.7 × compared to  8-bit uniform quantization. 
Revised: CNN evaluation where PG enables the 3-bit LSTM cell to improve the PPW by 1.2% and reduce the computational cost by 2.7 × compared to the 8-bit uniform quantization. 8 Average Bitwidth 77. M o d e l A cc u r a c y ( % ) UQ PACT

Revised: PG (a) ShifNet-20 on CIFAR8 Average Bitwidth 89.

Original: 4. K ERNEL S PEEDUP
Revised: M o d e l A cc u r a c y ( % ) UQ PACT PG (b) ResNet-18 on CIFAR9 Average Bitwidth M o d

Revised: [Figure]

Revised: cc

Revised: [Figure]

Revised: UQ PACT

Revised: PG (c) ShufﬂeNet on ImageNet   8 Average Bitwidth

Revised: [Figure]

Revised: UQ

Revised: [Figure]

Revised: Base

Revised: PG

Revised: Bits

Revised: [Table]

Revised: [Table]

Revised: S PARSE

Original:  The sparse update phase of PG can be implemented efﬁciently with a new kernel called sampled dense-dense matrix multiplication (SDDMM).
Revised: ERNEL S PEEDUP The sparse update phase of PG can be implemented efﬁciently with a new kernel called sampled dense-dense matrix multiplication (SDDMM).

Original: A convolutional layer with PG is then factorized into a regular low-precision convolution bounded with a low-precision SDDMM.
Revised: A convolutional layer with PG is then factorized into a regular low-precision convolution bounded with a low-precision SDDMM.

Original: To evaluate the actual speedup of PG, we implement the SDDMM kernel in Python leveraging a high performance JIT compiler Numba (Lam et al., 2015) and test it on the ResNet-18 model for CIFAR-10.
Revised: To evaluate the potential speedup of PG, we implement the SDDMM kernel in Python leveraging a high performance JIT compiler Numba (Lam et al., 2015) and test it on the ResNet-20 model for CIFAR-10.

Original: Table 6 shows the layer-wise sparsity and the kernel speedup compared to the dense  baseline on Intel Xeon Silver 4114 CPU (2.20GHz).
Revised: Table 6 shows the layer-wise sparsity and the kernel speedup compared to the dense matrix multiplication baseline on Intel Xeon Silver 4114 CPU (2.20GHz).

Original: With the large sparsity (from 76% to 99%) in each layer induced by PG, the SDDMM kernel achieves up to 8.3 × wall clock speedup over the general dense matrix-matrix multiplication (GEMM) kernel.
Revised: With the high sparsity (from 76% to 99%) in each layer induced by PG, the SDDMM kernel achieves up to 8.3 × wall clock speedup over the general dense matrix-matrix multiplication (GEMM) kernel.

Original: The signiﬁcant wall clock speedup shows an enormous potential of deploying PG on commodity hardware.
Revised: The signiﬁcant wall clock speedup shows a good potential of deploying PG on commodity hardware.

Original: [Table] For a GPU implementation, we face the challenge in replacing the GEMM kernel with  SDDMM kernel.
Revised: For a GPU implementation, we need to replace the GEMM kernel with the SDDMM kernel to accelerate the update phase.

Original: Popular deep learning framework such as Tensorﬂow does not incorporate SDDMM as an built-in operator.
Revised: Mainstream deep learning frameworks such as Tensorﬂow currently do not provide a built-in operator for SDDMM, which is essential for achieving high performance on GPUs.

Original: However, Nisa et al.
Revised: Nevertheless, Nisa et al.

Original: (2018) demonstrate that a highly optimized SDDMM kernel with a similar sparsity shown in Table 6 achieves about 4 × speedup over a GEMM kernel.
Revised: (2018) have recently demonstrated that a highly optimized SDDMM kernel with a similar sparsity shown in Table 6 can achieve about 4 × speedup over a GEMM ker- [Table]

Original:  This shows strong evidence that PG has a potential to obtain high speedup on GPUs as well.
Revised: nel. This shows strong evidence that PG has a potential to obtain high speedup on GPUs as well.

Original: Additionally, our approach is compatible with the specialized accelerator architectures proposed by Lin et al.
Revised: Additionally, our approach is a good ﬁt for the specialized accelerator architectures proposed by Lin et al.

Original: (2017) and Song et al.
Revised: (2017) and Song et al.

Original: Due to the enormous sparsity, DNNs with PG are estimated to get at least 3 × speedup and 5 × energy efﬁciency on these dedicated hardware accelerators.
Revised: Due to the high sparsity, DNNs with PG are estimated to get at least 3 × speedup and 5 × energy efﬁciency on these dedicated hardware accelerators.

Original: We leave the deployment of the SDDMM kernel on GPU and dedicated hardware to future work.
Revised: We leave the deployment of the SDDMM kernel on GPU and dedicated hardware to future work.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose precision gating, a dynamic dual-precision quantization method that can  reduce the computation and storage costs of DNNs.
Revised: We propose precision gating, a dynamic dual-precision quantization method that can effectively reduce the computational cost of DNNs.

Original: PG assigns higher precision to important features and lower precision to the remaining features at run-time.
Revised: PG assigns higher precision to important features and lower precision to the remaining features at run-time.

Original: The  technique is end-to-end trainable allowing individual models to learn to distinguish salient and non-salient features.
Revised: The proposed technique is end-to-end trainable, allowing individual models to learn to distinguish important and unimportant features.

Original: Experimental results show that PG outperforms state-of-the-art quantization and prediction approaches by a large margin on both CNN and RNN benchmarks on datasets such as Imagenet and Penn Tree Bank.
Revised: Experimental results show that PG outperforms state-of-the-art quantization and prediction approaches by a large margin on both CNN and RNN benchmarks on datasets such as Imagenet and Penn Tree Bank.

Revised: We will release the source code on the author’s website.

Revised: A CKNOWLEDGMENTS

Revised: This work was supported in part by the Semiconductor Research Corporation (SRC) and DARPA.

Revised: One of the Titan Xp GPUs used for this research was donated by the NVIDIA Corporation.

Revised: We thank Jordan Dotzel (Cornell) for his helpful discussions during the camera-ready revision

Original: A A PPENDIX A. F EATURE V ISUALIZATION
Revised: A A PPENDIX A. F EATURE V ISUALIZATION

Original: In precision gating we expect that the model will learn to compute features whose prediction values exceeding the threshold in high-precision while keeping others computed at low precision.
Revised: In precision gating, we expect that the model will learn to compute features whose prediction values exceed the threshold using a high-precision while keeping others computed using a low precision.

Original: In the image recognition task we expect that high-precision features are mostly in the region where an object lies.
Revised: In the image recognition task, we expect that high-precision features are mostly in the region where an object lies.

Original: To provide evidence that the precision gating can effectively learn to identity those regions, in Figure  we visualize the decision maps extracted from the ﬁnal convolutional layer that is modiﬁed to support precision gating in the ResNet-18 model.
Revised: To provide evidence that PG can effectively learn to identity those regions, in Figure 5, we visualize the decision maps extracted from the ﬁnal convolutional layer that is modiﬁed to support PG in the ResNet-20 model.

Original: A decision map is a gray scale 2D image that has the same spatial size as the output feature map in the same convolutional layer.
Revised: A decision map is a gray scale 2D image that has the same spatial size as the output feature map in the same convolutional layer.

Original: The brighter a pixel is in the decision map, the more probably the same spatial location in the output feature map will be computed in high precision.
Revised: The brighter a pixel is in the decision map, the more probably the same spatial location in the output feature map will be computed using a high precision.

Original: The ﬁrst row contains the original input images in CIFAR-10, and  their corresponding decision maps are shown in the second row.
Revised: The ﬁrst row contains the original input images in CIFAR-10, and the second row shows their corresponding decision maps.

Original: We can see clearly that in each decision map, the locations of bright pixels roughly align with the object in the original image, which is exactly what we expect.
Revised: We can see clearly that in each decision map, the locations of bright pixels roughly align with the object in the original image.

Original: A. A DDITIONAL R ESULTS
Revised: A. A DDITIONAL R ESULTS

Original: We provide more supplementary results in this section. [Table] [Table] [Table]
Revised: We provide more supplementary results in this section as shown in Table 7.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: and ShufﬂeNet V2 0.5 × on ImageNet. ”

Original: fp” is ﬂoating-point accuracy.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: (fp

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: (fp top-5 99.8%) 2/

Original: [Table]

Original: [Table]

Original: [Table]

Original: ShufﬂeNet V

Original: [Table]
Revised: [Table]

Original: .5 × 6/

Original: [Table]

Original: (fp top-1 59.0%)

Original: [Table]

Original: [Table]

Original: (fp

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]