Original: L INEAR S YMMETRIC Q UANTIZATION OF N EURAL N ETWORKS FOR L OW - PRECISION I NTEGER H ARD WARE
Revised: L INEAR S YMMETRIC Q UANTIZATION OF N EURAL N ETWORKS FOR L OW - PRECISION I NTEGER H ARD -

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: With the proliferation of specialized neural network processors that operate on low-precision integers, the performance of Deep Neural Network inference becomes increasingly dependent on the result of quantization.
Revised: With the proliferation of specialized neural network processors that operate on low-precision integers, the performance of Deep Neural Network inference becomes increasingly dependent on the result of quantization.

Original: Despite plenty of prior work on the quantization of weights or activations for neural networks, there is still a wide gap between the software quantizers and the low-precision accelerator implementation, which degrades either the efﬁciency of networks or that of the hardware for the lack of software and hardware coordination at designphase.
Revised: Despite plenty of prior work on the quantization of weights or activations for neural networks, there is still a wide gap between the software quantizers and the low-precision accelerator implementation, which degrades either the efﬁciency of networks or that of the hardware for the lack of software and hardware coordination at designphase.

Original: In this paper, we propose a learned linear symmetric quantizer for integer neural network processors, which not only quantizes neural parameters and activations to low-bit integer but also accelerates hardware inference by using batch normalization fusion and low-precision accumulators (e.g., 16-bit) and multipliers (e.g., 4-bit).
Revised: In this paper, we propose a learned linear symmetric quantizer for integer neural network processors, which not only quantizes neural parameters and activations to low-bit integer but also accelerates hardware inference by using batch normalization fusion and low-precision accumulators (e.g., 16-bit) and multipliers (e.g., 4-bit).

Original: We use a uniﬁed way to quantize weights and activations, and the results outperform many previous approaches for various networks such as
Revised: We use a uniﬁed way to quantize weights and activations, and the results outperform many previous approaches for various networks such as

Original: AlexNet, ResNet, and lightweight models like MobileNet while keeping friendly to the accelerator architecture.
Revised: AlexNet, ResNet, and lightweight models like MobileNet while keeping friendly to the accelerator architecture.

Original: Additional, we also apply the method to object detection models and witness high performance and accuracy in YOLO-v2.
Revised: Additional, we also apply the method to object detection models and witness high performance and accuracy in YOLO-v2.

Original: Finally, we deploy the quantized models on our specialized integer-arithmetic-only DNN accelerator to show the effectiveness of the proposed quantizer.
Revised: Finally, we deploy the quantized models on our specialized integer-arithmetic-only DNN accelerator to show the effectiveness of the proposed quantizer.

Original: We show that even with linear symmetric quantization, the results can be better than asymmetric or non-linear methods in 4-bit networks.
Revised: We show that even with linear symmetric quantization, the results can be better than asymmetric or non-linear methods in 4-bit networks.

Original: In evaluation, the proposed quantizer induces less than 0.4% accuracy drop in ResNet18, ResNet34, and AlexNet when quantizing the whole network as required by the integer processors.
Revised: In evaluation, the proposed quantizer induces less than 0.4% accuracy drop in ResNet18, ResNet34, and AlexNet when quantizing the whole network as required by the integer processors.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have shown excellent performance on various computer vision and natural language processing tasks, such as classiﬁcation (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015;
Revised: Deep neural networks have shown excellent performance on various computer vision and natural language processing tasks, such as classiﬁcation (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015;

Original: He et al., 2016), object detection (Girshick, 2015; Redmon et al., 2016;
Revised: He et al., 2016), object detection (Girshick, 2015; Redmon et al., 2016;

Original: He et al., 2017), segmentation (Long et al., 2015; Noh et al., 2015), machine translation (Zhang et al., 2018b), speech recognition (Nassif et al., 2019), etc.
Revised: He et al., 2017), segmentation (Long et al., 2015; Noh et al., 2015), machine translation (Zhang et al., 2018b), speech recognition (Nassif et al., 2019), etc.

Original: While the past few years witnessed the success of DNNs on cloud and server-end computers, neural networks have been recently pushed to embedded and mobile areas to enable edge intelligence.
Revised: While the past few years witnessed the success of DNNs on cloud and server-end computers, neural networks have been recently pushed to embedded and mobile areas to enable edge intelligence.

Original: For these scenarios, the power provision and computational strength on the edge computing devices are limited.
Revised: For these scenarios, the power provision and computational strength on the edge computing devices are limited.

Original: As a result, it is essential to have more efﬁcient network architectures and less expensive inference overhead.
Revised: As a result, it is essential to have more efﬁcient network architectures and less expensive inference overhead.

Original: Therefore, there is increasing attention from the research community to study the compression of modern deep neural networks that are typically over-parameterized and computationally costly.
Revised: Therefore, there is increasing attention from the research community to study the compression of modern deep neural networks that are typically over-parameterized and computationally costly.

Original: Several categories of approaches are proposed to decrease the computational overhead of neural networks, such as lightweight neural network architectures (Howard et al., 2017), neural architecture search (NAS) (Elsken et al., 2018), and network pruning (Han et al., 2015; 2016; Wen et al., 2016; 
Revised: Several categories of approaches are proposed to decrease the computational overhead of neural networks, such as lightweight neural network architectures (Howard et al., 2017), neural architecture search (NAS) (Elsken et al., 2018), and network pruning (Han et al., 2015; 2016; Wen et al., 2016; [Table]

Original: Molchanov et al., 2017).
Revised: al [Table]

Original: Besides these techniques, quantizing high-precision ﬂoating-point networks to lower bitwidth representation can also drastically decrease both the static parameters and the intermediate data generated during the network inference, resulting in reduced memory footprint and also computational intensity.
Revised: Besides these techniques, quantizing high-precision ﬂoating-point networks to lower bitwidth representation can also drastically decrease both the static parameters and the intermediate data generated during the network inference, resulting in reduced memory footprint and also computational intensity.

Original: And this paper focuses on the quantization of neural networks.
Revised: And this paper focuses on the quantization of neural networks.

Original: [Table]

Original: Quantization technique is also closely related to the implementation of specialized hardware that maps the procedure of network inference onto the energy-efﬁcient low-precision integer or ﬁxedpoint arithmetic circuits.
Revised: Quantization technique is also closely related to the implementation of specialized hardware that maps the procedure of network inference onto the energy-efﬁcient low-precision integer or ﬁxedpoint arithmetic circuits.

Original: In the hardware perspective, low-precision integer accelerators or processors are dominating the solutions targeted on neural network inference, especially for mobile and embedded scenarios.
Revised: In the hardware perspective, low-precision integer accelerators or processors are dominating the solutions targeted on neural network inference, especially for mobile and embedded scenarios.

Original: Google’s Tensor Processing Unit 1.0 (TPU) (Jouppi et al., 2017), Uniﬁed Deep
Revised: Google’s Tensor Processing Unit 1.0 (TPU) (Jouppi et al., 2017), Uniﬁed Deep

Original: Neural Network Accelerator (UNPU) (Lee et al., 2018), Eyeriss (Chen et al., 2018), Stripes (Judd et al., 2016), Pragmatic( ? ) and many other newly proposed hardware implementations are generally reliant on the effectiveness of the underlying quantization techniques, which are especially crucial for the low-precision integer hardware designed to process binary, ternary, 4-bit or 8-bit networks.
Revised: Neural Network Accelerator (UNPU) (Lee et al., 2018), Eyeriss (Chen et al., 2018), Stripes (Judd et al., 2016), Pragmatic(Albericio et al., 2017) and many other newly proposed hardware implementations are generally reliant on the effectiveness of the underlying quantization techniques, which are especially crucial for the low-precision integer hardware designed to process binary, ternary, 4-bit or 8-bit networks.

Original: In other words, quantization is not only a method to reduce the memory footprint as in traditional work, but also a mandatory step to make the network deployable on integer hardware.
Revised: In other words, quantization is not only a method to reduce the memory footprint as in traditional work, but also a mandatory step to make the network deployable on integer hardware.

Original: Though there is a lot of prior work that investigates low-precision quantization, they mainly target on reducing the memory overhead caused by ﬂoating or high precision data representation in the networks, but not focus on specialized integer hardware for network inference.
Revised: Though there is a lot of prior work that investigates low-precision quantization, they mainly target on reducing the memory overhead caused by ﬂoating or high precision data representation in the networks, but not focus on specialized integer hardware for network inference.

Original: To enable the neural network processors to work with low-precision integer operands and minimize the accuracy losses, a good network quantizer must satisfy the constraints as enlisted in Table 1.
Revised: To enable the neural network processors to work with low-precision integer operands and minimize the accuracy losses, a good network quantizer must satisfy the constraints as enlisted in Table 1.

Original: First , all the parameters, including weights, bias, activations, partial results that eventually accumulate to an activation, and even the scaling factors, which are indispensable for low-precision networks like binary and ternary representation, must be quantized into low bitwidth integers as required by the underlying specialized hardware.
Revised: First , all the parameters, including weights, bias, activations, partial results that eventually accumulate to an activation, and even the scaling factors, which are indispensable for low-precision networks like binary and ternary representation, must be quantized into low bitwidth integers as required by the underlying specialized hardware.

Original: In some prior work (Zhou et al., 2016;
Revised: In some prior work (Zhou et al., 2016;

Original: Zhu et al., 2017; Zhang et al., 2018a;
Revised: Zhu et al., 2017; Zhang et al., 2018a;

Original: Mishra et al., 2018; Choi, 2018), they either leave bias and scaling factors unquantized or keep the ﬁrst and last layer in full or high precision.
Revised: Mishra et al., 2018; Choi, 2018), they either leave bias and scaling factors unquantized or keep the ﬁrst and last layer in full or high precision.

Original: Besides, some designs rely on high-precision internal register or ALUs to support high-precision partial results that are generated during computation before the ﬁnal output of activations or features.
Revised: Besides, some designs rely on high-precision internal register or ALUs to support high-precision partial results that are generated during computation before the ﬁnal output of activations or features.

Original: For example, Krishnamoorthi (2018), which quantizes the weights and activations to 8-bit, directly use 32-bit accumulators to cache the intermediate values or partial results to avoid overﬂows.
Revised: For example, Krishnamoorthi (2018), which quantizes the weights and activations to 8-bit, directly use 32-bit accumulators to cache the intermediate values or partial results to avoid overﬂows.

Original: However, for 4-bit and lower bitwidth, the integer accelerators cannot afford high bitwidth accumulators, which indicates higher silicon area and power cost.
Revised: However, for 4-bit and lower bitwidth, the integer accelerators cannot afford high bitwidth accumulators, which indicates higher silicon area and power cost.

Original: For integer-only-arithmetic, we quantize the bias to ﬁxed-point numbers by using a straight-forward method.
Revised: For integer-only-arithmetic, we quantize the bias to ﬁxed-point numbers by using a straight-forward method.

Original: The value range of these numbers is wide, resulting in overﬂows of the low bitwidth accumulators.
Revised: The value range of these numbers is wide, resulting in overﬂows of the low bitwidth accumulators.

Original: To overcome this problem, we quantize the bias to 8-bit and ﬁnetune the bias of the model.
Revised: To overcome this problem, we quantize the bias to 8-bit and ﬁnetune the bias of the model.

Original: As shown in Figure 1, the bitwidth of accumulators can be reduced to 16-bit.
Revised: As shown in Figure 1, the bitwidth of accumulators can be reduced to 16-bit.

Original: Second , the BatchNorm (BN) layer does not necessarily need to be processed during inference for the reduction of computation and memory cost.
Revised: Second , the BatchNorm (BN) layer does not necessarily need to be processed during inference for the reduction of computation and memory cost.

Original: For most of the convolutional neural networks, BN layers are often after the Conv or FC layers.
Revised: For most of the convolutional neural networks, BN layers are often after the Conv or FC layers.

Original: In these situations, BN can be merged into the weights and biases of the corresponding Conv or FC layers.
Revised: In these situations, BN can be merged into the weights and biases of the corresponding Conv or FC layers.

Original: However, in Zhou et al. (2016); Zhang et al.
Revised: However, in Zhou et al. (2016); Zhang et al.

Original: (2018a), they use asymmetric or non-linear quantization, causing barriers to BN fusion.
Revised: (2018a), they use asymmetric or non-linear quantization, causing barriers to BN fusion.

Original: There are two ways to overcome this obstacle.
Revised: There are two ways to overcome this obstacle.

Original: One is “ BN folded training”(Krishnamoorthi, 2018), which adopts BN fusion before weights quantization in every training step; the other is to use symmetric linear quantization.
Revised: One is “ BN folded training”(Krishnamoorthi, 2018), which adopts BN fusion before weights quantization in every training step; the other is to use symmetric linear quantization.

Original: However, the ﬁrst method doubles the training time, while the second one has no additional computational overhead, which will be introduced in Section 3.4.
Revised: However, the ﬁrst method doubles the training time, while the second one has no additional computational overhead, which will be introduced in Section 3.4.

Original: Third , linear quantization is necessary for state-of-the-art accelerators.
Revised: Third , linear quantization is necessary for state-of-the-art accelerators.

Original: There are many non-linear quantization methods which achieve excellent bitwidth reduction efﬁcacy and accuracy tradeoffs.
Revised: There are many non-linear quantization methods which achieve excellent bitwidth reduction efﬁcacy and accuracy tradeoffs.

Original: In these cases, it requires additional transformation to have correct arithmetic results after quantizing the value into non-linear distribution.
Revised: In these cases, it requires additional transformation to have correct arithmetic results after quantizing the value into non-linear distribution.

Original: For example, as in Han et al. (2016); Park et al.
Revised: For example, as in Han et al. (2016); Park et al.

Original: (2017), it necessitates the operation of table lookup to have correct multiplication between quantized values.
Revised: (2017), it necessitates the operation of table lookup to have correct multiplication between quantized values.

Original: However, the linear quantization can make full use of the low-precision arithmetic components in off-the-shelf accelerators.
Revised: However, the linear quantization can make full use of the low-precision arithmetic components in off-the-shelf accelerators.

Original: Further, linear quantization can be divided into symmetric mode and asymmetric mode.
Revised: Further, linear quantization can be divided into symmetric mode and asymmetric mode.

Original: Asymmetric quantization has one more parameter (e.g., zero-point (Krishnamoorthi, 2018)) than symmetric quantization, and it requires additional subtraction or linearoperation before multiplication.
Revised: Asymmetric quantization has one more parameter (e.g., zero-point (Krishnamoorthi, 2018)) than symmetric quantization, and it requires additional subtraction or linearoperation before multiplication.

Original: As a result, the symmetrical mode is compatible with the mainstream integer accelerator chip design and do not require the redesign of datapath in these hardware.
Revised: As a result, the symmetrical mode is compatible with the mainstream integer accelerator chip design and do not require the redesign of datapath in these hardware.

Original: Fourth , different CNNs or applications usually use a variety of activation functions.
Revised: Fourth , different CNNs or applications usually use a variety of activation functions.

Original: For instance, the object detection model Redmon et al.
Revised: For instance, the object detection model Redmon et al.

Original: (2016) typically uses Leaky ReLU .
Revised: (2016) typically uses Leaky ReLU .

Original: And the bottleneck of ResNet block does not use any activation function.
Revised: And the bottleneck of ResNet block does not use any activation function.

Original: The quantization methods are expected to be adapted to these situations.
Revised: The quantization methods are expected to be adapted to these situations.

Original: However, Zhang et al.
Revised: However, Zhang et al.

Original: (2018a)
Revised: (2018a)

Original: ; Park et al.
Revised: ; Park et al.

Original: (2017) only focus on the quantization of activations after ReLU .
Revised: (2017) only focus on the quantization of activations after ReLU .

Original: In this paper, we demonstrate our method is friendly to different activation methods such as Leaky ReLU .
Revised: In this paper, we demonstrate our method is friendly to different activation methods such as Leaky ReLU .

Original: Some of the previous researches change the network structure for better quantization performance, e.g., Mishra et al.
Revised: Some of the previous researches change the network structure for better quantization performance, e.g., Mishra et al.

Original: (2018) double or even triple the convolutional ﬁlters to reduce accuracy degradation.
Revised: (2018) double or even triple the convolutional ﬁlters to reduce accuracy degradation.

Original: For the energy-efﬁcient integer neural network chips, it needs to remap the changed network architecture to hardware and adds to computational and memory access overhead due to the increased ﬁlters and parameters.
Revised: For the energy-efﬁcient integer neural network chips, it needs to remap the changed network architecture to hardware and adds to computational and memory access overhead due to the increased ﬁlters and parameters.

Original: As a result, keeping the network structure intact is important.
Revised: As a result, keeping the network structure intact is important.

Original: Concerning all the factors above, in this paper, we present a learned linear symmetric quantization (LLSQ) method and also evaluate it on a low-precision neural network accelerator through hardware-software co-design.
Revised: Concerning all the factors above, in this paper, we present a learned linear symmetric quantization (LLSQ) method and also evaluate it on a low-precision neural network accelerator through hardware-software co-design.

Original: Speciﬁcally, our mainly contributions are:
Revised: Speciﬁcally, our mainly contributions are:

Original: • Unlike most of other quantization methods, we quantize the whole network including the ﬁrst and last layers.
Revised: • Unlike most of other quantization methods, we quantize the whole network including the ﬁrst and last layers.

Original: We also quantize bias and scaling factors, in support of the low bitwidth integer arithmetic units and accumulators on the accelerator.
Revised: We also quantize bias and scaling factors, in support of the low bitwidth integer arithmetic units and accumulators on the accelerator.

Original: • We adopt learned linear symmetric quantization schemes which are hardware friendly (such as the convenience of BN fusion implementation) while achieving state-of-the-art prediction accuracy.
Revised: • We adopt learned linear symmetric quantization schemes which are hardware friendly (such as the convenience of BN fusion implementation) while achieving state-of-the-art prediction accuracy.

Original: • We design a specialized low-precision CNN inference accelerator to validate the methodology, which supports 2/4/8 integer operating and work with high efﬁciency.
Revised: • We design a specialized low-precision CNN inference accelerator to validate the methodology, which supports 2/4/8 integer operating and work with high efﬁciency.

Original: We then deploy our quantization model on the accelerator to illustrate the efﬁcacy of the workﬂow.
Revised: We then deploy our quantization model on the accelerator to illustrate the efﬁcacy of the workﬂow.

Original:  M OTIVATION
Revised:  M OTIVATION

Original: Edge or embedded neural network accelerators generally have three primary design goals— smallfootprint, high-throughput/low-latency, and low-power.
Revised: Edge or embedded neural network accelerators generally have three primary design goals— smallfootprint, high-throughput/low-latency, and low-power.

Original: For different applications and scenarios, the prior researches on specialized deep learning processors are often falling into different categories: cloud-oriented hardware for warehouse machines, low power mobile processors and ultra-low power accelerators for IoT or cyber-physical devices.
Revised: For different applications and scenarios, the prior researches on specialized deep learning processors are often falling into different categories: cloud-oriented hardware for warehouse machines, low power mobile processors and ultra-low power accelerators for IoT or cyber-physical devices.

Original: For mobile and embedded usage, specialized neural network processors are becoming increasingly popular as an efﬁcient hardware solution of inference.
Revised: For mobile and embedded usage, specialized neural network processors are becoming increasingly popular as an efﬁcient hardware solution of inference.

Original: DianNao (Chen et al., 2014) is proposed for fast inference of DNNs and it uses 16-bit ﬁxed-point multipliers for small silicon area and low- energy.
Revised: DianNao (Chen et al., 2014) is proposed for fast inference of DNNs and it uses 16-bit ﬁxed-point multipliers for small silicon area and lowenergy.

Original: Later, ShiDianNao (Du et al., 2015) is introduced and it burns extremely low energy consumption by putting all weights onto the SRAM to eliminate considerable DRAM accesses.
Revised: Later, ShiDianNao (Du et al., 2015) is introduced and it burns extremely low energy consumption by putting all weights onto the SRAM to eliminate considerable DRAM accesses.

Revised: Besides,

Revised: DeepBurning (Wang et al., 2016) simpliﬁes the design ﬂow of accelerator for different NN models.

Original: Eyeriss (Chen et al., 2018) is also another representative of low-power accelerators.
Revised: Eyeriss (Chen et al., 2018) is also another representative of low-power accelerators.

Original: And it presents a row-stationary (RS) dataﬂow to minimize data movement energy consumption on a spatial architecture.
Revised: And it presents a row-stationary (RS) dataﬂow to minimize data movement energy consumption on a spatial architecture.

Original: To further reduce computation overhead, EIE (Han et al., 2016) exploits the sparsity and low-bit compression of the NNs and achieves better throughput, energy and area efﬁciency.
Revised: To further reduce computation overhead, EIE (Han et al., 2016) exploits the sparsity and low-bit compression of the NNs and achieves better throughput, energy and area efﬁciency.

Original: These typical edge neural network processors are accepting ﬁxed-point data input and using ﬁxed-point processing elements to reduce the power and chip area overhead caused by ﬂoating-point arithmetic components and memory.
Revised: These typical edge neural network processors are accepting ﬁxed-point data input and using ﬁxed-point processing elements to reduce the power and chip area overhead caused by ﬂoating-point arithmetic components and memory.

Original: For the cloud scenarios, specialized architectures like TPU (Jouppi et al., 2017) and FPGA-based accelerator cards are also replacing conventional GPGPU and CPU for highthroughput inference tasks.
Revised: For the cloud scenarios, specialized architectures like TPU (Jouppi et al., 2017) and FPGA-based accelerator cards are also replacing conventional GPGPU and CPU for highthroughput inference tasks.

Original: Even for cloud-oriented inference architectures, ﬁxed-point processing architectures like TPU are favored because they are able to deliver much higher throughput for the given power budget and silicon area overhead.
Revised: Even for cloud-oriented inference architectures, ﬁxed-point processing architectures like TPU are favored because they are able to deliver much higher throughput for the given power budget and silicon area overhead.

Original: However, for the ﬁxed-point or integer hardware targeted on neural network acceleration, quantization is prerequisite to convert the ﬂoating-point network model into the ﬁxed-point format compatible with the specialized hardware, and it is also a critical step to ensure the accuracy of the network after conversion.
Revised: However, for the ﬁxed-point or integer hardware targeted on neural network acceleration, quantization is prerequisite to convert the ﬂoating-point network model into the ﬁxed-point format compatible with the specialized hardware, and it is also a critical step to ensure the accuracy of the network after conversion.

Original: Many prior quantization methods are intended to reduce the running overhead of networks but ignore the architecture and working mechanism of integer neural network processors, as illustrated in Table 1, and they sometimes face considerable accuracy losses, or performance penalty or even fail to be supported on the realistic integer datapath due to the unconsciousness of the underlying hardware.
Revised: Many prior quantization methods are intended to reduce the running overhead of networks but ignore the architecture and working mechanism of integer neural network processors, as illustrated in Table 1, and they sometimes face considerable accuracy losses, or performance penalty or even fail to be supported on the realistic integer datapath due to the unconsciousness of the underlying hardware.

Original: This problem becomes particularly important for the hardware that is designed to run low bitwidth networks such as binary, ternary, and 2/4-bit models.
Revised: This problem becomes particularly important for the hardware that is designed to run low bitwidth networks such as binary, ternary, and 2/4-bit models.

Original: For instance, Deep compression and WQ are clustering-based quantization methods, and they still need high-precision values to represent the weights, bias, and activations.
Revised: For instance, Deep compression and WQ are clustering-based quantization methods, and they still need high-precision values to represent the weights, bias, and activations.

Original: As a result, they are not compatible with the hardware that only supports low-precision computing.
Revised: As a result, they are not compatible with the hardware that only supports low-precision computing.

Original: LQ-Nets uses non-linear quantization based on the binary code and basis vector, and it can theoretically calculate the inner products between quantized weights and activations by bitwise operations only.
Revised: LQ-Nets uses non-linear quantization based on the binary code and basis vector, and it can theoretically calculate the inner products between quantized weights and activations by bitwise operations only.

Original: However, it requires intensive modiﬁcations to the design of current processors by adding a lot of look-up tables in the datapath.
Revised: However, it requires intensive modiﬁcations to the design of current processors by adding a lot of look-up tables in the datapath.

Original: Further, bias and scaling factors are not quantized in PACT and WRPN, resulting in performance penalty when employing additional high-precision or ﬂoat-point ALUs to deal with them.
Revised: Further, bias and scaling factors are not quantized in PACT and WRPN, resulting in performance penalty when employing additional high-precision or ﬂoat-point ALUs to deal with them.

Original: In contrast, our
Revised: In contrast, our

Original: LLSQ is designed to ease the model quantization ﬂow for the specialized integer neural network processors by conforming to the constraints speciﬁed in Table 1.
Revised: LLSQ is designed to ease the model quantization ﬂow for the specialized integer neural network processors by conforming to the constraints speciﬁed in Table 1.

Original: To validate the importance of hardware-aware quantizer and software/hardware co-design, we also design a specialized CNN accelerator for wearable applications.
Revised: To validate the importance of hardware-aware quantizer and software/hardware co-design, we also design a specialized CNN accelerator for wearable applications.

Original: And the specialized accelerator supports 2/4/8 integer operation and adopts the dataﬂow of low latency and energy design.
Revised: And the specialized accelerator supports 2/4/8 integer operation and adopts the dataﬂow of low latency and energy design.

Original:  N ETWORKS WITH L EARNED L INEAR S YMMETRIC Q UANTIZATION
Revised:  N ETWORKS WITH L EARNED L INEAR S YMMETRIC Q UANTIZATION

Original: In this section, we ﬁrstly give the overview of the proposed quantization scheme.
Revised: In this section, we ﬁrstly give the overview of the proposed quantization scheme.

Original: Then we detail the scheme including low-precision representation, quantized network training, and the deployment of quantization model on our specialized integer-only CNN accelerator for fast inference.
Revised: Then we detail the scheme including low-precision representation, quantized network training, and the deployment of quantization model on our specialized integer-only CNN accelerator for fast inference.

Original: 3. O VERVIEW OF LLSQ
Revised: 3. O VERVIEW OF LLSQ

Original: Many of the previous researches focus on the quantization-aware training in GPU, showing the potential of low-bit quantization on CNNs.
Revised: Many of the previous researches focus on the quantization-aware training in GPU, showing the potential of low-bit quantization on CNNs.

Original: Han et al.
Revised: Han et al.

Original: ; Park et al.
Revised: ; Park et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018a)
Revised: (2018a)

Original: propose non-linear quantization methods but lacks of a detailed description of the hardware feasibility.
Revised: propose non-linear quantization methods but lacks of a detailed description of the hardware feasibility.

Original: Krishnamoorthi (2018) provides a quantization scheme that quantizes weights and activations into 8-bit integers and integer-arithmetic-only implementation on ARM CPUs.
Revised: Krishnamoorthi (2018) provides a quantization scheme that quantizes weights and activations into 8-bit integers and integer-arithmetic-only implementation on ARM CPUs.

Original: The method achieves evident hardware acceleration effects, but does not fully exploit lower-precision quantization.
Revised: The method achieves evident hardware acceleration effects, but does not fully exploit lower-precision quantization.

Original: Based on the researches, we propose a quantization scheme for state-of-the-art specialized accelerators operating on low-precision integers only.
Revised: Based on the researches, we propose a quantization scheme for state-of-the-art specialized accelerators operating on low-precision integers only.

Original: Figure 1 shows an overview of the proposed scheme.
Revised: Figure 1 shows an overview of the proposed scheme.

Original: Compared with prior work, our proposed quantization scheme pays more attention to the constraints imposed by real hardware.

Original: We use a uniﬁed learned linear symmetric quantizer to quantize weights and activations.

Original: And the quantizer has only one parameter, known as the scaling factor.

Original: Linear symmetric quantization consumes little additional resources based on the mainstream integer accelerator designs while achieving state-of-the-art accuracy in various networks.

Original: After that, we adopt

Original: Train W
Revised: Train W

Original: Quan of W and A
Revised: Quan of W and A

Original: BN Fusion
Revised: BN Fusion

Original: Quan of B and S
Revised: Quan of B and S

Original: Scale Bu ﬀ er
Revised: Scale Bu ﬀ er

Original: W e i g h t B
Revised: W e i g h t B

Original: u ﬀ e r + psums …… …… ……
Revised: u ﬀ e r + psums …… …… ……

Original: M c x 4 b i t
Revised: M c x 4 b i t

Original: Mc x 4bit
Revised: Mc x 4bit

Original: Activation Bu ﬀ er
Revised: Activation Bu ﬀ er

Original: X
Revised: X

Original: X ……  + psums 16bit 16bit
Revised: X ……  + psums 16bit 16bit

Original: X
Revised: X

Original: X 8bit >> 24bit 24bit >> shift_B
Revised: X 8bit >> 24bit 24bit >> shift_B

Original: Mc x 4bit Mc =
Revised: Mc x 4bit Mc =

Original: MAC 1 <latexit sha1_base64="SMQKRqSFQLr/LkCKTnxefoU56sw=">AAAB7HicbVBNSwMxEJ3Ur1q/qh69BIvgqexWQY/VXrwIFdy20C4lm2bb0Gx2SbJCWfobvHhQxKs/yJv/xrTdg7Y+GHi8N8PMvCARXBvH+UaFtfWNza3idmlnd2//oHx41NJxqijzaCxi1QmIZoJL5hluBOskipEoEKwdjBszv/3ElOaxfDSThPkRGUoeckqMlbz7m0bf7ZcrTtWZA68SNycVyNHsl796g5imEZOGCqJ113US42dEGU4Fm5Z6qWYJoWMyZF1LJYmY9rP5sVN8ZpUBDmNlSxo8V39PZCTSehIFtjMiZqSXvZn4n9dNTXjtZ1wmqWGSLhaFqcAmxrPP8YArRo2YWEKo4vZWTEdEEWpsPiUbgrv88ipp1aruRbX2cFmp3+ZxFOEETuEcXLiCOtxBEzygwOEZXuENSfSC3tHHorWA8plj+AP0+QPfQY4R</latexit>
Revised: MAC 1 <latexit sha1_base64="SMQKRqSFQLr/LkCKTnxefoU56sw=">AAAB7HicbVBNSwMxEJ3Ur1q/qh69BIvgqexWQY/VXrwIFdy20C4lm2bb0Gx2SbJCWfobvHhQxKs/yJv/xrTdg7Y+GHi8N8PMvCARXBvH+UaFtfWNza3idmlnd2//oHx41NJxqijzaCxi1QmIZoJL5hluBOskipEoEKwdjBszv/3ElOaxfDSThPkRGUoeckqMlbz7m0bf7ZcrTtWZA68SNycVyNHsl796g5imEZOGCqJ113US42dEGU4Fm5Z6qWYJoWMyZF1LJYmY9rP5sVN8ZpUBDmNlSxo8V39PZCTSehIFtjMiZqSXvZn4n9dNTXjtZ1wmqWGSLhaFqcAmxrPP8YArRo2YWEKo4vZWTEdEEWpsPiUbgrv88ipp1aruRbX2cFmp3+ZxFOEETuEcXLiCOtxBEzygwOEZXuENSfSC3tHHorWA8plj+AP0+QPfQY4R</latexit>

Original: MAC M c <latexit sha1_base64="DD/zFcLWTfxWvbYo4aPkOxWnyP4=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRbBU9mtgh6rvXgpVLAf0i5LNk3b0CS7JFmhLP0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhhzpo3rfju5tfWNza38dmFnd2//oHh41NJRoghtkohHqhNiTTmTtGmY4bQTK4pFyGk7HNdmfvuJKs0i+WAmMfUFHko2YAQbKz3Wb2pBWg/INCiW3LI7B1olXkZKkKERFL96/YgkgkpDONa667mx8VOsDCOcTgu9RNMYkzEe0q6lEguq/XR+8BSdWaWPBpGyJQ2aq78nUiy0nojQdgpsRnrZm4n/ed3EDK79lMk4MVSSxaJBwpGJ0Ox71GeKEsMnlmCimL0VkRFWmBibUcGG4C2/vEpalbJ3Ua7cX5aqt1kceTiBUzgHD66gCnfQgCYQEPAMr/DmKOfFeXc+Fq05J5s5hj9wPn8AREWQDw==</latexit>
Revised: MAC M c <latexit sha1_base64="DD/zFcLWTfxWvbYo4aPkOxWnyP4=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRbBU9mtgh6rvXgpVLAf0i5LNk3b0CS7JFmhLP0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhhzpo3rfju5tfWNza38dmFnd2//oHh41NJRoghtkohHqhNiTTmTtGmY4bQTK4pFyGk7HNdmfvuJKs0i+WAmMfUFHko2YAQbKz3Wb2pBWg/INCiW3LI7B1olXkZKkKERFL96/YgkgkpDONa667mx8VOsDCOcTgu9RNMYkzEe0q6lEguq/XR+8BSdWaWPBpGyJQ2aq78nUiy0nojQdgpsRnrZm4n/ed3EDK79lMk4MVSSxaJBwpGJ0Ox71GeKEsMnlmCimL0VkRFWmBibUcGG4C2/vEpalbJ3Ua7cX5aqt1kceTiBUzgHD66gCnfQgCYQEPAMr/DmKOfFeXc+Fq05J5s5hj9wPn8AREWQDw==</latexit>

Original: B i a s B u ﬀ er
Revised: B i a s B [Figure] e

Original: PE
Revised: [Figure]

Original: PE
Revised: PE

Original: [Figure]
Revised: [Figure]

Original: PE
Revised: PE

Original: [Figure]
Revised: [Figure]

Original: R e t r a i n i n g R e t r a i n i n g
Revised: R e t r a i n i n g R e t r a i n i n g

Original: [Figure]
Revised: [Figure]

Revised: Compared with prior work, our proposed quantization scheme pays more attention to the constraints imposed by real hardware.

Revised: We use a uniﬁed learned linear symmetric quantizer to quantize weights and activations.

Revised: And the quantizer has only one parameter, known as the scaling factor.

Revised: Linear symmetric quantization consumes little additional resources based on the mainstream integer accelerator designs while achieving state-of-the-art accuracy in various networks.

Revised: After that, we adopt

Original: BN fusion for fast inference on hardware.
Revised: BN fusion for fast inference on hardware.

Original: As for bias and scaling factors, we also quantize them to low-bitwidth integers.
Revised: As for bias and scaling factors, we also quantize them to low-bitwidth integers.

Original: The integer accelerator illustrated in Figure 1 is an illustrative case of 4-bit quantization and hardware acceleration.
Revised: The integer accelerator illustrated in Figure 1 is an illustrative case of 4-bit quantization and hardware acceleration.

Original: 3. M AKING F ULL U SE OF THE P RE -T RAINED P ARAMETERS
Revised: 3. M AKING F ULL U SE OF THE P RE -T RAINED P ARAMETERS

Original: In experiments, we ﬁnd that it is more efﬁcient to start with the pre-trained full-precision parameters before quantization.
Revised: In experiments, we ﬁnd that it is more efﬁcient to start with the pre-trained full-precision parameters before quantization.

Original: Louizos et al.
Revised: Louizos et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2017) use pre-trained weights for fast convergence and deployment, while Zhang et al.
Revised: (2017) use pre-trained weights for fast convergence and deployment, while Zhang et al.

Original: (2018a);
Revised: (2018a);

Original: Choi (2018);
Revised: Choi (2018);

Original: Cai et al.
Revised: Cai et al.

Original: (2017) train quantized network from scratch to show the robustness of the algorithm.
Revised: (2017) train quantized network from scratch to show the robustness of the algorithm.

Original: However, for some object detection models, the backbone models and pre-trained weights are essential to the detection performance.
Revised: However, for some object detection models, the backbone models and pre-trained weights are essential to the detection performance.

Original: Redmon et al.
Revised: Redmon et al.

Original: (2016) shows that the pre-trained high-resolution classiﬁcation network gives an increase of almost 4% mAP .
Revised: (2016) shows that the pre-trained high-resolution classiﬁcation network gives an increase of almost 4% mAP .

Original: To have better performance in classiﬁcation, object detection, and other
Revised: To have better performance in classiﬁcation, object detection, and other

Original: CNN based tasks, in this paper, we use pre-trained parameters to initialize the networks.
Revised: CNN based tasks, in this paper, we use pre-trained parameters to initialize the networks.

Original: 3. L OW - PRECISION R EPRESENTATION AND Q UANTIZATION A LGORITHM
Revised: 3. L OW - PRECISION R EPRESENTATION AND Q UANTIZATION A LGORITHM

Original: We use channel-wise quantization for Conv layers and layer-wise quantization for FC layers and activations.
Revised: We use channel-wise quantization for Conv layers and layer-wise quantization for FC layers and activations.

Original: And we adopt the symmetric linear quantization to quantize weights or activations into k bits words(e.g.
Revised: And we adopt the symmetric linear quantization to quantize weights or activations into k bits words(e.g.

Original: , 4-bit), which can be deﬁned as
Revised: , 4-bit), which can be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: [Equation] where x r ∈ R is one kernel of weights or one layer of activations, the variable α ∈ R + is the quantization parameter, known as the scaling factor, while q ∈ {− 2 k − 1 , . .
Revised: (1) where x r ∈ R is one kernel of weights or one layer of activations, the variable α ∈ R + is the quantization parameter, known as the scaling factor, while q ∈ {− 2 k − 1 , . .

Original: , 2 k − 1 − 1 } is the integer values ﬂowing in the integer accelerator and x q ∈ {− 2 k − 1 α, . . .
Revised: , 2 k − 1 − 1 } is the integer values ﬂowing in the integer accelerator and x q ∈ {− 2 k − 1 α, . . .

Original: , 0 , α, . . .
Revised: , 0 , α, . . .

Original: , (2 k − 1 − 1) α } is the quantized weights or activations.
Revised: , (2 k − 1 − 1) α } is the quantized weights or activations.

Original: Note that for activations, which are non-negative values if the ActFun is ReLU , we clamp them to [0 , 2 k − 1] , resulting in q ∈ { 0 , 1 , . . .
Revised: Note that for activations, which are non-negative values if the ActFun is ReLU , we clamp them to [0 , 2 k − 1] , resulting in q ∈ { 0 , 1 , . . .

Original: , 2 k − 1 } and x q ∈ { 0 , α, . . .
Revised: , 2 k − 1 } and x q ∈ { 0 , α, . . .

Original: , (2 k − 1 − 1) α } , respectively.
Revised: , (2 k − 1 − 1) α } , respectively.

Original: As deﬁned above, we use α as our quantization parameter.
Revised: As deﬁned above, we use α as our quantization parameter.

Original: And we optimize it with:
Revised: And we optimize it with:

Original: [Equation]
Revised: [Equation]

Original:  where x r , x q are the same factors deﬁned in Equation 1, p ( x r ) is the probability density distribution of x r , and l ∈ { 1 , 2 } is an optional constraint (We use 2 in our experiments).
Revised: [Table] where x r , x q are the same factors deﬁned in Equation 1, p ( x r ) is the probability density distribution of x r , and l ∈ { 1 , 2 } is an optional constraint (We use 2 in our experiments).

Original: In Figure 2, we present the relationship between quantization error and α .
Revised: In Figure 2, we present the relationship between quantization error and α .

Original: When ﬁxing weights x r , we can ﬁnd the optimal[Table] α ∗ by using the brute-force search approach, which induces high computation cost.
Revised: When ﬁxing weights x r , we can ﬁnd the optimal α ∗ by using the brute-force search approach, which induces high computation cost.

Original: Besides, the weights are updated during the re-training phase and the optimal value α ∗ changes accordingly.
Revised: Besides, the weights are updated during the re-training phase and the optimal value α ∗ changes accordingly.

Original: In other words, the optimal value for the factors is not ﬁxed and it takes considerable computational overhead to ﬁnd the dynamic optimal value
Revised: In other words, the optimal value for the factors is not ﬁxed and it takes considerable computational overhead to ﬁnd the dynamic optimal value. [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: /8zqJ7l/7GRNxoqkg80X9hNs6sqfP2z0mKdE8NQSJZOZWmwxRItEmooIJwV1+eZU0qxX3slK9N2ncwBx5OIESnIELV1CDO6hDAwhweIY3eLcerVfrw/qct+asxcwx/IH19QOFVpKb</latexit>  1 ↵ ⇤ <latexit sha1_base64="//DoCojG0Ru9MRXMPcMnJet6zVI=">AAACAXicbVC7SgNBFJ31GeNr1ULFZjAIYhF2Y2HKoBaWEcwDsjHcncwmQ2YfzMwKYVkbf8XGQhFbf0LstLH1M5w8Ck08cOFwzr3ce48bcSaVZX0YM7Nz8wuLmaXs8srq2rq5sVmVYSwIrZCQh6LugqScBbSimOK0HgkKvstpze2dDfzaDRWShcGV6ke06UMnYB4joLTUMnecc8oVOJ4Akthp4gCPunB9lLbMnJW3hsDTxB6TXKn49bb9+b1bbpnvTjsksU8DRThI2bCtSDUTEIoRTtOsE0saAelBhzY0DcCnspkMP0jxgVba2AuFrkDhofp7IgFfyr7v6k4fVFdOegPxP68RK6/YTFgQxYoGZLTIizlWIR7EgdtMUKJ4XxMggulbMemCDkPp0LI6BHvy5WlSLeTt43zhUqdxikbIoD20jw6RjU5QCV2gMqoggm7RPXpET8ad8WA8Gy+j1hljPLOF/sB4/QH9PZr
Revised: I6iN1HYvPOTxELvFslNxZrBXibsg5VrJO3+Z1NJ6t/jt9SKShFRowlGpjuvE2s9QakY4HRe8RNEYyQgHtGOowJAqP5vdO7ZPjdKz [Figure] [Figure] eZU0qxX3slK9N2ncwBx5OIESnIELV1CDO6hDAwhweIY3eLcerVfrw/qct

Original: [Figure]
Revised: [Figure]

Revised: asxcwx

Revised: [Figure]

Revised: IH19QOFVpKb

Revised: [Figure]

Revised: latexit

Revised: [Figure]

Original: Inspired by Zhang et al.
Revised: Inspired by Zhang et al.

Original: (2018a), we ﬁnd through experiments that there is no need to ﬁnd the optimal value α ∗ , and it works well enough to ﬁnd a near-optimal value ˜ α ∗ .
Revised: (2018a), we ﬁnd through experiments that there is no need to ﬁnd the optimal value α ∗ , and it works well enough to ﬁnd a near-optimal value ˜ α ∗ .

Original: Generally, quantization can be considered as a regularization of the networks, and the quantization parameter α needs only to be adjusted to a near-optimal value to preserve the network capacity.
Revised: Generally, quantization can be considered as a regularization of the networks, and the quantization parameter α needs only to be adjusted to a near-optimal value to preserve the network capacity.

Original: Then the problem becomes how to ﬁnd a ˜ α ∗ in the forward pass of the network training phase.
Revised: Then the problem becomes how to ﬁnd a ˜ α ∗ in the forward pass of the network training phase.

Original: At the beginning of training, we assign α an initial value.
Revised: At the beginning of training, we assign α an initial value.

Original: Then in every training iteration, we explore between 2 α and α/to ﬁnd a better search direction d better ∈ {− 1 , 0 , 1 } , and use − α 2 d better as the simulated gradient (SG) of α which is detailed in Equation 9.
Revised: Then in every training iteration, we explore between 2 α and α/to ﬁnd a better search direction d better ∈ {− 1 , 0 , 1 } , and use − α 2 d better as the simulated gradient (SG) of α which is detailed in Equation 9.

Original: The gradients of other parameters are still obtained by backpropagation algorithm.
Revised: The gradients of other parameters are still obtained by backpropagation algorithm.

Original: After that, we update all parameters with the gradients or simulated gradients.
Revised: After that, we update all parameters with the gradients or simulated gradients.

Original: Another method is updating α by the exponential moving average (EMA).
Revised: Another method is updating α by the exponential moving average (EMA).

Original: We experiment both of the methods, and the results show that SG is generally better than EMA on various networks (See Table 2).
Revised: We experiment both of the methods, and the results show that SG is generally better than EMA on various networks (See Table 2).

Original: If not speciﬁcally stated, we use the SG method in experiments.
Revised: If not speciﬁcally stated, we use the SG method in experiments.

Original: The re-training process with weights and activations quantized is summarized in Step 1 of Algorithm 1.
Revised: The re-training process with weights and activations quantized is summarized in Step 1 of Algorithm 1.

Original: 3. BN L AYER F USION OF Q UANTIZED N ETWORKS
Revised: 3. BN L AYER F USION OF Q UANTIZED N ETWORKS

Original: As described in Section 1, merging the BN layers into convolutional layers can reduce the latency of network inference by removing additional computation overhead.
Revised: As described in Section 1, merging the BN layers into convolutional layers can reduce the latency of network inference by removing additional computation overhead.

Original: The operator of quantized Convand FC layers can be expressed as
Revised: The operator of quantized Convand FC layers can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where α, q are the same as Equation 1, α a q a , α w q w donate the quantized activations and weights, while b is the bias and o is the output feature vector.
Revised: where α, q are the same as Equation 1, α a q a , α w q w donate the quantized activations and weights, while b is the bias and o is the output feature vector.

Original: Note that α a , α w and b are full precision values.
Revised: Note that α a , α w and b are full precision values.

Original: And the BN layer can be formulated as follows:
Revised: And the BN layer can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where µ and σ 2 are EMA statistics, γ and β are learned parameters in BN layers.
Revised: where µ and σ 2 are EMA statistics, γ and β are learned parameters in BN layers.

Original: Obviously, we can merge BN layers and ﬁgure out the corrected parameters:
Revised: Obviously, we can merge BN layers and ﬁgure out the corrected parameters:

Original: [Equation]
Revised: [Equation]

Original: 3. B IAS AND S CALING F ACTOR Q UANTIZATION FOR L OW - BIT I NTEGER O NLY A RITHMETIC
Revised: 3. B IAS AND S CALING F ACTOR Q UANTIZATION FOR L OW - BIT I NTEGER O NLY A RITHMETIC

Original: Further, the outputs of layers are quantized according to Equation 1.
Revised: Further, the outputs of layers are quantized according to Equation 1.

Original: For integer-only-arithmetic, the bias use α a α w as its scaling factor.
Revised: For integer-only-arithmetic, the bias use α a α w as its scaling factor.

Original: And for the multiplier α a α w α o , we use bit-shift quantization (See Equation 10) so that no multiplication but bit-shift operation is needed in hardware.
Revised: And for the multiplier α a α w α o , we use bit-shift quantization (See Equation 10) so that no multiplication but bit-shift operation is needed in hardware.

Original: [Equation]
Revised: [Equation]

Original: where q
Revised: where q

Original: [Equation]
Revised: [Equation]

Original: = clamp
Revised: = clamp

Original: [Equation]
Revised: [Equation]

Original: Note that α a α w is a very small number, resulting in large quantization noise when adopting the clamp operation.
Revised: Note that α a α w is a very small number, resulting in large quantization noise when adopting the clamp operation.

Original: In addition, the quantization of the scaling factors α can also raise the quantization noise of weights and activations.
Revised: In addition, the quantization of the scaling factors α can also raise the quantization noise of weights and activations.

Original: Parameter re-training summarized in Step 2 of Algorithm 1 is required.
Revised: Parameter re-training summarized in Step 2 of Algorithm 1 is required.

Original: In the re-training phase, we adopt STE (Bengio et al., 2013) to realize the non-differentiable quantization function.
Revised: In the re-training phase, we adopt STE (Bengio et al., 2013) to realize the non-differentiable quantization function.

Original: For weights and bias, we have
Revised: For weights and bias, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For activations, we have∂y
Revised: For activations, we have

Original: [Equation]
Revised: [Equation]

Original: otherwise (8)
Revised: otherwise (8)

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, three sets of experiments on Cifar10, ImageNet and Pascal VOC datasets are presented.
Revised: In this section, three sets of experiments on Cifar10, ImageNet and Pascal VOC datasets are presented.

Original: First, we conduct our proposed learned linear symmetric quantization (LLSQ) on weights and activations, leaving the ﬁrst and last layers in full precision for a fair comparison with Zhang et al.
Revised: First, we conduct our proposed learned linear symmetric quantization (LLSQ) on weights and activations, leaving the ﬁrst and last layers in full precision for a fair comparison with Zhang et al.

Original: (2018a).
Revised: (2018a).

Original: Second, we quantize the whole networks including the ﬁrst and last layers, which is referred as LLSQF (LLSQ for Full network).
Revised: Second, we quantize the whole networks including the ﬁrst and last layers, which is referred as LLSQF (LLSQ for Full network).

Original: Finally, we quantize the remaining bias and scaling factors.
Revised: Finally, we quantize the remaining bias and scaling factors.

Original: LLSQ is implemented in PyTorch (Paszke et al., 2017), and most of the baselines it uses in evaluation are from PyTorch Model Zoo 2 .
Revised: LLSQ is implemented in PyTorch (Paszke et al., 2017), and most of the baselines it uses in evaluation are from PyTorch Model Zoo 2 .

Original: 4. Q UANTIZATION OF W EIGHTS AND A CTIVATIONS
Revised: 4. Q UANTIZATION OF W EIGHTS AND A CTIVATIONS

Original: We ﬁrstly employ the VGG-Small network on Cifar10 to verify the LLSQ method.
Revised: We ﬁrstly employ the VGG-Small network on Cifar10 to verify the LLSQ method.

Original: After that, we use AlexNet (Krizhevsky et al., 2012), ResNet18, ResNet34 (He et al., 2016), particularly the lightweight and hard-to-compress network architecture of MobileNet (Howard et al., 2017; Sandler et al., 2018) etc. to conduct more detailed experiments on the ImageNet dataset.
Revised: After that, we use AlexNet (Krizhevsky et al., 2012), ResNet18, ResNet34 (He et al., 2016), particularly the lightweight and hard-to-compress network architecture of MobileNet (Howard et al., 2017; Sandler et al., 2018) etc. to conduct more detailed experiments on the ImageNet dataset.

Original: Finally, we also quantize
Revised: Finally, we also quantize

Original: YOLOv2 (Redmon & Farhadi, 2017) to demonstrate that LLSQ also works well for complicated applications and especially the task adopting the activation functions like Leaky ReLU other than
Revised: YOLOv2 (Redmon & Farhadi, 2017) to demonstrate that LLSQ also works well for complicated applications and especially the task adopting the activation functions like Leaky ReLU other than

Original: ReLU used in previous work.
Revised: ReLU used in previous work.

Original: VGG-S MALL ON C IFAR
Revised: VGG-S MALL ON C IFAR

Original: The VGG-Small architecture is the same with Louizos et al.
Revised: The VGG-Small architecture is the same with Louizos et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018a), consisting of six Conv layers, three MaxPool layers, and one FC layer.
Revised: (2018a), consisting of six Conv layers, three MaxPool layers, and one FC layer.

Original: We adopt a cosine learn rate scheduler to train the VGG-Small reference and the quantized models.
Revised: We adopt a cosine learn rate scheduler to train the VGG-Small reference and the quantized models.

Original: Speciﬁcally, we train the reference network for 400 epochs using an initial learning rate of 2e-2.
Revised: Speciﬁcally, we train the reference network for 400 epochs using an initial learning rate of 2e-2.

Original: And for the training of the quantized network, we use a warmup learning rate scheduler in the ﬁrst ten epochs with an initial learning rate of 2e-3.
Revised: And for the training of the quantized network, we use a warmup learning rate scheduler in the ﬁrst ten epochs with an initial learning rate of 2e-3.

Original: In all quantization experiments, the total training epochs are 100.
Revised: In all quantization experiments, the total training epochs are 100.

Original: The VGG-Small quantization results are provided in Table 3.
Revised: The VGG-Small quantization results are provided in Table 3.

Original: With 3-bit weights and 3-bit activations, the accuracy
Revised: With 3-bit weights and 3-bit activations, the accuracy

Original: [Table]
Revised: [Table]

Original: using our method is better than state-of-the-art method
Revised: using our method is better than state-of-the-art method

Original: [Table]
Revised: [Table]

Original: LQ-Nets.
Revised: LQ-Nets.

Original: And even when the ﬁrst and last layers are all quantized in the same way, the loss of accuracy is minimal
Revised: And even when the ﬁrst and last layers are all quantized in the same way, the loss of accuracy is minimal

Original: [Table]
Revised: [Table]

Original: I MAGE N ET D ATASET
Revised: I MAGE N ET D ATASET

Original: We then quantize AlexNet, ResNet18, ResNet34 and MobileNetv2 on ILSVRC2012 dataset with different bitwidth conﬁguration to demonstrate the effectiveness of the method.
Revised: We then quantize AlexNet, ResNet18, ResNet34 and MobileNetv2 on ILSVRC2012 dataset with different bitwidth conﬁguration to demonstrate the effectiveness of the method.

Original: All of the pretrained ﬂoat-point weights except MobileNetv2 3 are downloaded from the PyTorch Model Zoo
Revised: All of the pretrained ﬂoat-point weights except MobileNetv2 3 are downloaded from the PyTorch Model Zoo

Original: [Table]
Revised: [Table]

Original: and they are trained for 90 epochs with a step learning rate scheduler.
Revised: and they are trained for 90 epochs with a step learning rate scheduler.

Original: After loading the pre-trained weights, we employ a warmup learning scheduler in the ﬁrst three epochs and the cosine scheduler in the remained 57 epochs with an initial learning rate of 2e-2.
Revised: After loading the pre-trained weights, we employ a warmup learning scheduler in the ﬁrst three epochs and the cosine scheduler in the remained 57 epochs with an initial learning rate of 2e-2.

Original: As shown in Figure 3a, when quantizing both weights and activations, our degradation of accuracy is signiﬁcantly smaller than LQ-Nets, PACT, and RQ.
Revised: As shown in Figure 3a, when quantizing both weights and activations, our degradation of accuracy is signiﬁcantly smaller than LQ-Nets, PACT, and RQ.

Original: Especially, LLSQ outperforms the baselines when quantizing weights and activations into 4-bit.
Revised: Especially, LLSQ outperforms the baselines when quantizing weights and activations into 4-bit.

Original: And it also outperforms other non-linear quantization methods with different bitwidth.
Revised: And it also outperforms other non-linear quantization methods with different bitwidth.

Original: Figure 3b shows that even with the ﬁrst and last layers quantized, it can still achieve near baseline performance.
Revised: Figure 3b shows that even with the ﬁrst and last layers quantized, it can still achieve near baseline performance.

Original: In overall, the accuracy drop is less than 0.4% in ResNet18, ResNet34, and AlexNet when quantizing the whole network.
Revised: In overall, the accuracy drop is less than 0.4% in ResNet18, ResNet34, and AlexNet when quantizing the whole network.

Original: We also quantize
Revised: We also quantize

Original: MobileNetv2, a more compact network, and obtain results that are signiﬁcantly better than RQ.
Revised: MobileNetv2, a more compact network, and obtain results that are signiﬁcantly better than RQ.

Original: Please check Table 7 for detailed results.
Revised: Please check Table 7 for detailed results.

Original: O BJECT D ETECTION ON P ASCAL VOC
Revised: O BJECT D ETECTION ON P ASCAL VOC

Original: We also apply the proposed LLSQ to YOLOv2.
Revised: We also apply the proposed LLSQ to YOLOv2.

Original: The backbone of YOLOv2 is Darknet-19, and its activation function is Leaky ReLU , so that the activations contain negative values.
Revised: The backbone of YOLOv2 is Darknet-19, and its activation function is Leaky ReLU , so that the activations contain negative values.

Original: For YOLOv on Pascal VOC, we adopt the same quantization conﬁguration (See Section 3.3) of the weights to the activations.
Revised: For YOLOv on Pascal VOC, we adopt the same quantization conﬁguration (See Section 3.3) of the weights to the activations.

Original: Results are listed in Table 4.
Revised: Results are listed in Table 4.

Original: As shown in the table, LLSQ induces minor losses of mAP in different bitwidth presentation.
Revised: As shown in the table, LLSQ induces minor losses of mAP in different bitwidth presentation.

Original: For comparison, we also quantize the activations into signed 5-bit integers using PACT, and consequently face considerable mAP losses (54.8 mAP ).
Revised: For comparison, we also quantize the activations into signed 5-bit integers using PACT, and consequently face considerable mAP losses (54.8 mAP ).

Original: Please note that we use the open-source PyTorch implementation of YOLOv2 4 as the baseline.
Revised: Please note that we use the open-source PyTorch implementation of YOLOv2 4 as the baseline.

Original: We train the quantized model for 170 epochs (2/3 of baseline) with an initial learning rate of 1e-4 (1/10 of baseline).
Revised: We train the quantized model for 170 epochs (2/3 of baseline) with an initial learning rate of 1e-4 (1/10 of baseline).

Original: [Table]
Revised: [Table]

Original: D eg r ada t i on 
Revised: D eg r ada t i on ( % ) -0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: LLSQ LLSQF 
Revised: LLSQ LLSQF 2.

Revised: PACT LQ-Nets

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. BN F USION AND Q UANTIZATION OF B IAS AND THE S CALING F ACTOR
Revised: 4. BN F USION AND Q UANTIZATION OF B IAS AND THE S CALING F ACTOR

Original: We adopt BN fusion in the PostAct ( Conv (cid:213) BN (cid:213) ReLU ) networks according to the formula in Section 3.4.
Revised: We adopt BN fusion in the PostAct ( Conv (cid:213) BN (cid:213) ReLU ) networks according to the formula in Section 3.4.

Original: And the scaling factor of bias is the product of the corresponding scaling factors belonging to the activations and the weights, respectively.
Revised: And the scaling factor of bias is the product of the corresponding scaling factors belonging to the activations and the weights, respectively.

Original: After that, we visualize the bias value distribution of VGG-Small.
Revised: After that, we visualize the bias value distribution of VGG-Small.

Original: Figure 4 shows b/α b is distributed between a vast range (-1000, 1000), resulting in overﬂows of low bitwidth accumulators.
Revised: Figure 4 shows b/α b is distributed between a vast range (-1000, 1000), resulting in overﬂows of low bitwidth accumulators.

Original: And the overﬂow phenomena have a signiﬁcantly harmful impact on the network performance.
Revised: And the overﬂow phenomena have a signiﬁcantly harmful impact on the network performance.

Original: To deal with this issue, we quantize the bias and the scaling factors to 8-bit words, and then ﬁne-tune the networks to restore the original performance.
Revised: To deal with this issue, we quantize the bias and the scaling factors to 8-bit words, and then ﬁne-tune the networks to restore the original performance.

Original: Generally, we need ﬁne-tuning for one epoch only.
Revised: Generally, we need ﬁne-tuning for one epoch only.

Original: After the quantization of bias and scaling factor, we have a fully quantized model and have it deployed onto our integer-only accelerator with 16 bitwidth accumulators.
Revised: After the quantization of bias and scaling factor, we have a fully quantized model and have it deployed onto our integer-only accelerator with 16 bitwidth accumulators.

Original: Table 3 and 7 show that the accuracy loss is negligible with w 4 a 4 b 8 α 8 quantization on both VGG-Small and AlexNet.
Revised: Table 3 and 7 show that the accuracy loss is negligible with w 4 a 4 b 8 α 8 quantization on both VGG-Small and AlexNet.

Original: [Table]
Revised: [Table]

Original: /gD6+sHRb6ToQ==</

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 4. D EPLOYMENT ONTO R EALISTIC H ARDWARE
Revised: 4. D EPLOYMENT ONTO R EALISTIC H ARDWARE

Original: The introduced linear symmetric quantization is intended to deploy the quantized networks to specialized integer-only-arithmetic CNN accelerators or other integer-only hardware.
Revised: The introduced linear symmetric quantization is intended to deploy the quantized networks to specialized integer-only-arithmetic CNN accelerators or other integer-only hardware.

Original: Our accelerators adopt the typical 2D systolic array architecture (Chen et al., 2018), but they are featured with 4-bit or 2-bit low-precision operation.
Revised: Our accelerators adopt the typical 2D systolic array architecture (Chen et al., 2018), but they are featured with 4-bit or 2-bit low-precision operation.

Original: As shown in Figure 1, the 8/4/2-bit accelerator has a 32x7 array of  processing elements (PE).
Revised: As shown in Figure 1, the 8/4/2-bit accelerator has a 32x7 array of [Table] processing elements (PE).

Original: And the MAC unit in each PE consists of a 4-bit multiplier and a 16-bit accumulator.
Revised: And the MAC unit in each PE consists of a 4-bit multiplier and a 16-bit accumulator.

Original: For the 4-bit accelerator, we use INT4 representation for weights, UINT4 for activations, INT8 for the bias and scaling factors, respectively.
Revised: For the 4-bit accelerator, we use INT4 representation for weights, UINT4 for activations, INT8 for the bias and scaling factors, respectively.

Original: For the 2-bit accelerator, we use INT for weights, UINT2 for activations, INT8 for the bias and scaling factors, respectively.
Revised: For the 2-bit accelerator, we use INT for weights, UINT2 for activations, INT8 for the bias and scaling factors, respectively.

Original: Through the quantization process described in the paper, we can have a fully quantized network that works directly on the CNN accelerator.
Revised: Through the quantization process described in the paper, we can have a fully quantized network that works directly on the CNN accelerator.

Original: In addition, as we use linear symmetric quantization, we can use a straight-forward way to conduct multiply-accumulate operations without introducing shifters or lookup tables, which means the quantized models can run on state-of-the-art integer accelerators and ensures that their output accuracy degradation is minimal as presented in the above sections.
Revised: In addition, as we use linear symmetric quantization, we can use a straight-forward way to conduct multiply-accumulate operations without introducing shifters or lookup tables, which means the quantized models can run on state-of-the-art integer accelerators and ensures that their output accuracy degradation is minimal as presented in the above sections.

Original: Finally, we implement the 8/4/2-bit integer neural network processors with Synopsys Design Compiler (DC) under the 40nm technology, clocked at 800MHz.
Revised: Finally, we implement the 8/4/2-bit integer neural network processors with Synopsys Design Compiler (DC) under the 40nm technology, clocked at 800MHz.

Original: Table 5 shows that the 4/2-bit implementation achieves up to 2.56x lower silicon area and 5.56x lower power compared to that of the 8-bit baseline.
Revised: Table 5 shows that the 4/2-bit implementation achieves up to 2.56x lower silicon area and 5.56x lower power compared to that of the 8-bit baseline.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we introduced a learned linear symmetric quantization (LLSQ) to quantize the whole network including the bias and scaling factors.
Revised: In this paper, we introduced a learned linear symmetric quantization (LLSQ) to quantize the whole network including the bias and scaling factors.

Original: We also use BN fusion and low bitwidth accumulators to reduce the network inference overhead and the hardware resources in integer neural accelerators.
Revised: We also use BN fusion and low bitwidth accumulators to reduce the network inference overhead and the hardware resources in integer neural accelerators.

Original: We show that our proposed method performs well for various networks on Cifar10, ImageNet and
Revised: We show that our proposed method performs well for various networks on Cifar10, ImageNet, and

Original: Pascal VOC datasets.
Revised: Pascal VOC datasets.

Original: We also show that even the linear symmetric quantizer can obtain better results than asymmetric or non-linear quantization in the case of 4-bit networks.
Revised: We also show that even the linear symmetric quantizer can obtain better results than asymmetric or non-linear quantization in the case of 4-bit networks.

Original: Finally, we deploy the quantized network onto our specialized integer-only neural network accelerator.
Revised: Finally, we deploy the quantized network onto our specialized integer-only neural network accelerator.

Original: Currently, the bitwidth of every layer in a network is all the same.
Revised: Currently, the bitwidth of every layer in a network is all the same.

Original: Prior researches empirically ﬁnd that different layers have different sensitivity to bitwidth of quantization.
Revised: Prior researches empirically ﬁnd that different layers have different sensitivity to bitwidth of quantization.

Original: Hence in the future, we will explore a framework to support more ﬂexible bitwidth for different layers or ﬁner-grained quantization.
Revised: Hence in the future, we will explore a framework to support more ﬂexible bitwidth for different layers or ﬁner-grained quantization.

Revised: A CKNOWLEDGMENTS

Revised: This work was supported in part by the National Natural Science Foundation of China under Grant 61874124 and Grant 61902375.

Original: A PPENDIX I MAGE N ET DETAILED
Revised: A PPENDIX I MAGE N ET DETAILED

Original: We conduct experiments on AlexNet, ResNet18, ResNer and MobileNetv2.
Revised: We conduct experiments on AlexNet, ResNet18, ResNer and MobileNetv2.

Original: For all of the experiments, we adopt channel-wise quantization for Conv layers and layer-wise quantization for FC layers as well as the activations.
Revised: For all of the experiments, we adopt channel-wise quantization for Conv layers and layer-wise quantization for FC layers as well as the activations.

Original: The AlexNet architecture is the same as the PyTorch Model Zoo,  and it consists of ﬁve Conv layers, three FC layers, three MaxPool layers and two Dropout layers.
Revised: The AlexNet architecture is the same as the PyTorch Model Zoo, [Table] and it consists of ﬁve Conv layers, three FC layers, three MaxPool layers, and two Dropout layers.

Original: To prevent over-ﬁtting, we keep the Dropout layers when quantizing AlexNet.
Revised: To prevent over-ﬁtting, we keep the Dropout layers when quantizing AlexNet.

Original: As shown in Figure 5d, we use the same learning rate scheduler for all experiments on ImageNet.
Revised: As shown in Figure 5d, we use the same learning rate scheduler for all experiments on ImageNet.

Original: The test curves are also shown in Figure 5.
Revised: The test curves are also shown in Figure 5.

Original: As we begin with the pre-trained full-precision weights, the test accuracy is already acceptable after one-epoch training.
Revised: As we begin with the pre-trained full-precision weights, the test accuracy is already acceptable after one-epoch training.

Original: The ﬁnal results are listed in Table 7.
Revised: The ﬁnal results are listed in Table 7.

Original: Training time.
Revised: Training time.

Original: The proposed LLSQ requires about 2 / 3 training epochs than that of ﬂoating-point network training.
Revised: The proposed LLSQ requires about 2 / 3 training epochs than that of ﬂoating-point network training.

Original: In each training iteration, LLSQ needs extra computation cost to optimize the [Table] [Figure] [Table]
Revised: In each training iteration, LLSQ needs extra computation cost to optimize the quantizers.

Original: Train the

Original: [Table]

Original: from

Original: [Table]

Original: [Figure]

Original: 0x

Original: Quantize w/a to 4/4 according

Original: [Table]

Original: Step1 of Alg.

Original: [Figure]

Original: Quantize w/a to 3/3 according to Step1 of Alg.

Original: [Figure]

Original: 69x

Original: Quantize b/α to 8/8 according to Step2 of Alg.

Original: [Figure]

Original: 01x quantizers.

Original: Speciﬁcally, the simulated gradients generation of the scaling factors is the major cost.
Revised: Speciﬁcally, the simulated gradients generation of the scaling factors is the major cost.

Original: Table 6 shows the total training time comparison of ResNet18 network.
Revised: Table 6 shows the total training time comparison of ResNet18 network.

Original: The quantization training time is 70% of baseline only.
Revised: The quantization training time is 70% of baseline only.

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: and ResNet34 on ImageNet.

Original: T HE LLSQ A LGORITHM
Revised: T HE LLSQ A LGORITHM

Original: Generate simulate gradients for α :
Revised: Generate simulate gradients for α :

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where x r is one kernel of weights or one layer of activations.
Revised: where x r is one kernel of weights or one layer of activations.

Original: arg min([ E l , E m , E r ]) ∈ { 0 , 1 , 2 } selects the index of the smallest number in the array [ E l , E m , E r ] .
Revised: arg min([ E l , E m , E r ]) ∈ { 0 , 1 , 2 } selects the index of the smallest number in the array [ E l , E m , E r ] .

Original: The bit-shift quantization can be formulated as:
Revised: The bit-shift quantization can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where α ∈ R + len ( α ) is the scaling factors to be quantized, k ∈ Z is the bitwidth, and qcode ∈
Revised: where α ∈ R + len ( α ) is the scaling factors to be quantized, k ∈ Z is the bitwidth, and qcode ∈

Original: Z is the parameter of the bit-shift quantizer simply obtained by:
Revised: Z is the parameter of the bit-shift quantizer simply obtained by:

Original: [Equation]
Revised: [Equation]

Original: Algorithm 1 LLSQ Input: Dataset ( x , y ), where x is input and y is label; Pre-trained full-precision parameters ( w , b ), where w is weights and b is bias; Suppose the network consists of L layers, w ( i ) l represents the i th kernel of weights of the l th layer while a l is the output of the l th layer.
Revised: Input: Dataset ( x , y ), where x is input and y is label; Pre-trained full-precision parameters ( w , b ), where w is weights and b is bias; Suppose the network consists of L layers, w ( i ) l represents the i th kernel of weights of the l th layer while a l is the output of the l th layer.

Original: Output: The quantized scaling factors:
Revised: Output: The quantized scaling factors:

Original: [Equation]
Revised: [Equation]

Original: The quantized weights w q and bias ˆ bq .
Revised: The quantized weights w [Equation] and bias ˆ b

Original:  Step 1: Quantize weights and activations and Re-training // Re-training of quantized networks can converge faster and end with a higher accuracy due to the mechanism of BN layers.
Revised: [Equation] Step 1: Quantize weights and activations and Re-training // Re-training of quantized networks can converge faster and end with a higher accuracy due to the mechanism of BN layers.

Original: repeat Forward: a 0 ← input for l = 1 , · · · , L do for w ( i ) l in w l do //
Revised: repeatForward: a 0 ← input for l = 1 , · · · , L do for w ( i ) l in w l do //

Original: This is accelerated in parallel when implemented.
Revised: This is accelerated in parallel when implemented.

Revised: w

Original: [Equation]
Revised: [Equation]

Original: Generate simulated gradients for α w ( i) l per Eq. (9) end for w
Revised: i

Original: [Equation]
Revised: [Equation]

Original: a ql ← Quantize( [Equation] per Eq [Equation]
Revised: l ← Quantize( w ( i ) l | α w ( i ) l ) per Eq.

Original: [Equation]
Revised: Generate simulated gradients for α w ( i ) l per Eq. (9) end for w ql ← Concat i w ( i ) q l a l ← ReLU ( BN ( Conv ( a ql − 1 , w ql , b l ))) a ql ← Quantize( a l | α a l ) per Eq.

Original: Generate simulated gradients for α a l per Eq.
Revised: Generate simulated gradients for α a l per Eq.

Original: (9) end for
Revised: (9) end for

Original: Backward:
Revised: Backward:

Original: Generate ∆ G for weights and bias and ∆ E for activation per Eq.
Revised: Generate ∆ G for weights and bias and ∆ E for activation per Eq.

Original: (7), (8) and backpropagation algorithm.
Revised: (7), (8) and backpropagation algorithm.

Original: Update w , b , α w , α a iter ← iter +until iter ≥ iter max // need about 60 epochs, e.g. iter max = 60 len ( dataset ) batchsize
Revised: Update w , b , α w , α a iter ← iter +until iter ≥ iter max // need about 60 epochs, e.g. iter max = 60 len ( dataset ) batchsize

Original: Step2: Quantize bias and scaling factor after BN fusion and Re-training iter ←ˆ α w , ˆ b ← BN fusion per Eq.
Revised: Step Quantize bias and scaling factor after BN fusion and Re-training iter ←ˆ α w , ˆ b ← BN fusion per Eq.

Original: (5) repeatForward: a 0 ← input for l = 1 , · · · , L do ˆ α q w l ← SQ( ˆ α w l ) per Eq.
Revised: (5) repeat Forward: a 0 ← input for l = 1 , · · · , L do ˆ α q w l ← SQ( ˆ α w l ) per Eq.

Original: (10) for w ( i ) l in w l do //
Revised: (10) for w ( i ) l in w l do //

Original: This is accelerated in parallel when implemented.
Revised: This is accelerated in parallel when implemented.

Original: w ( i ) q l ← Quantize( w ( i ) l
Revised: w ( i ) q l ← Quantize( w ( i ) l

Original: | ˆ α q w ( i ) l ) per Eq.
Revised: | ˆ α q w ( i ) l ) per Eq.

Original: (1) ˆ b ( i ) q l ← Quantize( ˆ b ( i ) l | α q a l − 1 ˆ α q w ( i ) l ) per Eq.
Revised: ˆ b ( i ) q l ← Quantize( ˆ b ( i ) l | α q a l − 1 ˆ α q w ( i ) l ) per Eq.

Original: Generate simulated gradients for ˆ α w ( i ) l per Eq.
Revised: Generate simulated gradients for ˆ α w ( i ) l per Eq.

Original: (9) end for w ql ← Concat i w ( i ) q l ˆ b
Revised: (9) end for w ql ← Concat i w ( i ) q l ˆ b

Original: ql ← Concat i ˆ b ( i ) q l a l ← ReLU ( Conv ( a ql − 1 , w ql , ˆ b ql )) 
Revised: ql ← Concat i ˆ b ( i ) q l a l ← ReLU ( Conv ( a ql − 1 , w ql , ˆ b ql )) α q a l ← SQ( α a l ) per Eq. (10) a ql ← Quantize( a i | α q a l ) per Eq. Generate simulated gradients for α a l per Eq.

Original: [Table]
Revised: (9) end for Backward: Generate ∆ G for weights and bias and ∆ E for activation per Eq. (7), (8) and backpropagation algorithm.

Revised: Update w , ˆ b , ˆ α w , α a iter ← iter +until iter ≥ iter max // only need one epoch, e.g. iter

Revised: =

Revised: ( dataset ) batchsize

Revised: [Table]

Original: LLSQF(ours)
Revised: LLSQF(ours)

Original: AlexNet 4/ 56.
Revised: AlexNet 4/ 56.

Original: RQ
Revised: RQ

Original: Mobilenet(v1)
Revised: Mobilenet(v1)

Original: Reference 70.
Revised: Reference 70.

Original: (Louizos et al., 2019) 8/ 70.
Revised: (Louizos et al., 2019) 8/ 70.

Original: RQ+ST
Revised: RQ+ST

Original: Mobilenet(v1) 8/ 70.
Revised: Mobilenet(v1) 8/ 70.

Original: (Louizos et al., 2019) 6/ 67.
Revised: (Louizos et al., 2019) 6/ 67.

Original: LLSQ(ours) *
Revised: LLSQ(ours) *

Original: MobileNet(v2)
Revised: MobileNet(v2)

Original: Reference 71.
Revised: Reference 71.

Original: [Table]
Revised: [Table]