Original: SQ UANT : O N - THE -F LY D ATA -F REE Q UANTIZATION VIA D IAGONAL H ESSIAN A PPROXIMATION
Revised: SQ UANT : O N - THE -F LY D ATA -F REE Q UANTIZATION VIA D IAGONAL H ESSIAN A PPROXIMATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Quantization of deep neural networks (DNN) has been proven effective for compressing and accelerating DNN models.
Revised: Quantization of deep neural networks (DNN) has been proven effective for compressing and accelerating DNN models.

Original: Data-free quantization (DFQ) is a promising approach without the original datasets under privacy-sensitive and conﬁdential scenarios.
Revised: Data-free quantization (DFQ) is a promising approach without the original datasets under privacy-sensitive and conﬁdential scenarios.

Original: However, current DFQ solutions degrade accuracy, need synthetic data to calibrate networks, and are time-consuming and costly.
Revised: However, current DFQ solutions degrade accuracy, need synthetic data to calibrate networks, and are time-consuming and costly.

Original: This paper proposes an on-the-ﬂy DFQ framework with sub-second quantization time, called SQuant, which can quantize networks on inference-only devices with low computation and memory requirements.
Revised: This paper proposes an on-the-ﬂy DFQ framework with sub-second quantization time, called SQuant, which can quantize networks on inference-only devices with low computation and memory requirements.

Original: With the theoretical analysis of the second-order information of DNN task loss, we decompose and approximate the Hessian-based optimization objective into three diagonal sub-items, which have different areas corresponding to three dimensions of weight tensor: element-wise, kernel-wise, and output channel-wise.
Revised: With the theoretical analysis of the second-order information of DNN task loss, we decompose and approximate the Hessian-based optimization objective into three diagonal sub-items, which have different areas corresponding to three dimensions of weight tensor: element-wise, kernel-wise, and output channel-wise.

Original: Then, we progressively compose sub-items and propose a novel data-free optimization objective in the discrete domain, minimizing
Revised: Then, we progressively compose sub-items and propose a novel data-free optimization objective in the discrete domain, minimizing

Original: Constrained Absolute Sum of Error (or CASE in short), which surprisingly does not need any dataset and is even not aware of network architecture.
Revised: Constrained Absolute Sum of Error (or CASE in short), which surprisingly does not need any dataset and is even not aware of network architecture.

Original: We also design an efﬁcient algorithm without back-propagation to further reduce the computation complexity of the objective solver.
Revised: We also design an efﬁcient algorithm without back-propagation to further reduce the computation complexity of the objective solver.

Original: Finally, without ﬁne-tuning and synthetic datasets, SQuant accelerates the data-free quantization process to a sub-second level with > 30% accuracy improvement over the existing data-free post-training quantization works, with the evaluated models under 4-bit quantization. 
Revised: Finally, without ﬁne-tuning and synthetic datasets, SQuant accelerates the data-free quantization process to a sub-second level with > 30% accuracy improvement over the existing data-free post-training quantization works, with the evaluated models under 4-bit quantization. We have open-sourced the SQuant framework 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the widespread application of DNN, more and more DNN models are deployed on both computation-constrained and memory-constrained environments, e.g., smartphones, IoT devices, and self-driving cars.
Revised: With the widespread application of DNN, more and more DNN models are deployed on both computation-constrained and memory-constrained environments, e.g., smartphones, IoT devices, and self-driving cars.

Original: The desire for lightweight and energy-efﬁcient DNN deployment solutions is increasing.
Revised: The desire for lightweight and energy-efﬁcient DNN deployment solutions is increasing.

Original: Quantization is one of the most promising techniques to convert weights and activations to lower bit formats and simultaneously reduce computational time and memory consumption.
Revised: Quantization is one of the most promising techniques to convert weights and activations to lower bit formats and simultaneously reduce computational time and memory consumption.

Original: There are two kinds of quantization: Post-training quantization (PTQ) (Banner et al., 2018; Choukroun et al., 2019;
Revised: There are two kinds of quantization: Post-training quantization (PTQ) (Banner et al., 2018; Choukroun et al., 2019;

Original: Zhao et al., 2019; Nagel et al., 2020) and Quantization-aware training (QAT) (Gupta et al., 2015; Jacob et al., 2018; Wang et al., 2019; Zhuang et al., 2021).
Revised: Zhao et al., 2019; Nagel et al., 2020) and Quantization-aware training (QAT) (Gupta et al., 2015; Jacob et al., 2018; Wang et al., 2019; Zhuang et al., 2021).

Original: QAT requires to simulate quantization in the training process, which invokes time-consuming retraining and hyper-parameter tuning.
Revised: QAT requires to simulate quantization in the training process, which invokes time-consuming retraining and hyper-parameter tuning.

Original: In contrast, PTQ directly quantizes well-trained models without retraining.
Revised: In contrast, PTQ directly quantizes well-trained models without retraining.

Original: However, they still need training datasets to calibrate (Nagel et al., 2020) quantized models but are often unavailable due to privacy and security issues, such as medical and conﬁdential scenarios.
Revised: However, they still need training datasets to calibrate (Nagel et al., 2020) quantized models but are often unavailable due to privacy and security issues, such as medical and conﬁdential scenarios.

Original: In contrast, data-free quantization (DFQ) has recently been presented as a promising way to quantize models without original datasets (Nagel et al., 2019;
Revised: In contrast, data-free quantization (DFQ) has recently been presented as a promising way to quantize models without original datasets (Nagel et al., 2019;

Original: Cai et al., 2020; Zhang et al., 2021;
Revised: Cai et al., 2020; Zhang et al., 2021;

Original: Xu et al., 2020; Liu et al., 2021; Qin et al., 2021; Choi et al., 2020).
Revised: Xu et al., 2020; Liu et al., 2021; Qin et al., 2021; Choi et al., 2020).

Original: From a deployment perspective, DFQ is the most attractive quantization method since we can apply it to any trained models as a black box postprocessing step.
Revised: From a deployment perspective, DFQ is the most attractive quantization method since we can apply it to any trained models as a black box postprocessing step.

Original: However, current DFQ methods cannot achieve high accuracy and fast processing time simultaneously.
Revised: However, current DFQ methods cannot achieve high accuracy and fast processing time simultaneously.

Original: Traditionally, DFQ (Nagel et al., 2019) uses rounding quantization, leading to the rounding-to-nearest strategy.
Revised: Traditionally, DFQ (Nagel et al., 2019) uses rounding quantization, leading to the rounding-to-nearest strategy.

Original: Such a strategy causes signiﬁcant accuracy loss, especially in low-bit settings.
Revised: Such a strategy causes signiﬁcant accuracy loss, especially in low-bit settings.

Original: To bridge the accuracy gap between data-free and data-driven quantization, researchers propose a series of data-generative DFQ methods.
Revised: To bridge the accuracy gap between data-free and data-driven quantization, researchers propose a series of data-generative DFQ methods.

Original: They use gradient-based methods to generate fake datasets for trained models.
Revised: They use gradient-based methods to generate fake datasets for trained models.

Original: With the synthetic data, they can employ a data-driven calibration and ﬁne-tuning strategy to improve accuracy.
Revised: With the synthetic data, they can employ a data-driven calibration and ﬁne-tuning strategy to improve accuracy.

Original: However, data generation typically adopts the time-consuming gradient-based methods, which require multiple iterations to generate each input.
Revised: However, data generation typically adopts the time-consuming gradient-based methods, which require multiple iterations to generate each input.

Original: For example, prior works often spend hours generating a calibration dataset and ﬁne-tuning the network (Xu et al., 2020; Liu et al., 2021; Zhang et al., 2021).
Revised: For example, prior works often spend hours generating a calibration dataset and ﬁne-tuning the network (Xu et al., 2020; Liu et al., 2021; Zhang et al., 2021).

Original: To solve this dilemma, we propose SQuant, a fast and accurate data-free quantization framework for convolutional neural networks, employing the constrained absolute sum of error (CASE) of weights as the rounding metric.
Revised: To solve this dilemma, we propose SQuant, a fast and accurate data-free quantization framework for convolutional neural networks, employing the constrained absolute sum of error (CASE) of weights as the rounding metric.

Original: By leveraging Hessian information of network loss due to quantization, we propose a novel diagonal Hessian approximation, which decomposes the optimization objective into three data-free sub-items: element-wise, kernel-wise, and output channel-wise, each of which corresponds to a single or a set of dimensions of the weight tensor.
Revised: By leveraging Hessian information of network loss due to quantization, we propose a novel diagonal Hessian approximation, which decomposes the optimization objective into three data-free sub-items: element-wise, kernel-wise, and output channel-wise, each of which corresponds to a single or a set of dimensions of the weight tensor.

Original: We progressively compose and optimize these three sub-items in the discrete space.
Revised: We progressively compose and optimize these three sub-items in the discrete space.

Original: The ﬁnal approximate objective eliminates the requirement of data generation.
Revised: The ﬁnal approximate objective eliminates the requirement of data generation.

Original: We propose a progressive algorithm with linear complexity to solve the optimization objective, further accelerating DFQ time to a sub-second level.
Revised: We propose a progressive algorithm with linear complexity to solve the optimization objective, further accelerating DFQ time to a sub-second level.

Original: For example,
Revised: For example,

Original: SQuant only needs an average of 4 ms and 84 ms for quantizing a layer and the overall network of
Revised: SQuant only needs an average of 4 ms and 84 ms for quantizing a layer and the overall network of

Original: ResNet18, respectively.
Revised: ResNet18, respectively.

Original: As it does not require back-propagation nor ﬁne-tuning, SQuant can run on inference-only devices with limited computation and memory resources on the ﬂy.
Revised: As it does not require back-propagation nor ﬁne-tuning, SQuant can run on inference-only devices with limited computation and memory resources on the ﬂy.

Original: That opens up new opportunities and scenarios for adopting quantization.
Revised: That opens up new opportunities and scenarios for adopting quantization.

Original: Compared with state-of-the-art DFQ methods, SQuant achieves higher accuracy on all evaluated models under the 4/6/8-bit settings.
Revised: Compared with state-of-the-art DFQ methods, SQuant achieves higher accuracy on all evaluated models under the 4/6/8-bit settings.

Original: SQuant only introduces 0.1% accuracy loss on average under the 8-bit setting.
Revised: SQuant only introduces 0.1% accuracy loss on average under the 8-bit setting.

Original: Under fewer bit precisions, the advantage of SQuant further expands.
Revised: Under fewer bit precisions, the advantage of SQuant further expands.

Original: SQuant only introduces 1.8% accuracy loss on average under the 6-bit setting.
Revised: SQuant only introduces 1.8% accuracy loss on average under the 6-bit setting.

Original: Under the 4-bit setting, SQuant can achieve more than 30% accuracy improvement compared with data-free PTQ methods.
Revised: Under the 4-bit setting, SQuant can achieve more than 30% accuracy improvement compared with data-free PTQ methods.

Original: In a word,
Revised: In a word,

Original: SQuant pushes the accuracy and processing time of DFQ to a new frontier.
Revised: SQuant pushes the accuracy and processing time of DFQ to a new frontier.

Original:  P RELIMINARIES 2. N OTATIONS
Revised:  P RELIMINARIES 2. N OTATIONS

Original: We speciﬁcally use x , y and w to denote the input, output, and weight variables, respectively.
Revised: We speciﬁcally use x , y and w to denote the input, output, and weight variables, respectively.

Original: Constant and scalar are denoted by italic letters, e.g., c , M .
Revised: Constant and scalar are denoted by italic letters, e.g., c , M .

Original: Column vector and ﬂattened matrix are denoted by bold lowercase letters, e.g., w , and matrices (or tensors) are represented by uppercase letters, e.g., W .
Revised: Column vector and ﬂattened matrix are denoted by bold lowercase letters, e.g., w , and matrices (or tensors) are represented by uppercase letters, e.g., W .

Original: The subscript and superscript can further represent the element indices and the layer of a network, respectively, e.g., W (cid:96) i , j . E [ · ] denotes the expectation operator, and the network loss function is represented by L ( · ) .
Revised: The subscript and superscript can further represent the element indices and the layer of a network, respectively, e.g., W (cid:96) i , j . E [ · ] denotes the expectation operator, and the network loss function is represented by L ( · ) .

Original: For convenience in this paper, we call the row of FC (fully connected layer) weight as the output channel and the column of FC weight as the input channel, which are the counterparts to Conv (convolution layer) weight.
Revised: For convenience in this paper, we call the row of FC (fully connected layer) weight as the output channel and the column of FC weight as the input channel, which are the counterparts to Conv (convolution layer) weight.

Original: We use M , N , and K to denote output channel size, input channel size, and kernel height × kernel width, respectively.
Revised: We use M , N , and K to denote output channel size, input channel size, and kernel height × kernel width, respectively.

Original: Speciﬁcally, FC has the shape of ( M , N , 1 ) .
Revised: Speciﬁcally, FC has the shape of ( M , N , 1 ) .

Original: 2. Q UANTIZATION
Revised: 2. Q UANTIZATION

Original: Most previous works adopt the rounding-to-nearest approach for quantizing deep neural networks by rounding elements w to the nearest quantization grid values with a ﬁxed-point data type.
Revised: Most previous works adopt the rounding-to-nearest approach for quantizing deep neural networks by rounding elements w to the nearest quantization grid values with a ﬁxed-point data type.

Original: The quantization and dequantization for a quantized element (cid:98) w can be described as (cid:98) w = s · clip ( (cid:98) ws (cid:101) , min , max ) , where s denotes the quantization scale parameter and, min and max are the lower and upper thresholds for the clipping function clip ( · ) .
Revised: The quantization and dequantization for a quantized element (cid:98) w can be described as (cid:98) w = s · clip ( (cid:98) ws (cid:101) , min , max ) , where s denotes the quantization scale parameter and, min and max are the lower and upper thresholds for the clipping function clip ( · ) .

Original: The operator (cid:98)·(cid:101) represents the rounding-to-nearest, i.e., minimizing the mean squared error (MSE) between the quantized and the original value.
Revised: The operator (cid:98)·(cid:101) represents the rounding-to-nearest, i.e., minimizing the mean squared error (MSE) between the quantized and the original value.

Original: 2. H ESSIAN -B ASED O PTIMIZATION FOR N EURAL N ETWORKS
Revised: 2. H ESSIAN -B ASED O PTIMIZATION FOR N EURAL N ETWORKS

Original: The Hessian-based approach is one of the most promising optimizations to further improve the quantization (Dong et al., 2019b;a; Nagel et al., 2020;
Revised: The Hessian-based approach is one of the most promising optimizations to further improve the quantization (Dong et al., 2019b;a; Nagel et al., 2020;

Original: Shen et al., 2020; Qian et al., 2020; Wu et al., 2020; Hubara et al., 2020; Li et al., 2021; Yao et al., 2021) and pruning (Yu et al., 2021) performance for DNN models.
Revised: Shen et al., 2020; Qian et al., 2020; Wu et al., 2020; Hubara et al., 2020; Li et al., 2021; Yao et al., 2021) and pruning (Yu et al., 2021) performance for DNN models.

Original: Some of those works exploit the Hessian matrix to approximate loss degradation due to the quantization perturbation of weight, ∆ W , by
Revised: Some of those works exploit the Hessian matrix to approximate loss degradation due to the quantization perturbation of weight, ∆ W , by

Original: [Equation]
Revised: [Equation]

Original: where the equation comes from second-order Taylor series expansion, g W is the gradient and H W is the full network Hessian matrix w.r.t.
Revised: where the equation comes from second-order Taylor series expansion, g W is the gradient and H W is the full network Hessian matrix w.r.t.

Original: original weight, W .
Revised: original weight, W .

Original: Since a well-trained model has already converged, the gradient term will be close to 0 and thus can be safely ignored.
Revised: Since a well-trained model has already converged, the gradient term will be close to 0 and thus can be safely ignored.

Original: However, computing
Revised: However, computing

Original: H W is infeasible because of the large memory overhead and computation complexity.
Revised: H W is infeasible because of the large memory overhead and computation complexity.

Original: To tackle this problem, we approximate H W as a layer-wise Hessian matrix H W (cid:96) under the assumption of cross-layer independence (Dong et al., 2017; Nagel et al., 2020), i.e., H W (cid:96) =
Revised: To tackle this problem, we approximate H W as a layer-wise Hessian matrix H W (cid:96) under the assumption of cross-layer independence (Dong et al., 2017; Nagel et al., 2020), i.e., H W (cid:96) =

Original: x (cid:96) x (cid:96) T ⊗ ∇ 2 y (cid:96) L , where ⊗ denotes Kronecker product of two matrices, ∇ 2 y (cid:96) L is the Hessian of the task loss w.r.t.
Revised: x (cid:96) x (cid:96) T ⊗ ∇ 2 y (cid:96) L , where ⊗ denotes Kronecker product of two matrices, ∇ 2 y (cid:96) L is the Hessian of the task loss w.r.t.

Original: y (cid:96) .
Revised: y (cid:96) .

Original: For the m -th output channel of Conv or FC, H W (cid:96) can be approximatively simpliﬁed into output channel-wise (Nagel et al., 2020;
Revised: For the m -th output channel of Conv or FC, H W (cid:96) can be approximatively simpliﬁed into output channel-wise (Nagel et al., 2020;

Original: Yu et al., 2021; Wu et al., 2020;
Revised: Yu et al., 2021; Wu et al., 2020;

Original: Qian et al., 2020),
Revised: Qian et al., 2020),

Original: [Equation]
Revised: [Equation]

Original: where ∇ 2 y (cid:96) L is approximately a diagonal matrix.
Revised: where ∇ 2 y (cid:96) L is approximately a diagonal matrix.

Original: Then the ﬁnal optimization objective is
Revised: Then the ﬁnal optimization objective is

Original: [Equation]
Revised: [Equation]

Original: which is the MSE between the output activation produced from original and quantized weights.
Revised: which is the MSE between the output activation produced from original and quantized weights.

Original: Each sub-problem deals with a single output channel ∆ W (cid:96) m , : .
Revised: Each sub-problem deals with a single output channel ∆ W (cid:96) m , : .

Original: We will further approximate Eq.
Revised: We will further approximate Eq.

Original: (4) to remove any input data dependency from the optimization objective in Sec. 3.2.
Revised: (4) to remove any input data dependency from the optimization objective in Sec. 3.2.

Original:  M ETHODOLOGY 3. O VERVIEW
Revised:  M ETHODOLOGY 3. O VERVIEW

Original: Although we can obtain a good quantization strategy by minimizing Eq. (5) for each output channel, it is an NP-hard combinatorial optimization problem.
Revised: Although we can obtain a good quantization strategy by minimizing MSE for each output channel, it is an NP-hard combinatorial optimization problem.

Original: Even approaching an acceptable local minimum requires signiﬁcant effort and involves input activations without the data-free promise.
Revised: Even approaching an acceptable local minimum requires signiﬁcant effort and involves input activations without the data-free promise.

Original: To avoid the combinatorial optimization problem and eliminate the requirement of data, we propose the SQuant framework.
Revised: To avoid the combinatorial optimization problem and eliminate the requirement of data, we propose the SQuant framework.

Original: First, SQuant approximates Eq.
Revised: First, SQuant approximates Eq.

Original: (4) with three diagonal Hessian matrices corresponding to the dimensions of weight, in Sec. 3.2.
Revised: (4) with three diagonal Hessian matrices corresponding to the dimensions of weight, in Sec. 3.2.

Original: Due to the quantization with a ﬁxed-point data type, SQuant transforms the problem into a data-free optimization problem in the discrete domain.
Revised: Due to the quantization with a ﬁxed-point data type, SQuant transforms the problem into a data-free optimization problem in the discrete domain.

Original: SQuant dedicates to optimizing each layer’s weight employing a ﬂipping approach (Nagel et al., 2020) without any input activation.
Revised: SQuant dedicates to optimizing each layer’s weight employing a ﬂipping approach (Nagel et al., 2020) without any input activation.

Original: To achieve our proposed optimization objective, minimizing
Revised: To achieve our proposed optimization objective, minimizing

Original: CASE (Constrained Absolute Sum of Error), SQuant progressively composes three approximate sub-items under constraint relaxation, introduced in Sec. 3.3.
Revised: CASE (Constrained Absolute Sum of Error), SQuant progressively composes three approximate sub-items under constraint relaxation, introduced in Sec. 3.3.

Original: Finally, SQuant needs to work out a ﬂipping set f to minimize the CASE of each kernel and output channel.
Revised: Finally, SQuant needs to work out a ﬂipping set f to minimize the CASE of each kernel and output channel.

Original: We design an efﬁcient algorithm with a linear computation complexity to ﬁnd a proper f based on Eq.
Revised: We design an efﬁcient algorithm with a linear computation complexity to ﬁnd a proper f based on Eq.

Original: (9), in Sec.3.4.
Revised: (8), in Sec.

Original: 3. D IAGONAL H ESSIAN A PPROXIMATION
Revised: D IAGONAL H ESSIAN A PPROXIMATION

Original: In this work, we propose a new approximation of the Hessian matrix to cover non-diagonal elements and decompose Eq.
Revised: In this work, we propose a new approximation of the Hessian matrix to cover non-diagonal elements and decompose Eq.

Original: (4) into three sub-items that correspond to the three dimensions of the weighttensor as illustrated in Fig.
Revised: (4) into three sub-items that correspond to the three dimensions of the weight

Original: • SQuant-E for element-wise optimization covers the diagonal elements of H w (cid:96) m ( H-E ). • SQuant-Kfor kernel-wise optimization covers the diagonal blocks of Hw (cid:96) m
Revised: H-K: SQuant-K H

Original: ( H-K ).
Revised: [Figure]

Original: • SQuant-C for output channel-wise optimization covers the whole H w (cid:96) m
Revised: C: SQuant-C K N ×× K

Original: ( H-C ).
Revised: H-E: SQuant-E

Original: [Figure]
Revised: [Figure]

Original:  𝔼
Revised: ⋅ 𝔼

Original: [Equation]
Revised: [Figure]

Original: [Figure] [Equation]
Revised: ⋅ Δ

Original: [Figure]
Revised: [Figure]

Original:  ⋅ Δ WTm
Revised: : ⋅ Δ W

Original: [Figure]
Revised: [Figure]

Original: : ⋅ Δ W
Revised: tensor as illustrated in Fig. 1: SQuant-E for element-wise optimization covers the diagonal elements of H w (cid:96) m ( H-E ); SQuant-K for kernel-wise optimization covers the diagonal blocks of H w (cid:96) m ( H-K ); SQuant-C for output channel-wise optimization covers the whole H w (cid:96) m

Original: [Figure]
Revised: ( H-C ).

Original: The E [ x (cid:96) x (cid:96) T ] can be approximated by the following equation:
Revised: The E [ x (cid:96) x (cid:96) T ] can be approximated by the following equation:

Original: [Equation]
Revised: [Equation]

Original: where C = c m J NK ,
Revised: where C = c m J NK ,

Original: K = [Equation]
Revised: K =   k 1 J K . . .

Original: 
Revised: k N J K 

Original: [Equation] E =   e [Equation]
Revised:  , and E =   e 1 ,. . .

Original: e N , K [Equation]  .
Revised: e N , K   .

Original: In the above equations, J NK is an all-one matrix with the dimension of N × K (denoted as NK ), and c m is a constant value for m -th output channel.
Revised: In the above equations, J NK is an all-one matrix with the dimension of N × K (denoted as NK ), and c m is a constant value for m -th output channel.

Original: K is a diagonal block matrix, where J K represents an all-one matrix with the dimension of K × K .
Revised: K is a diagonal block matrix, where J K represents an all-one matrix with the dimension of K × K .

Original: The n -th diagonal block corresponds to n -th kernel in convolution and has its own constant value k n .
Revised: The n -th diagonal block corresponds to n -th kernel in convolution and has its own constant value k n .

Original: E is a diagonal matrix with the diagonal elements of e n , i , each of which is a constant value corresponding to i -th element of n -th kernel.
Revised: E is a diagonal matrix with the diagonal elements of e n , i , each of which is a constant value corresponding to i -th element of n -th kernel.

Original: Eq.
Revised: Eq.

Original: (6) provides an approximation that preserves as much information from three different levels of E [ x (cid:96) x (cid:96) T ] as possible, which we explain in Appendix A.1.
Revised: (5) provides an approximation that preserves as much information from three different levels of E [ x (cid:96) x (cid:96) T ] as possible, which we explain in Appendix A.1.

Original: The matrix C catches the common component of the Hessian matrix, while the matrix E reserves the individual components in the diagonal line of the Hessian matrix.
Revised: The matrix C catches the common component of the Hessian matrix, while the matrix E reserves the individual components in the diagonal line of the Hessian matrix.

Original: In addition, we consider kernel-wise approximation for convolution layers by using matrix K .
Revised: In addition, we consider kernel-wise approximation for convolution layers by using matrix K .

Original: For each inference, the weights of a kernel, W (cid:96) m , n , : , scan the same feature map.
Revised: For each inference, the weights of a kernel, W (cid:96) m , n , : , scan the same feature map.

Original: As a result, the corresponding x has nearly the same expectation values in the center area, with a small perturbation in the marginal area due to padding.
Revised: As a result, the corresponding x has nearly the same expectation values in the center area, with a small perturbation in the marginal area due to padding.

Original: Therefore, k n J K as a kernel-wise approximation achieves a low approximate error for convolution.
Revised: Therefore, k n J K as a kernel-wise approximation achieves a low approximate error for convolution.

Original: For any E [ x (cid:96) x (cid:96) T ] , we can always ﬁnd a decomposition that satisﬁes e n , i , k n , c m > 0, for which we present the decomposition method in
Revised: For any E [ x (cid:96) x (cid:96) T ] , we can always ﬁnd a decomposition that satisﬁes e n , i , k n , c m > 0, for which we present the decomposition method in

Original: Appendix A.2.
Revised: Appendix A.2.

Original: Substituting Eq.
Revised: Substituting Eq.

Original: (6) into Eq.
Revised: (5) into Eq.

Original: (4) yields the following equation.
Revised: (4) yields the following equation.

Original: [Equation]
Revised: [Equation]

Original: 3. D ATA - FREE O PTIMIZATION
Revised: 3. D ATA - FREE O PTIMIZATION

Original: Since that all constants and all ∆ WJ ∆ W T items are greater than zero in Eq.
Revised: To achieve the data-free optimization objective, we omit the coefﬁcients ( e n , i , k n and c m ) in Eq.

Original: (7) (see Appendix A.2)
Revised: (6), which leads to the approximate objective in Eq.

Original: [Equation]
Revised: (8) optimized by our fast SQuant framework. We present the omitting process and empirically verify that the approximation does almost not inﬂuence the performance in Appendix A.2 and Appendix A.3.

Original: it can be easily found that there are no training samples needed to minimize argmin
Revised: It can be easily found that there are no training samples needed to minimize, argmin

Original: [Equation]
Revised: [Equation]

Original: Next, we transform the overall objective Eq.
Revised: Next, we transform the overall objective Eq.

Original: (9) in the discrete space and explain how to compose and optimize the three approximated sub-items in order.
Revised: (8) in the discrete space and explain how to compose and optimize the three approximated sub-items in order.

Original: Without loss of generality, we assume all weights have been scaled with the scale parameter s m for W (cid:96) m , : .
Revised: Without loss of generality, we assume all weights have been scaled with the scale parameter s m for W (cid:96) m , : .

Original: Sub-item Analysis For the element-wise item, i.e., the ﬁrst item in Eq.
Revised: Sub-item Analysis For the element-wise item, i.e., the ﬁrst item in Eq.

Original: (9), the problem is reduced to the following objective, which we call SQuant-E .
Revised: (8), the problem is reduced to the following objective, which we call SQuant-E .

Original: [Equation]
Revised: [Equation]

Original: SQuant-E is essentially the rounding method when r e = 0 .
Revised: SQuant-E is essentially the rounding method when r e = 0 .

Original: Rounding does not introduce any approximate error and has O ( 1 ) complexity for each weight element.
Revised: Rounding does not introduce any approximate error and has O ( 1 ) complexity for each weight element.

Original: However, as many previous works pointed out (Nagel et al., 2020), rounding-to-nearest is not optimal because it only considers the diagonal elements the matrix E
Revised: However, as many previous works pointed out (Nagel et al., 2020), rounding-to-nearest is not optimal because it only considers the diagonal elements the matrix E

Original: [ x (cid:96) x
Revised: [ x (cid:96) x

Original: (cid:96) T ] while ignores the rest majority elements.
Revised: (cid:96) T ] while ignores the rest majority elements.

Original: For kernel-wise item (the second item in Eq.
Revised: For kernel-wise item (the second item in Eq.

Original: (9)), we have the following objective called SQuant-K ,
Revised: (8)), we have the following objective called SQuant-K ,

Original: [Equation]
Revised: [Equation]

Original: where | ∑ i ∆ W (cid:96) m , n , i | is the Absolute Sum of Error ( ASE ) of each kernel-wise weight matrix in the convolution and r k equals 0.5 because of the discrete quantization.
Revised: where | ∑ i ∆ W (cid:96) m , n , i | is the Absolute Sum of Error ( ASE ) of each kernel-wise weight matrix in the convolution and r k equals 0.5 because of the discrete quantization.

Original: In other words, SQuant is based on the insight of S um of ( S igned) error instead of the accumulation of absolute (unsigned) error.
Revised: In other words, SQuant is based on the insight of S um of ( S igned) error instead of the accumulation of absolute (unsigned) error.

Original: Similarly, for the output channel-wise item (the third item in Eq.
Revised: Similarly, for the output channel-wise item (the third item in Eq.

Original: , we have SQuant-C ,
Revised: (8)), we have SQuant-C ,

Original: [Equation]
Revised: [Equation]

Original: Relaxation Obviously, r e = 0 .
Revised: Relaxation Obviously, r e = 0 .

Original: 5 is against r k = 0 .
Revised: 5 is against r k = 0 .

Original: 5 because rounding ( r e = 0 .
Revised: 5 because rounding ( r e = 0 .

Original: 5) only guarantees the upper-bound ˙
Revised: 5) only guarantees the upper-bound ˙

Original: r k = 0 .
Revised: r k = 0 .

Original: 5 K for SQuant-K. Some elements need to relax the constraint r e to a larger number, such as 1 . 0, to satisfy r k = 0 .
Revised: 5 K for SQuant-K. Some elements need to relax the constraint r e to a larger number, such as 1 . 0, to satisfy r k = 0 .

Original: Similarly, SQuant-C also needs to relax r k = 1 . 0.
Revised: Similarly, SQuant-C also needs to relax r k = 1 . 0.

Original: CASE Flipping We adopt the ﬂipping approach (Nagel et al., 2020) to minimize the ASE.
Revised: CASE Flipping We adopt the ﬂipping approach (Nagel et al., 2020) to minimize the ASE.

Original: Due to the discrete quantization, rounded elements can be ﬂipped (from rounding up to rounding down and vice versa) with ± 1 integer mutation.
Revised: Due to the discrete quantization, rounded elements can be ﬂipped (from rounding up to rounding down and vice versa) with ± 1 integer mutation.

Original: Formally, we need to work out a ﬂipping set f m to satisfy the overall objective Eq.
Revised: Formally, we need to work out a ﬂipping set f m to satisfy the overall objective Eq.

Original: (9) by composing these three sub-items in order (SQuant-E → SQuant-K →
Revised: (8) by composing these three sub-items in order (SQuant-E → SQuant-K →

Original: SQuant-C) with constraints relaxation.
Revised: SQuant-C) with constraints relaxation.

Original: After optimization, the f m will be
Revised: After optimization, the f m will be

Original: [Equation]
Revised: [Equation]

Original: (13) where f m is the index set of ﬂipped elements for m -th output channel.
Revised: where f m is the index set of ﬂipped elements for m -th output channel.

Original: Speciﬁcally, we need to ﬂip k = (cid:98) ASE (cid:101) elements, whose perturbation has the same sign as ∑ i ∆ W (cid:96) m , n , i .
Revised: Speciﬁcally, we need to ﬂip k = (cid:98) ASE (cid:101) elements, whose perturbation has the same sign as ∑ i ∆ W (cid:96) m , n , i .

Original: We prove the equivalence for
Revised: We prove the equivalence for

Original: Eq.
Revised: Eq.

Original: (11) and Eq.
Revised: (10) and Eq.

Original: (12) by illustrating the transformation process to a discrete problem in Appendix B.1.
Revised: (11) by illustrating the transformation process to a discrete problem in Appendix B.1.

Original: However, any k elements can satisfy Eq.
Revised: However, any k elements can satisfy Eq.

Original: (11) leading to large search space.
Revised: (10) leading to large search space.

Original: Fortunately, based on
Revised: Fortunately, based on

Original: Eq.
Revised: Eq.

Original: (10), SQuant-K can select speciﬁc k elements with the top- k largest perturbation because they will have the smallest perturbation after ﬂipping under the constraint of SQuant-E. Therefore, we adopt the Constrained ASE ( CASE ) to optimize the SQuant-E&K composition via the top- k perturbation algorithm, which is the only solution for minimizing the CASE proven in Appendix B.2.
Revised: (9), SQuant-K can select speciﬁc k elements with the top- k largest perturbation because they will have the smallest perturbation after ﬂipping under the constraint of SQuant-E. Therefore, we adopt the Constrained ASE ( CASE ) to optimize the SQuant-E&K composition via the top- k perturbation algorithm, which is the only solution for minimizing the CASE proven in Appendix B.2.

Original: Obviously,
Revised: Obviously,

Original: SQuant-E&K&C needs to “ﬂip” the “SQuanted” kernel after SQuant-E&K. Notice that we can only ﬂip one element for a kernel to satisfy the constraint r k = 1 . 0.
Revised: SQuant-E&K&C needs to “ﬂip” the “SQuanted” kernel after SQuant-E&K. Notice that we can only ﬂip one element for a kernel to satisfy the constraint r k = 1 . 0.

Original: The following section will introduce an efﬁcient SQuant algorithm with a linear computation complexity for CASE ﬂipping.
Revised: The following section will introduce an efﬁcient SQuant algorithm with a linear computation complexity for CASE ﬂipping.

Original: 3. O N - THE -F LY SQ UANT
Revised: 3. O N - THE -F LY SQ UANT

Original: Progressive Algorithm We design a progressive algorithm illustrated in algorithm 1 to meet our stated optimization objective, i.e., minimizing the CASE of weight perturbation.
Revised: Progressive Algorithm We design a progressive algorithm illustrated in algorithm 1 to meet our stated optimization objective, i.e., minimizing the CASE of weight perturbation.

Original: The critical insight of the progressive algorithm is to gradually calibrate the deviation from the optimal global solution introduced by the ﬁne-grained diagonal sub-item.
Revised: The critical insight of the progressive algorithm is to gradually calibrate the deviation from the optimal global solution introduced by the ﬁne-grained diagonal sub-item.

Original: To calibrate the SQuant-E, SQuant-K ﬂips certain rounded elements.
Revised: To calibrate the SQuant-E, SQuant-K ﬂips certain rounded elements.

Original: After the SQuant-K calibration, SQuant-C then further ﬂips SQuanted kernels.
Revised: After the SQuant-K calibration, SQuant-C then further ﬂips SQuanted kernels.

Original: We start by rounding the weight and updating its perturbation to satisfy r e = 0 .
Revised: We start by rounding the weight and updating its perturbation to satisfy r e = 0 .

Original: 5 (Line 4-5).
Revised: 5 (Line 4-5).

Original: Then we run the SQuant-K (Line 6) to ﬂip speciﬁc elements under r e = 1 . 0, satisfy r k = 0 .
Revised: Then we run the SQuant-K (Line 6) to ﬂip speciﬁc elements under r e = 1 . 0, satisfy r k = 0 .

Original: 5, and update kernel perturbation (Line 7).
Revised: 5, and update kernel perturbation (Line 7).

Original: The follow-up SQuant-C (Line 8) further ﬂips speciﬁc kernels under r k = 1 . 0 and satisfy r c = 0 .
Revised: The follow-up SQuant-C (Line 8) further ﬂips speciﬁc kernels under r k = 1 . 0 and satisfy r c = 0 .

Original: Finally, we derive the quantized weights (Line 9).
Revised: Finally, we derive the quantized weights (Line 9).

Original: Flip Algorithm SQuant-K and SQuant-C can utilize the same ﬂip function.
Revised: Flip Algorithm SQuant-K and SQuant-C can utilize the same ﬂip function.

Original: The goal of the ﬂip algorithm is to ﬁnd a proper element set f to ﬂip and minimize the CASE depicted in algorithm 2.
Revised: The goal of the ﬂip algorithm is to ﬁnd a proper element set f to ﬂip and minimize the CASE depicted in algorithm 2.

Original: First, we need to compute the accumulated perturbation ( e ) (Line 2).
Revised: First, we need to compute the accumulated perturbation ( e ) (Line 2).

Original: We select weights with positive perturbation to decrease the positive e and vice versa for negative e .
Revised: We select weights with positive perturbation to decrease the positive e and vice versa for negative e .

Original: Therefore, we set 0 for the elements with a different sign (Line 3) to disable them.
Revised: Therefore, we set 0 for the elements with a different sign (Line 3) to disable them.

Original: Obviously, we need only k
Revised: Obviously, we need only k

Original: =
Revised: =

Original: (cid:98)| e |(cid:101) elements and reduce | e | < r k = 0 .
Revised: (cid:98)| e |(cid:101) elements and reduce | e | < r k = 0 .

Original: 5 (Line 4).
Revised: 5 (Line 4).

Original: Finally, we ﬂip k weights with the largest | p | (Line 5-6).
Revised: Finally, we ﬂip k weights with the largest | p | (Line 5-6).

Original: For now, we have SQuanted the kernel and tuned kernel CASE to | e | ≤ r k = 0 .
Revised: For now, we have SQuanted the kernel and tuned kernel CASE to | e | ≤ r k = 0 .

Original: Speciﬁcally, for FC and Conv with a kernel size of ( 1 , 1 ) , we can skip the SQuant-K. As mentioned in Section 3.3, SQuant-C ﬂips only one element in each kernel.
Revised: Speciﬁcally, for FC and Conv with a kernel size of ( 1 , 1 ) , we can skip the SQuant-K. As mentioned in Section 3.3, SQuant-C ﬂips only one element in each kernel.

Original: Therefore, we update the kernel perturbation (Line 7 of algorithm 1) for SQuant-C to ﬂip kernel illustrated in Appendix B.3.
Revised: Therefore, we update the kernel perturbation (Line 7 of algorithm 1) for SQuant-C to ﬂip kernel illustrated in Appendix B.3.

Original: As a result, SQuant successfully identiﬁes the optimum combination f under a low computation complexity, which we analyze as follows.
Revised: As a result, SQuant successfully identiﬁes the optimum combination f under a low computation complexity, which we analyze in Appendix B.4.

Original: Complexity Analysis The original optimization problem described by Eq.
Revised: Input: Rounded/SQuanted Weight w ; Weight perturbation p . Output: Updated Quantized Weight w .

Original: (4) is NP-hard with
Revised: 1 def SQuantFlip( w , p ) : e = ∑ i p i // Accumulated perturbation.

Original: O ( M · 2 NK ) .
Revised: p [ e · p < 0 ]

Original: Based on the SQuant approximation, the new optimization objective is to minimize
Revised: = 0 // Disable Elements/kernels

Original: CASE whose complexity is O ( M · 2 N ) for SQuant-C, O ( MN · 2 K ) for SQuant-K and O ( MNK ) for
Revised: (cid:98)| e |(cid:101) / k elements kernels based on the

Original: SQuant-E. SQuant is optimized to a top- k algorithm with a signiﬁcant complexity reduction to
Revised: CASE

Original: O ( n · log ( k )) for each sub-problem.
Revised: f = TopK( | p | , k ) indices

Original: Our experiments show that after SQuant-K pre-optimization, SQuant-C only requires a tiny top- k number, such as k = 32, to satisfy all cases. For a 3 × 3 kernel with 9 elements, SQuant-K only needs k = 4 because the kernel CASE is always ≤ 0 .
Revised: Indices of k largest perturbation

Original: 5 K =4 .
Revised: w [ f ] =

Original: Finally, their complexity can reduce to linear, O ( M · N · 5 ) for SQuant-C and O ( MN · 9 · 2 ) for SQuant-K.
Revised: Flip( w [ f ] ) //

Revised: Flip k elements/kernels with same sign as e return w

Original: On-the-Fly Framework From the overall perspective of the optimization, SQuant-K has MN subproblems, while SQuant-C has M sub-problems.
Revised: On-the-Fly Framework From the overall perspective of the optimization, SQuant-K has MN subproblems, while SQuant-C has M sub-problems.

Original: Because of the independence of sub-problems,
Revised: Because of the independence of sub-problems,

Original: SQuant is friendly for DNN accelerators, e.g., GPU, allowing each sub-problem to be accelerated in parallel.
Revised: SQuant is friendly for DNN accelerators, e.g., GPU, allowing each sub-problem to be accelerated in parallel.

Original: Without the requirement of back-propagation nor ﬁne-tuning, SQuant can run on inferenceonly devices with constrained computation and memory resources on the ﬂy.
Revised: Without the requirement of back-propagation nor ﬁne-tuning, SQuant can run on inferenceonly devices with constrained computation and memory resources on the ﬂy.

Original: That provides new opportunities for optimizing weight quantization.
Revised: That provides new opportunities for optimizing weight quantization.

Original: In the next section, we demonstrate the impressive efﬁciency and accuracy of SQuant.
Revised: In the next section, we demonstrate the impressive efﬁciency and accuracy of SQuant.

Original:  E XPERIMENTS4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS

Original: For demonstrating the strength of SQuant, we evaluate the SQuant as well as four SOTA methods,
Revised: For demonstrating the strength of SQuant, we evaluate the SQuant as well as four SOTA methods,

Original: DFQ (Nagel et al., 2019), ZeroQ (Cai et al., 2020), DSG (Zhang et al., 2021; Qin et al., 2021), and 
Revised: DFQ (Nagel et al., 2019), ZeroQ (Cai et al., 2020), DSG (Zhang et al., 2021; Qin et al., 2021), and [Table]

Original:  GDFQ (Xu et al., 2020), with 5 different CNN models including ResNet-18 & 50 (He et al., 2016),
Revised: [Table] GDFQ (Xu et al., 2020), with 5 different CNN models including ResNet-18 & 50 (He et al., 2016),

Original: Inception V3 (Szegedy et al., 2016), SqueezeNext (Gholami et al., 2018) and ShufﬂeNet (Zhang et al., 2018) on the golden standard dataset ImageNet (Krizhevsky et al., 2012).
Revised: Inception V3 (Szegedy et al., 2016), SqueezeNext (Gholami et al., 2018) and ShufﬂeNet (Zhang et al., 2018) on the golden standard dataset ImageNet (Krizhevsky et al., 2012).

Original: In our experiments, SQuant is dedicated to weight quantization, including setting quantization range and selecting the grid point with per-channel quantization, which is friendly for hardware accelerators.
Revised: In our experiments [Table] SQuant is dedicated to weight quantization, including setting quantization range and selecting the grid point with per-channel quantization, which is friendly for hardware accelerators.

Original: With the BN-based approach, we adopt a simple rounding method and a wide quantization range for activation suggested by DFQ (Nagel et al., 2019) without breaking the data-free premise.
Revised: With the BN-based approach [Table] we adopt a simple rounding method and a wide quantization range for activation suggested by DFQ (Nagel et al., 2019) without breaking the data-free premise.

Original: We clip activation tensors in a layerwise manner (per-tensor).
Revised: We clip activation tensors in a layerwise manner (per-tensor).

Original: We utilize a uniform distribution as the initialization for the activation quantization range.
Revised: We utilize a uniform distribution as the initialization for the activation quantization range.

Original: All DFQ algorithms are implemented with
Revised: All DFQ algorithms are implemented with

Original: PyTorch (Paszke et al., 2019) and evaluated on Nvidia GPU A100-40GB.
Revised: PyTorch (Paszke et al., 2019) and evaluated on Nvidia GPU A100-40GB.

Original: Unless otherwise stated, we employ both weight and activation quantization in all experiments.
Revised: Unless otherwise stated, we employ both weight and activation quantization in all experiments.

Original: Also, uniform quantization grids are used in all experiments, and hyper-parameters, e.g., r e = r k
Revised: Also, uniform quantization grids are used in all experiments, and hyper-parameters, e.g., r e = r k

Original: = 1 . 0 and r c = 0 .
Revised: = 1 . 0 and r c = 0 .

Original: 5, for all SQuant experiments are the same.
Revised: 5, for all SQuant experiments are the same.

Original: 4. C OMPARISON TO SOTA M ETHODS
Revised: 4. C OMPARISON TO SOTA M ETHODS

Original: Table 1 and Table 2 show the results on the ImageNet datasets for various bit-width choices, comparing our SQuant against other data-free methods.
Revised: Table 1 and Table 2 show the results on the ImageNet datasets for various bit-width choices, comparing our SQuant against other data-free methods.

Original: Among these methods, ZeroQ, DSG, and GDFQ are data-generative approaches with back-propagation.
Revised: Among these methods, ZeroQ, DSG, and GDFQ are data-generative approaches with back-propagation.

Original: The former two are PTQ methods, while the last is a QAT method, which retrains the network with the synthetic data.
Revised: The former two are PTQ methods, while the last is a QAT method, which retrains the network with the synthetic data.

Original: DFQ is the only true data-free method with weight equalization and bias correction.
Revised: DFQ is the only true data-free method with weight equalization and bias correction.

Original: Experiments show that SQuant signiﬁcantly outperforms all other SOTA DFQ methods, even with synthetic dataset calibrating their networks.
Revised: Experiments show that SQuant signiﬁcantly outperforms all other SOTA DFQ methods, even with synthetic dataset calibrating their networks.

Original: The 8-bit quantization preserves better network accuracy  than the lower-bit quantization does because of higher precision.
Revised: The 8-bit quantization preserves better network accuracy [Table] than the lower-bit quantization does because of higher precision.

Original: The beneﬁt of SQuant becomes more prominent as the bit-width decreases.
Revised: The beneﬁt of SQuant becomes more prominent as the bit-width decreases.

Original: SQuant outperforms the PTQ methods, i.e., DFQ, ZeroQ, and
Revised: SQuant outperforms the PTQ methods, i.e., DFQ, ZeroQ, and

Original: DSG, more than 30% on all models with 4-bit quantization.
Revised: DSG, more than 30% on all models with 4-bit quantization.

Original: It is noteworthy that SQuant surpasses
Revised: It is noteworthy that SQuant surpasses

Original: GDFQ in all cases and even surpasses more than 15% in ResNet50 under 4-bit quantization, although
Revised: GDFQ in all cases and even surpasses more than 15% in ResNet50 under 4-bit quantization, although

Original: GDFQ is a quantization-aware training method.
Revised: GDFQ is a quantization-aware training method.

Original: Table 1 and Table 2 also show that GDFQ signiﬁcantly outperforms ZeroQ and DSG under lower-bit settings (e.g., 4-bit).
Revised: Table 1 and Table 2 also show that GDFQ signiﬁcantly outperforms ZeroQ and DSG under lower-bit settings (e.g., 4-bit).

Original: Since we use the same activation quantization method for evaluating these methods, the results indicate that the weight quantization plays a critical role in the overall model quantization.
Revised: Since we use the same activation quantization method for evaluating these methods, the results indicate that the weight quantization plays a critical role in the overall model quantization.

Original: However, GDFQ requires ﬁne-tuning (FT) with back-propagation (BP).
Revised: However, GDFQ requires ﬁne-tuning (FT) with back-propagation (BP).

Original: In contrast,
Revised: In contrast,

Original: SQuant adopts a direct optimization objective of weight perturbation, which does not require ﬁnetuning nor BP, and still outperforms GDFQ in the 4-bit setting.
Revised: SQuant adopts a direct optimization objective of weight perturbation, which does not require ﬁnetuning nor BP, and still outperforms GDFQ in the 4-bit setting.

Original: These results clearly illustrate the advantages of SQuant, a CASE-based optimization framework, which is to minimize the CASE of weight perturbation.
Revised: These results clearly illustrate the advantages of SQuant, a CASE-based optimization framework, which is to minimize the CASE of weight perturbation.

Original: 4. SQ UANT E FFICIENCY
Revised: 4. SQ UANT E FFICIENCY

Original: The trade-off between efﬁciency and accuracy is challenging for previous DFQ methods.
Revised: The trade-off between efﬁciency and accuracy is challenging for previous DFQ methods.

Original: Before
Revised: Before

Original: SQuant, DFQ is the fastest one since it does not require back-propagation and ﬁne-tuning, but it performs poorly, especially in low-bit cases.
Revised: SQuant, DFQ is the fastest one since it does not require back-propagation and ﬁne-tuning, but it performs poorly, especially in low-bit cases.

Original: GDFQ performs relatively well but takes hours to complete 400 epochs that produce synthetic data from weights and ﬁne-tune the network.
Revised: GDFQ performs relatively well but takes hours to complete 400 epochs that produce synthetic data from weights and ﬁne-tune the network.

Original: SQuant employs the direct optimization objective, minimizing the CASE of weight perturbation, pushes the quantization procedure to a sub-second level.
Revised: SQuant employs the direct optimization objective, minimizing the CASE of weight perturbation, pushes the quantization procedure to a sub-second level.

Original: Table 3 shows the 4-bit quantization time of the ﬁve models using SQuant, ZeroQ, and GDFQ.
Revised: Table 3 shows the 4-bit quantization time of the ﬁve models using SQuant, ZeroQ [Table] and GDFQ.

Original: The efﬁcient algorithm design also contributes to the surprising results.
Revised: The efﬁcient algorithm design also contributes to the surprising results.

Original: Note that the SQuant results in Table 3 are the sum of all layer quantization time, [Table] [Table] [Table]
Revised: Note that the SQuant results in Table 3 are the sum of all layer quantization time, and it will be faster if we quantize layers in parallel.

Original: and it will be faster if we quantize layers in parallel.

Original: A single layer takes SQuant just 3 milliseconds on average because SQuant does not involve complex algorithms, such as back-propagation and ﬁne-tuning.
Revised: A single layer takes SQuant just 3 milliseconds on average because SQuant does not involve complex algorithms, such as back-propagation and ﬁne-tuning.

Original: That means we can implement the SQuant algorithm on inference-only devices such as smartphones and IoT devices and quantize the network on the ﬂy.
Revised: That means we can implement the SQuant algorithm on inference-only devices such as smartphones and IoT devices and quantize the network on the ﬂy [Table]

Original: 4. A BLATION STUDY
Revised: 4. A BLATION STUDY

Original: SQuant Granularity We decouple the effect of SQuant-K and SQuant-C, which have different granularities to optimize CASE.
Revised: SQuant Granularity We decouple the effect of SQuant-K and SQuant-C, which have different granularities to optimize CASE.

Original: As shown in Table 4, their accuracies both outperform SQuant-E (i.e., rounding), and combining them leads to higher accuracy for ResNet18.
Revised: As shown in Table 4, their accuracies both outperform SQuant-E [Table] [Table] 49.86 DSG + AdaRound (cid:55) (cid:55)  56.09 SQuant (cid:88) (cid:88)  60.

Original:  SQuant-E&C has a lower accuracy than SQuant-E&K because SQuant-C has a more signiﬁcant approximation error than
Revised: ZeroQ + AdaRound (cid:55) (cid:55)  63.86 DSG + AdaRound (cid:55) (cid:55)  66.87 SQuant (cid:88) (cid:88)  69. ZeroQ + AdaRound (cid:55) (cid:55)  68.39 DSG + AdaRound (cid:55) (cid:55)  68.97 SQuant (cid:88) (cid:88)  71. (i.e., rounding), and combining them leads to higher accuracy for ResNet18. SQuant-E&C has a lower accuracy than SQuant-E&K because SQuant-C has a more signiﬁcant approximation error than

Original: SQuant-K.
Revised: SQuant-K.

Original: On the other hand, SQuant-E alone is not optimal because it uses a smaller granularity and ignores a large amount of Hessian information as we analyze in Section 3.
Revised: On the other hand, SQuant-E alone is not optimal because it uses a smaller granularity and ignores a large amount of Hessian information as we analyze in Section 3.

Original: This ablation study shows that SQuant-E&K&C achieves the best accuracy by exploiting the most Hessian information ( H-C ), and SQuant-E&K also achieves a higher accuracy with H-K than SQuant-E with H-E .[Table]
Revised: This ablation study shows that SQuant-E&K&C achieves the best accuracy by exploiting the most Hessian information ( H-C ), and SQuant-E&K also achieves a higher accuracy with H-K than SQuant-E with H-E .

Original: [Table] Comparison to Data-free AdaRound AdaRound (Nagel et al., 2020) is a novel data-driven PTQ method, which also utilizes the Hessian-based approach to round-up or round-down the weights with an approximation assumption.
Revised: Comparison to Data-free AdaRound AdaRound (Nagel et al., 2020) is a novel data-driven PTQ method, which also utilizes the Hessian-based approach to round-up or round-down the weights with an approximation assumption.

Original: Under the data-free premise, we augment ZeroQ and DSG with the AdaRound by feeding their generated synthetic data to AdaRound.
Revised: Under the data-free premise, we augment ZeroQ and DSG with the AdaRound by feeding their generated synthetic data to AdaRound.

Original: Results in Tableshow that SQuant has better accuracy than data-free AdaRound because SQuant directly optimizes the CASE objective instead of the MSE of the output activation adopted by AdaRound.
Revised: Results in Tableshow that SQuant has better accuracy than data-free AdaRound because SQuant directly optimizes the CASE objective instead of the MSE of the output activation adopted by AdaRound.

Original: It is hard for DSG + AdaRound to ﬁnd the optimal solution with an excessively long optimization path and gradient-based approaches.
Revised: It is hard for DSG + AdaRound to ﬁnd the optimal solution with an excessively long optimization path and gradient-based approaches.

Original: Even though AdaRound tries a shorter way to ﬁne-tune the weights in the layer-wise fashion, SQuant still outperforms AdaRound in quantization time and accuracy.
Revised: Even though AdaRound tries a shorter way to ﬁne-tune the weights in the layer-wise fashion, SQuant still outperforms AdaRound in quantization time and accuracy.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Revised: Compression is a promising method to reduce the DNN model’s memory and computation cost.

Revised: Pruning (Han et al., 2015b;a) is one of the effective approaches to exploit the inherent redundancy of DNN.

Revised: However, pruning will cause sparse irregular memory accesses.

Revised: Therefore, pruning needs software (Gale et al., 2020; Guan et al., 2020; Qiu et al., 2019; Guo et al., 2020a; Guan et al., 2021;

Revised: Fedus et al., 2021) and hardware (Gondimalla et al., 2019; Guo et al., 2020b; Zhang et al., 2020;

Original: Data-driven Quantization  Quantization-aware training (QAT) (Gupta et al., 2015; Jacob et al., 2018; Wang et al., 2019; Zhuang et al., 2021) is one of the most promising techniques to retrain networks and mitigate the accuracy drop introduced by quantization.
Revised: Wang et al., 2021) optimization to accelerate. Quantization is more practical because it can be supported directly by existing accelerators. Quantization-aware training (QAT) (Gupta et al., 2015; Jacob et al., 2018; Wang et al., 2019; Zhuang et al., 2021) is one of the most promising techniques to retrain networks and mitigate the accuracy drop introduced by quantization.

Original: However, the training procedure is time-consuming and costly.
Revised: However, the training procedure is time-consuming and costly.

Original: Therefore, post-training quantization (PTQ) (Banner et al., 2018; 
Revised: Therefore, post-training quantization (PTQ) (Banner et al., 2018; Choukroun et al., 2019;

Original: Choukroun et al., 2019; Zhao et al., 2019; Nagel et al., 2020) has earned lots of attention due to the absence of any ﬁne-tuning or retraining process, at the expense of accuracy.Data-free Quantization
Revised: Zhao et al., 2019; Nagel et al., 2020) has earned lots of attention due to the absence of any ﬁne-tuning or retraining process, at the expense of accuracy.

Original: Recently, several methods for CNN quantization without the original training datasets have been proposed.
Revised: Recently, several methods for CNN quantization without the original training datasets have been proposed.

Original: These methods are known as data-free quantization (DFQ), including PTQ (Nagel et al., 2019;
Revised: These methods are known as data-free quantization (DFQ), including PTQ (Nagel et al., 2019;

Original: Cai et al., 2020; Zhang et al., 2021) and QAT (Xu et al., 2020;
Revised: Cai et al., 2020; Zhang et al., 2021) and QAT (Xu et al., 2020;

Original: Liu et al., 2021; Qin et al., 2021; Choi et al., 2020).
Revised: Liu et al., 2021; Qin et al., 2021; Choi et al., 2020).

Original: DFQ (Nagel et al., 2019) and ACIQ (Nagel et al., 2019) rely on weight equalization or bias correction without requiring synthetic data.
Revised: DFQ (Nagel et al., 2019) and ACIQ (Nagel et al., 2019) rely on weight equalization or bias correction without requiring synthetic data.

Original: However, they produce a pretty lower accuracy in lower bit-width (such as 6bit and 4bit).

Original: Other works synthesize the data to calibrate or ﬁne-tune the network based on the batch normalization statistics (Cai et al., 2020) or adversarial knowledge distillation techniques (Liu et al., 2021; Choi et al., 2020).
Revised: Other works synthesize the data to calibrate or ﬁne-tune the network based on the batch normalization statistics (Cai et al., 2020) or adversarial knowledge distillation techniques (Liu et al., 2021; Choi et al., 2020).

Original: They achieve better accuracy using a time-consuming process, e.g., back-propagation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper approximates and composes the original Hessian optimization objective into the CASE of weight perturbation with a data-free premise.
Revised: This paper approximates and composes the original Hessian optimization objective into the CASE of weight perturbation with a data-free premise.

Original: Surprisingly, CASE only involves the weight perturbation and requires no knowledge of any datasets or network architecture.
Revised: Surprisingly, CASE only involves the weight perturbation and requires no knowledge of any datasets or network architecture.

Original: Based on that, we proposed the on-the-ﬂy SQuant framework.
Revised: Based on that, we proposed the on-the-ﬂy SQuant framework.

Original: We used a progressive algorithm to minimize CASE directly and signiﬁcantly improve accuracy than other DFQ methods.
Revised: We used a progressive algorithm to minimize CASE directly and signiﬁcantly improve accuracy than other DFQ methods.

Original: SQuant considerably reduces optimization complexity and accelerates the data-free quantization procedure, which previously requires back-propagation with massive computation and memory resources consumption seen in other works.
Revised: SQuant considerably reduces optimization complexity and accelerates the data-free quantization procedure, which previously requires back-propagation with massive computation and memory resources consumption seen in other works.

Original: In summary, SQuant outperforms other data-free quantization approaches in terms of accuracy and pushes the quantization processing time to a sub-second level. 
Revised: In summary, SQuant outperforms other data-free quantization approaches in terms of accuracy and pushes the quantization processing time to a sub-second level. A CKNOWLEDGMENTS

Revised: We would like to thank the anonymous reviewers for their constructive feedback.

Revised: This work was supported by the National Key R&D Program of China under Grant 2021ZD0110104, and the

Revised: National Natural Science Foundation of China (NSFC) grant (U21B2017, 62072297, and 61832006).

Original: A A PPROXIMATION AND D ECOMPOSITION A. A PPROXIMATED H ESSIAN M ATRIX FOR DATA - FREE QUANTIZATION
Revised: A A PPROXIMATION AND D ECOMPOSITION A. A PPROXIMATED H ESSIAN M ATRIX FOR DATA - FREE QUANTIZATION

Original: The quantization loss function for the entire network is
Revised: The quantization loss function for the entire network is

Original: [Equation]
Revised: [Equation]

Original: Consider a convolution layer deﬁned as
Revised: Consider a convolution layer deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Here, Y has three dimensions, output channel, output feature map height, and output feature map width, i.e., ( M × OH × OW ) indexing by ( m , h , w ) , W has four dimensions, output channel, input channel, kernel height, kernel width, i.e., ( M × N × KH × KW ) indexing by ( m , n , i , j ), and X has three dimensions, input channel, input feature map height, and input feature map width, i.e., ( N × IH × IW ) indexing by ( n , h − i , w − j ) .
Revised: Here, Y has three dimensions, output channel, output feature map height and output feature map width, i e., ( M × OH × OW ) indexing by ( m , h , w ) , W has four dimensions, output channel, input channel, kernel height, kernel width, i e., ( M × N × KH × KW ) indexing by ( m , n , i , j ) and X has three dimensions, input channel, input feature map height and input feature map width, i.e., ( N × IH × IW ) indexing by ( n , h − i , w − j ) .

Original: Ignoring the interaction between layers and outputchannels following Nagel et al.
Revised: Ignoring the interaction between layers and output

Original: (2020), for a speciﬁc convolution layer l and output channel m , the elements of corresponding output channel-wise Hessian H W (cid:96) m is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Assuming ∇ 2 y (cid:96) L is a diagonal matrix yields Eq.
Revised: Assuming ∇ 2 y (cid:96) L is a diagonal matrix yields Eq.

Original: (30) in (Nagel et al., 2020)
Revised: (30) in (Nagel et al., 2020)

Original: [Equation]
Revised: [Equation]

Original: To make Eq.
Revised: To make Eq.

Original: (14) get irrelevant to training samples [Equation] we assume that input feature maps auto-correlate with each other in a similar way
Revised: (13) get irrelevant to training samples, we assume that input feature maps auto-correlate with each other in a similar way, resulting in

Original: [Equation]
Revised: [Equation]

Original: resulting in [Equation] [Equation] for all n , i , j , n (cid:48) , i (cid:48) and j (cid:48) , where c m is a constant.
Revised: for all n , i , j , n (cid:48) , i (cid:48) and j (cid:48) , where c m is a constant.

Original: It should be noted that Eq.
Revised: It should be noted that Eq.

Original: (23) is a strong assumption.
Revised: (22) is a strong assumption.

Original: For more accurate approximation, we further look into each input channel (i.e., n = n (cid:48) , i (cid:54) = i (cid:48) , and j (cid:54) =
Revised: For more accurate approximation, we further look into each input channel (i.e., n = n (cid:48) , i (cid:54) = i (cid:48) , and j (cid:54) =

Original: j (cid:48) ), and ﬁnd
Revised: j (cid:48) ), and ﬁnd

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  for all i , j , i (cid:48) , and j (cid:48) , where k m , n is a constant.
Revised: (23) for all i , j , i (cid:48) , and j (cid:48) , where k m , n is a constant.

Original: It is generally true because the kernel size is usually much smaller than the size of feature maps, so the shift introduced by different i , j , i (cid:48) , and j (cid:48) is a small perturbation of E [ H W (cid:96) m , n i , j , i (cid:48) , j (cid:48) ] compared with the summation over the entire feature map.
Revised: It is generally true because the kernel size is usually much smaller than the size of feature maps, so the shift introduced by different i , j , i (cid:48) , and j (cid:48) is a small perturbation of E [ H W (cid:96) m , n i , j , i (cid:48) , j (cid:48) ] compared with the summation over the entire feature map.

Original: Finally, we focus on each diagonal element of Hessian matrix (i.e., n = n (cid:48) , i = i (cid:48) , and j = j (cid:48) ) and denote
Revised: Finally, we focus on each diagonal element of Hessian matrix (i.e., n = n (cid:48) , i = i (cid:48) , and j = j (cid:48) ) and denote

Original: E [
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: (cid:96) 
Revised: (cid:96) m , n , i

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where e m , n , i , j is a constant.
Revised: where e m , n , i , j is a constant.

Original: Please note that the output channel-wise expected Hessian matrix
Revised: Please note that the output channel-wise expected Hessian matrix

Original: E [ H W (cid:96) m n , i , j , n (cid:48) , i (cid:48) , j (cid:48) ] is a principle submatrix of E [ H ] , so it must positive semi-deﬁne.
Revised: E [ H W (cid:96) m n , i , j , n (cid:48) , i (cid:48) , j (cid:48) ] is a principle submatrix of E [ H ] , so it must positive semi-deﬁne.

Original: Therefore, we set e m , n , i , j > k m , n > c m > 0 to ensure the approximation to E [ H W (cid:96) m n , i , j , n (cid:48) , i (cid:48) , j (cid:48) ] is also positive semi-deﬁne and nontrivial.
Revised: Therefore, we set e m , n , i , j > k m , n > c m > 0 to ensure the approximation to E [ H W (cid:96) m n , i , j , n (cid:48) , i (cid:48) , j (cid:48) ] is also positive semi-deﬁne and nontrivial.

Original: Considering Eq.
Revised: Considering Eq.

Original: (23), Eq. (24), and Eq.
Revised: (22), Eq. (23), and Eq.

Original: (25) at the same time, we can get the approximation to expected Hessian shown in Eq.
Revised: (24) at the same time, we can get the approximation to expected Hessian shown in Eq.

Original: Extending the discussion to fully connected layer is straightforward thus omitted here.
Revised: Extending the discussion to fully connected layer is straightforward thus omitted here.

Original: A. D ECOMPOSITION
Revised: A. D ECOMPOSITION

Original: In this section, we present the decomposition method for H W (cid:96) m =
Revised: In this section, we present the decomposition method for H W (cid:96) m =

Original: l m E [ x (cid:96) x (cid:96) T ] illustrated in the algorithm 3.
Revised: l m E [ x (cid:96) x (cid:96) T ] illustrated in the algorithm 3.

Original: First, we construct three matrices with the shape of ( NK , NK ) , C (cid:48)
Revised: First, we construct three matrices with the shape of ( NK , NK ) , C (cid:48)

Original: =
Revised: =

Original: J NK ,
Revised: J NK ,

Original: K (cid:48)
Revised: K (cid:48)

Original: =  
Revised: =  

Original: J K . . .
Revised: J K ...

Original: J K 
Revised: J K 

Original:  , and E (cid:48)
Revised:  , and E (cid:48)

Original: =   . . .
Revised: =   ...

Original: 
Revised: 

Original:  .
Revised:  .

Original: Here, J NK is an all-one matrix with dimension NK =
Revised: Here, J NK is an all-one matrix with dimension NK =

Original: N × K and J K represents an all-one matrix with dimension K .
Revised: N × K and J K represents an all-one matrix with dimension K .

Original: The n -th diagonal block corresponds to n -th kernel in convolution and has the same constant k n .
Revised: The n -th diagonal block corresponds to n -th kernel in convolution and has the same constant k n .

Original: E is a diagonal matrix whose diagonal elements are 1.
Revised: E is a diagonal matrix whose diagonal elements are 1.

Original: Algorithm3: E [ x (cid:96) x (cid:96) T ] Decomposition.
Revised: Algorithm

Original: Input: E [ x (cid:96) x (cid:96) T ] , H ;
Revised: Input: E [ x (cid:96) x (cid:96) T ] , H ;

Original: Channel-wise matrix, C (cid:48) .
Revised: Channel-wise matrix, C (cid:48) .

Original: Kernel-wise matrix, K (cid:48) .
Revised: Kernel-wise matrix, K (cid:48) .

Original: Element-wise matrix, E (cid:48) .
Revised: Element-wise matrix, E (cid:48) .

Original: Output: Matrix E , K , and C .
Revised: Output: Matrix E , K , and C .

Original: 1 H (cid:48) =
Revised: 1 H (cid:48) =

Original: | H | 2 c m = ( 1 − ε ) · min ( H (cid:48) ) //
Revised: | H | 2 c m = ( 1 − ε ) · min ( H (cid:48) ) //

Original: Output channel-wise.
Revised: Output channel-wise.

Original: 3 C =
Revised: 3 C =

Original: c m C (cid:48) 4 foreach n ∈ [ 1 ,..., N ] do // Kernel-wise.
Revised: c m C (cid:48) 4 foreach n ∈ [ 1 ,..., N ] do // Kernel-wise.

Original: k n = ( 1 − ε (cid:48) n ) · min ( H (cid:48) n : n + K , n : n + K − c m )
Revised: k n = ( 1 − ε (cid:48) n ) · min ( H (cid:48) n : n + K , n : n + K − c m )

Original: K n , : = k n K (cid:48) n , : foreach i ∈ [ 1 ,..., K ] do // Element-wise.
Revised: K n , : = k n K (cid:48) n , : foreach i ∈ [ 1 ,..., K ] do // Element-wise.

Original: e n , i = H (cid:48)
Revised: e n , i = H (cid:48)

Original: n × K + i , n × K + i − c m − k n
Revised: n × K + i , n × K + i − c m − k n

Original: E n , i =
Revised: E n , i =

Original: e n , i E (cid:48)
Revised: e n , i E (cid:48)

Original: n , i 10 return E , K , C
Revised: n , i 10 return E , K , C

Original: In algorithm 3, 0 < ε , ε (cid:48) < 1, and we can get the matrices E , K , and C .
Revised: In algorithm 3, 0 < ε , ε (cid:48) < 1, and we can get the matrices E , K , and C .

Original: Evidently, algorithm 3 can make c m > 0, k n > 0, and e n , i > 0 for any E [ x (cid:96) x
Revised: Evidently, algorithm 3 can make c m > 0, k n > 0, and e n , i > 0 for any E [ x (cid:96) x

Original: (cid:96) T ] ≈ E + K + C .
Revised: (cid:96) T ] ≈ E + K + C .

Original: A. A PPROXIMATION E RROR A NALYSIS
Revised: A. A PPROXIMATION E RROR A NALYSIS

Original: To achieve the data-free optimization objective, we omit the coefﬁcients ( e n , i , k n and c m ) in Eq.
Revised: To achieve the data-free optimization objective, we omit the coefﬁcients ( e n , i , k n and c m ) in Eq.

Original: (7), which leads to the approximate objective in Eq.
Revised: (6), which leads to the approximate objective in Eq.

Original: (9) optimized by our fast SQuant framework.
Revised: (8) optimized by our fast SQuant framework.

Original: In this section, we quantify the approximation error of this approximation using samples from the real dataset.
Revised: We approximate Eq.

Original: [Table]
Revised: (6) to Eq.

Original:  is  the number of the 
Revised: (8) to enable fast data-free quantization. The approximation error is insigniﬁcant as our comprehensive results have shown the high accuracy of the ﬁnal quantized model in Table 1 and Table 2 of the manuscript. The intuition behind the approximation is that we use an iterative process which progressively reduces each term of Eq.

Original: Flipped elements after SQuant optimization. “Correct” is  the number of elements has the same optimization direction as the precise objective.
Revised: Because each term’s coefﬁcient ( e n , i , k n , and c m ) is positive, the reduction of each term would generally lead to the reduction of the precise objective in Eq.

Original: AP is the approximation precision.
Revised: In this section, we provide an empirical analysis of the approximation error between Eq. andEq.

Original: In this empirical experiment, we use the real dataset to generate the precise coefﬁcients of e n , i , k n , and c m in Eq.
Revised: In this empirical experiment, we use the real dataset to generate the precise coefﬁcients of e n , i , k n , and c m in Eq.

Original: To quantify the approximation error in our SQuant framework, we evaluate a metric called approximation precision and show that we achieve a nearly 95% approximation precision.
Revised: To quantify the approximation error in our SQuant framework, we evaluate a metric called approximation precision and show that we achieve a nearly 95% approximation precision.

Original: Since SQuant uses the ﬂipping-based iterative optimization framework to minimize Eq.
Revised: Since SQuant uses the ﬂipping-based iterative optimization framework to minimize Eq.

Original: (9), we deﬁne an element as correctly ﬂipped if its ﬂipping leads to the decrease of the precise objective Eq.
Revised: (8), we deﬁne an element as correctly ﬂipped if its ﬂipping leads to the decrease of the precise objective Eq.

Original: (7) and approximate objective Eq.
Revised: (6) and approximate objective Eq.

Original: The approximation precision (AP) is the ratio of the correct element based on data-free Eq.
Revised: The approximation precision (AP) is the ratio of the correct element based on data-free Eq.

Original: (9) compared to data-driven Eq.
Revised: (8) compared [Table] [Table]

Original: (7), i.e.,
Revised: is the number of the

Original: [Equation]
Revised: Flipped elements after SQuant optimization. “Correct” is the number of elements has the same optimization direction as the precise objective. AP is the approximation precision.

Original: We perform the above approximation error analysis on ResNet18 with ImageNet under 4-bit weightonly quantization.
Revised: We perform the above approximation error analysis on ResNet18 with ImageNet under 4-bit weightonly quantization.

Original: We evaluate the SQuanted weight on the inference datasets.
Revised: We evaluate the SQuanted weight on the inference datasets.

Original: We compute the coefﬁcients e n , i , k n , and c m with 1000 samples.
Revised: We compute the coefﬁcients e n , i , k n , and c m with 1000 samples.

Original: Table 6 shows the results, which clearly show that
Revised: Table 6 shows the results, which clearly show that

Original: SQuant-E&K&C achieves nearly 100% approximation precision.
Revised: SQuant-E&K&C achieves nearly 100% approximation precision.

Original: In other words, nearly all ﬂipped elements can indeed reduce the precise objective in Eq.
Revised: In other words, nearly all ﬂipped elements can indeed reduce the precise objective in Eq.

Original: Based on this empirical study, we show that the approximation from Eq.
Revised: Based on this empirical study, we show that the approximation from Eq.

Original: (7) to Eq.
Revised: (6) to Eq.

Original: (9) is effective for our data-free quantization.
Revised: (8) is effective for our data-free quantization.

Original: B S QUANT A LGORITHM B. D ISCRETE O PTIMIZATION P ROBLEM
Revised: B S QUANT A LGORITHM B. D ISCRETE O PTIMIZATION P ROBLEM

Original: We introduce the transformation of the discrete optimization problem.
Revised: We introduce the transformation of the discrete optimization problem.

Original: We know that the quantization is to round the scaled elements to the integer grid.
Revised: We know that the quantization is to round the scaled elements to the integer grid.

Original: Each quantization step has two rounding directions, rounding up and down, with a step size of 1.
Revised: Each quantization step has two rounding directions, rounding up and down, with a step size of 1.

Original: We have the quantization example within [ 0 , 1 ] shown in
Revised: We have the quantization example within [ 0 , 1 ] shown in

Original: Fig.
Revised: Fig.

Original: Δ w = − 0.4 Δ w = − 0.

Original: Flipped Δ w = 0.6 Δ w = 0.

Original: Δ w = 0.3 Δ w = 0.

Original: Quantization Step

Original: Flipped Δ w = − 0.7 Δ w = − 0.

Original: [Figure]

Original: Clearly, the element 0 .
Revised: Clearly, the element 0 .

Original: 7 (0 . 4) can be rounded up (down) to 1 . 

Revised: 7 (0 . 4) can be rounded up (down) to 1 .

Original: with + 0 .
Revised: with + 0 .

Original: 3 ( − 0 .
Revised: 3 ( − 0 .

Original: 4) orignal perturbation and ﬂipped to 0 . 

Revised: 4) orignal perturbation and ﬂipped to 0 .

Original: (1 . 0) with − 0 .
Revised: (1 . 0) with − 0 .

Original: 7 ( + 0 . 6) ﬂipped perturbation with − 1 ( + 1) mutation.
Revised: 7 ( + 0 . 6) ﬂipped perturbation with − 1 ( + 1) mutation.

Original: Each ﬂipping operation leads to a ± 1 integer mutation and increases the perturbation to [ 0 . 5 , 1 . 0 ] .
Revised: Each ﬂipping operation leads to a ± 1 integer mutation and increases the perturbation to [ 0 . 5 , 1 . 0 ] .

Original: We prove that we can always ﬁnd k
Revised: We prove that we can always ﬁnd k

Original: = (cid:98)| ∑ i ∆ W (cid:96) m , n , i |(cid:101) elements to ﬂip and reduce | ∑ i ∆ (cid:99) W (cid:96) m , n , i | ≤ 0 . 5 . 
Revised: = (cid:98)| ∑ i ∆ W (cid:96) m , n , i |(cid:101) elements to ﬂip and reduce | ∑ i ∆ (cid:99) W (cid:96) m , n , i | ≤ 0 . 5 . Δ w = − 0.4 Δ w = − 0. Flipped Δ w = 0.6 Δ w = 0. Δ w = 0.3 Δ w = 0.

Revised: Quantization Step

Revised: Flipped Δ w = − 0.7 Δ w = − 0.

Original:  Proof.
Revised: UP [Figure] Proof.

Original: Assume n -th kernel has K elements, a rounded up elements with index set f a have positive perturbation, b rounded down elements with index set f b have negative perturbation, and K = a + b .
Revised: Assume n -th kernel has K elements, a rounded up elements with index set f a have positive perturbation, b rounded down elements with index set f b have negative perturbation, and K = a + b .

Original: Then, we have
Revised: Then, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, let ∑ t ∆ W (cid:96) m , n , t > ∑ j | ∆ W 
Revised: Without loss of generality, let ∑ t ∆ W (cid:96) m , n , t > ∑ j | ∆ W (cid:96) m , n , j |

Original: [Equation]
Revised: [Equation]

Original: m , n , j | ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can always ﬁnd k
Revised: Therefore, we can always ﬁnd k

Original: = (cid:98)| ∑ i ∆ W (cid:96) m , n , i |(cid:101) ≤ (cid:98) 0 .
Revised: = (cid:98)| ∑ i ∆ W (cid:96) m , n , i |(cid:101) ≤ (cid:98) 0 .

Original: 5 · a (cid:101) elements in f a with size of a to ﬂip down and make | ∑ i ∆ (cid:99) W (cid:96) m , n , i | ≤ 0 .
Revised: 5 · a (cid:101) elements in f a with size of a to ﬂip down and make | ∑ i ∆ (cid:99) W (cid:96) m , n , i | ≤ 0 .

Original: For example, if ∑ i ∆ W (cid:96) m , n , i = 3 . 2, we need to ﬂip 3 elements with positive perturbation (rounding up) to negative (rounding down) with − 3 . 0 mutation.
Revised: For example, if ∑ i ∆ W (cid:96) m , n , i = 3 . 2, we need to ﬂip 3 elements with positive perturbation (rounding up) to negative (rounding down) with − 3 . 0 mutation.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: For each kernel | ∑ i [Equation] W (cid:96) m , n , i has the minimum value (cid:12) (cid:12) (cid:12)
Revised: For each kernel | ∑ i ∆ W (cid:96) m , n , i | ≥ 0, ∑ i ∆ (cid:99) W (cid:96) m , n , i has the minimum value (cid:12)(cid:12)(cid:12)

Original: k−|
Revised: k

Original:  ∑ i ∆ W (cid:96) m , n , i | (cid:12) (cid:12)(cid:12) ≤ 0 .5.
Revised: −| ∑ i ∆ W (cid:96) m , n , i | (cid:12)(cid:12)(cid:12) ≤ 0 .

Original: The sufﬁciency of Eq.
Revised: The sufﬁciency of Eq.

Original: (11) has been proven:
Revised: (10) has been proven:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: When a = K , b = 0 and all perturbation = 0 .
Revised: When a = K , b = 0 and all perturbation = 0 .

Original: 5, original ∑ i ∆ W (cid:96) m , n , i have the upper bound 0 .
Revised: 5, original ∑ i ∆ W (cid:96) m , n , i have the upper bound 0 .

Original: 5 · K .
Revised: 5 · K .

Original: Proof.
Revised: Proof.

Original: If we ﬂip k − 1 or k + 1 elements for n -th kernel, the ∑ i ∆ (cid:99) W (cid:96) m , n , i can reduce to (cid:12)(cid:12)(cid:12) k − 1 − | ∑ i ∆ W (cid:96) m , n , i | (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) k + 1 −| ∑ i ∆ W (cid:96) m , n , i | (cid:12)(cid:12)(cid:12) , respectively.
Revised: If we ﬂip k − 1 or k + 1 elements for n -th kernel, the ∑ i ∆ (cid:99) W (cid:96) m , n , i can reduce to (cid:12)(cid:12)(cid:12) k [Equation] 1 − | ∑ i ∆ W (cid:96) m , n , i | (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) k + 1 −| ∑ i ∆ W (cid:96) m , n , i | (cid:12)(cid:12)(cid:12) , respectively.

Original: Obviously, 
Revised: Obviously, (cid:12)(cid:12)(cid:12) k

Original: [Equation]
Revised: [Equation]

Original: With other numbers (cid:54) =
Revised: With other numbers (cid:54) =

Original: k , we can also draw the same conclusions.
Revised: k , we can also draw the same conclusions.

Original: Therefore, when r k ≤ 0 . 5, there is only one value, i.e., the minimum value, with k ﬂipped elements satisfy the Eq.
Revised: Therefore, when r k ≤ 0 . 5, there is only one value, i.e., the minimum value, with k ﬂipped elements satisfy the Eq.

Original: The necessity of Eq.
Revised: The necessity of Eq.

Original: (11) has been proven: argmin
Revised: (10) has been proven: argmin

Original: [Equation]
Revised: [Equation]

Original: [Equation] Similarly, we can extend all conclusions to SQuant-C. For SQuant, we only consider the ﬂipping operation in one quantization step and select the elements whose sign is the same as ∑ i ∆ W (cid:96) m , n , i because ﬂipping with more quantization steps (e.g., ﬂip 0 .
Revised: For SQuant, we only consider the ﬂipping operation in one quantization step and select the elements whose sign is the same as ∑ i ∆ W (cid:96) m , n , i because ﬂipping with more quantization steps (e.g., ﬂip 0 .

Original: 7 to − 1 . 0) and the elements with different perturbation signs will cause a more signiﬁcant perturbation and will violate the Eq.
Revised: 7 to − 1 . 0) and the elements with different perturbation signs will cause a more signiﬁcant perturbation and will violate the Eq.

Original: We explain that in the next section.
Revised: We explain that in the next section.

Original: B. P ROOF OF T OK - k P ERTURBATION A LGORITHM
Revised: B. P ROOF OF T OK - k P ERTURBATION A LGORITHM

Original: Proof.
Revised: Proof.

Original: We will prove the SQuant-E&K will lead to the top- k algorithm.
Revised: We will prove the SQuant-E&K will lead to the top- k algorithm.

Original: We have the composition
Revised: We have the composition

Original: SQuant-E and SQuant-K optimization objective for n -th kernel,argmin
Revised: SQuant-E and SQuant-K optimization objective for n -th kernel,

Original: [Equation]
Revised: [Equation]

Original: (35) which is the ﬁrst two items of Eq.
Revised: which is the ﬁrst two items of Eq.

Original: Without loss of generality, we assume e =
Revised: Without loss of generality, we assume e =

Original: ∑ i ∆ W (cid:96) m , n , i > 0, then
Revised: ∑ i ∆ W (cid:96) m , n , i > 0, then

Original: SQuant needs to ﬂip k = (cid:98) e (cid:101) elements with perturbation > 0 to transform ∑ i ∆ W (cid:96) m , n , i to e − k and is still constant e − k regardless of which k elements are.
Revised: SQuant needs to ﬂip k = (cid:98) e (cid:101) elements with perturbation > 0 to transform ∑ i ∆ W (cid:96) m , n , i to e − k and is still constant e − k regardless of which k elements are.

Original: Therefore, the k elements are only determined by the ﬁrst item of Eq.
Revised: Therefore, the k elements are only determined by the ﬁrst item of Eq.

Original: ∑ i ( ∆ W (cid:96) m , n , i ) 2 .
Revised: ∑ i ( ∆ W (cid:96) m , n , i ) 2 .

Original: We denote f as the index set of the k ﬂipped elements and the original perturbation O = ∆ W (cid:96) m , n , : for n -th kernel.
Revised: We denote f as the index set of the k ﬂipped elements and the original perturbation O = ∆ W (cid:96) m , n , : for n -th kernel.

Original: Therefore, O j > 0 , j ∈ f .
Revised: Therefore, O j > 0 , j ∈ f .

Original: Substituting f and O in Eq.
Revised: Substituting f and O in Eq.

Original: (35), we have the optimization objective for f after ﬂipping k elements,
Revised: (34), we have the optimization objective for f after ﬂipping k elements,

Original: [Equation]
Revised: [Equation]

Original: Therefore, the Eq.
Revised: Therefore, the Eq.

Original: (40) is essentially the top- k perturbation algorithm.
Revised: (39) is essentially the top- k perturbation algorithm.

Original: We can easily extend the top- k algorithm in SQuant-C and design the perturbation update algorithm in B.3.
Revised: We can easily extend the top- k algorithm in SQuant-C and design the perturbation update algorithm in B.3.

Original: B. P ERTURBATION U PDATE A LGORITHM
Revised: B. P ERTURBATION U PDATE A LGORITHM

Original: SQuant-K initializes all rounded elements as ﬂip candidates.
Revised: SQuant-K initializes all rounded elements as ﬂip candidates.

Original: After SQuant-K, we update the ﬂip candidates for SQuant-C as shown in algorithm 4 based on the insight of top- k perturbation algorithm ( B.2).
Revised: After SQuant-K, we update the ﬂip candidates for SQuant-C as shown in algorithm 4 based on the insight of top- k perturbation algorithm ( B.2).

Original: Over SQuant First we deﬁne the situation of k > | e | as “Over SQuant” (line 6).
Revised: Over SQuant First we deﬁne the situation of k > | e | as “Over SQuant” (line 6).

Original: For example, if we have a kernel with e = + 1 . 6, we need to SQuant it to − 0 .
Revised: For example, if we have a kernel with e = + 1 . 6, we need to SQuant it to − 0 .

Original: 4 to satisfy in ( − 0 .
Revised: 4 to satisfy in ( − 0 .

Original: 5 ] with ﬂipping k = 2 elements { 2 . 6 , 2 . 7 } to { 2 , 2 } .
Revised: 5 ] with ﬂipping k = 2 elements { 2 . 6 , 2 . 7 } to { 2 , 2 } .

Original: Obviously, when SQuant-C needs this kernel to calibrate, the last element 2 .
Revised: Obviously, when SQuant-C needs this kernel to calibrate, the last element 2 .

Original: 7 should be the ﬁrst and the only candidate (line 7,8) to ﬂip back to the original rounded number 3 to make the e =
Revised: 7 should be the ﬁrst and the only candidate (line 7,8) to ﬂip back to the original rounded number 3 to make the e =

Original: + 0 .
Revised: + 0 .

Original: 6, due to it has the largest element perturbation in the k ﬂiped elements and the smallest element perturbation ( |− 0 .
Revised: 6, due to it has the largest element perturbation in the k ﬂiped elements and the smallest element perturbation ( |− 0 .

Original: 3 | < 0 . 5 ) after it ﬂips back.
Revised: 3 | < 0 . 5 ) after it ﬂips back.

Original: It is vice versa for e < 0.
Revised: It is vice versa for e < 0.

Original: Under SQuant For “Under SQuant” (line 9), we need to make the ﬁrst un-ﬂipped element as the ﬂip candidate (line 10, 11) for SQuant-C, and will lead the kernel to ”Over SQuant” with absolute kernel perturbation in ( 0 . 5 , 1 . 0 ) when SQuant-C ﬂips this element of the kernel. 
Revised: Under SQuant For “Under SQuant” (line 9), we need to make the ﬁrst un-ﬂipped element as the ﬂip candidate (line 10, 11) for SQuant-C, and will lead the kernel to ”Over SQuant” with absolute kernel perturbation in ( 0 . 5 , 1 . 0 ) when SQuant-C ﬂips this element of the kernel. Finally, each kernel has only one candidate ﬂip element for SQuant-C to satisfy Eq.

Original: Algorithm 4: Perturbation Update Algorithm. Input perturbation p ;
Revised: In practice, it is easy to fuse the perturbation update algorithm with the ﬂip algorithm without extra overhead.

Original: Output
Revised: The original optimization problem described by Eq.

Original: 1 def UpdatePerturbation( p ) : e =
Revised: (4) is NP-hard with O ( M · 2 NK ) .

Original: ∑ i p i //
Revised: Based on the

Original: Accumulated perturbation (signed CASE). p [ e · p < 0 ] = 0 // Disable Elements/kernels with different sign from e .
Revised: SQuant approximation, the new optimization objective is to minimize CASE whose complexity is a algorithm with a signiﬁcant complexity reduction to O ( n log ( k )) for sub-

Original:  k = (cid:98)| e |(cid:101) // Flip kelements/kernels based on the CASE. f = TopK( | p | , k ).
Revised: Our experiments show that after SQuant-K pre-optimization, SQuant-C only requires a tiny top- k number, such as k = 32, to satisfy all cases.

Original: / Indices of k largest perturbation. if k > | e | then // Over SQuant.
Revised: For a 3 × 3 kernel with 9 elements, SQuant-K only needs k = 4 because the kernel CASE is always ≤ 0 .

Original: i = f k //
Revised: 5 K = 4 .

Original: The k -th (last) element of f . v = p i // 0 . 5 ≤ | v |
Revised: Finally, their complexity can reduce to linear, O ( M · N · 5 ) for SQuant-C and O ( MN · 9 · 2 ) for SQuant-K.