Paper: DECOUPLE QUANTIZATION STEP AND OUTLIER-MIGRATED RECONSTRUCTION FOR PTQ (ICLR-2024)

Reviewer comment (Presentation):
It doesn’t seem appropriate that the arrangements and citations of figures and tables. For instance, the first citation of Figure 2 is on page 2, and Figure 2 is on page 5.

Author response:
## 3.2 Arrangements and Citations of Figures and Tables

Due ICLR’s limited space, the placement and citation of figures and tables is a little far. To make full use of limited PDF space, we place experiment tables as together as possible. It is a better choice to place Figure 2 on page 2, thus readers can better understand our OMR’s motivation. We has re-arranged our Figures as suggested, which can be seen on our re-polished paper PDF.

Reviewer comment (Evaluation):
In Outlier-Migrated Reconstruction, how can the sensitivity of each channel be measured, if only a portion of channels will be added?

Author response:
## 3.3 Sensitivity Criterion on OMR

If we only select the most sensitive channels on the most sensitive OMR-Solvable structures, OMR can be more efficient.

The sensitivity criterion on channels is as follows, where we choose the sum of each channels’ activation in outlier range$[X_{clip}, 2X_{clip}]$, then we sort each channel in order and choose the top-k percent for OMR.  

$$
S_i=\sum_j{||x_{ij}||^2},
$$ where $x_{i,j}$ is the $j$-th values in range $(X_{clip}, 2X_{xlip}]$  of $i$-th channel.
$$ 
S^\*={sort}(S_i|i=1,...,N), S^{\*k}={Top\\_K}(S^{\*})
$$


Other sensitivity criterion can also be applied, which can be explored as a future work.

Reviewer comment (Theory):
It seems that OMR is designed to mitigate outlier of activations and outlier of weight parameters can’t be utilized with this method. Does OMR help mitigate outlier of weight parameters?

Author response:
## 3.4 OMR can be Used on Weight

It is right that OMR can also be used on weight like activation. **The core of OMR, migrating outliers into safe clipping range then compensating in the following layers, can be extended to weight**. Detailed analysis is as follows,

- When OMR is used on activation, we want to save outlier activation in range $[X_{clip}, 2X_{clip})$. According to equal mathematical transformation, we can transform activation modification into weight modification. We can modify the $i$-th and $i+1$-th weight as follows, thus a functionally-identical network is caused without retraining except for more outlier activation being preserved.

$$
\begin{equation}W_{i,j}^{l\prime} = \begin{cases}W_{i,j}^{l}, & \text{if } 0<j≤c_{out}; \\\\W_{i,j-c_{out}}^{l}, & \text{if } c_{out}<j≤2c_{out}.\end{cases}
b_{j}^{l\prime} = \begin{cases}b_{j}^{l}, & \text{if } 0<j≤c_{out}; \\\\b_{j-c_{out}}^{l}-X_{clip}, & \text{if } c_{out}<j≤2c_{out}.\end{cases}
\end{equation}
$$

$$
\begin{equation}W_{i,j}^{l+1\prime} = \begin{cases}W_{i,j}^{l}, & \text{if } 0<i≤c_{out}; \\\\W_{i-c_{out},j}^{l}, & \text{if } c_{out}<i≤2c_{out}.\end{cases}
\end{equation}
$$

- When OMR is used on weight $W$, we want to save outlier weight in range $[W_{clip}, 2W_{clip)}$. We can directly transform $i$-th and $i+1$-th weight as follows,
    
    $$
    \begin{equation}W_{i,j}^{l\prime} = \begin{cases}W_{i,j}^{l}, & \text{if } 0<j≤c_{out}; \\\\W_{i,j-c_{out}}^{l}-W_{clip}, & \text{if } c_{out}<j≤2c_{out}.\end{cases}
    b_{j}^{l\prime} = \begin{cases}b_{j}^{l}, & \text{if } 0<j≤c_{out}; \\\\b_{j-c_{out}}^{l}, & \text{if } c_{out}<j≤2c_{out}.\end{cases}
    \end{equation}
    $$
    
    $$
    \begin{equation}W_{i,j}^{l+1\prime} = \begin{cases}W_{i,j}^{l}, & \text{if } 0<i≤c_{out}; \\\\W_{i-c_{out},j}^{l}, & \text{if } c_{out}<i≤2c_{out}.\end{cases}
    \end{equation}
    $$

Reviewer comment (Novelty):
My main concern is about the novelty. The proposed Outlier-Migrated Reconstruction is well studied by previous works [1].

Author response:
## 2.1 Novelty of OMR
We compare OMR with OCS in Sec.4.3. It is true that OCS studied the outlier problem. However, the outlier problem is **not well studied by OCS** due to its shortcomings.
- OCS does not enlarge quantization levels. OCS saves outliers **while sacrifices the precision of inner values**.
    - OCS saves outliers by halving operation. OCS squeezes the outlier values into a narrower range, however, which also squeezes the inner values into a narrower range. Thus OCS save outliers at the sacrifice of the precision of inner values.
    - OCS failed to help achieve better performance on low-bit quantization.
- **Our OMR, for the first time, proposes to save more outliers while preserve the precision of inner values**. Thus OMR actually enlarge quantization levels and equals to earning one more unavailable bit, which is extremely helpful for 2,3-bit, quantization.
- **Their comparison visualization can be seen in the supplementary PDF Sec-2**.

Reviewer comment (Theory):
It is hard for me to understand why DJOS works. If the quant-step is fixed and the dequant-step is learnable, it is the same with the learning process of BN. In other words, network quantization with BN has already a mechanism of learned dequant-step.

Author response:
## 2.2 DJOS V.s. BN
- Why DJOS works:
    Previous methods use a single and fixed quant-step of weight $s_w$. 
    For the first time, we propose 1) to make $s_w$ learnable; 2) to decouple single $s_w$ into $s_w$ and $s^\prime_w$; 3) to make $s_w$ fixed and $s^\prime_w$ learnable.
    - Why making $s_w$ learnable :
        - In general, more params under necessity during training will benefit model's optimization[1], especially under a extremely low-bit param space. Experiments also demonstrate decoupling brings one more parameter and brings lower reconstruction loss (both training and evaluation).
    - Why decoupling single $s_w$ into $s_w, s^\prime_w$ works:
        - Since integer weight can be obtained before inference, we can safely decouple traditional single $s_w$ into $s_w, s^\prime_w$. Integer activation need real-time calculation, thus we can not decouple activation’s quant-step $s_x$.
    - Why fixed $s_w$ and learnable $s^\prime_w$ works:
        - PTQ is in shortage of enough labeled training set and long time optimization,  and STE’s mismatched gradient is a little far way from the floor operation. In extremely-low-bit quantization, these two problems make the optimization of $s_w$ extremely difficult. Thus a fixed $s_w$ is better than wrongly-updated $s_w$. Therefore, it will achieve better performance to make the dequant-step $s^\prime_w$ learnable only.
- Comparison Between DJOS and BN:
    - **It is not true to say “When we have BN, we do not need fake quantization”. They are two orthogonal methods, BN helps for fp32 model training and DJOS helps for low-bit quantization.**
    - Correlation: From the point of math formula, DJOS share similar form as BN.
$$
\text{DJOS}:\hat{w}=s^\prime_w\cdot \\{\lfloor \frac{x}{s_w}\rfloor +h(\alpha)\\}\ \ \ \ \text{with $s_w$ fixed and $s^\prime_w$ learnable}
$$
$$BN: {y}=\gamma\cdot \frac{x-\mu}{\sigma}+\beta, \text{with $\mu,\sigma$ statistically updated and $\gamma, \beta$ learnable}
$$
- However, they are diffenrent in physical sense.
    - BN is used to reduce the internal covariate shift, which **maps fp32 values(any distribution) into a $N(0,1)$ Gussian-like distribution then learn to recover them**. It is usually used on activation.
    - DJOS is a fake quantization method, which **maps fp32 values into integer space then re-map them into concrete fp32 counterparts. DJOS does not change original distribution**. **DJOS is used on weight and can not be used on activation**, because integer weight can be obtained before inference while integer activation has to be computed on line during inference.

[1] I. Goodfellow, 2016. Deep Learning. MIT Press

Reviewer comment (Experiments):
In the experiments, the baselines of different methods are not the same, making the comparison unfair.

Author response:
## 2.3 Baseline Pretrained Model
We borrowed FP32 pretrained model from BRECQ(https://github.com/yhhhli/BRECQ),  also the same as QDROP and NWQ. Thus our comparison is fair.  PD-Quant states different FP32 Acc@1. Their FP32 Acc@1 on their paper  are as follows:
| FP32 Acc@1| Mobile-V2 | Res-18 | Reg-600 | Mnas-2.0 |
| --- | --- | --- | --- | --- |
| DOMR(ours) | 72.49 | 71.08 | 73.71 | 76.68 |
| BRECQ | 72.49 | 71.08 | 73.71 | 76.68 |
| QDROP | 72.49 | 71.08 | 73.71 | 76.68 |
| NWQ | 72.49 | 71.06 | 73.71 | 76.68 |
| PD-Quant | 72.62 | 71.01 | 73.52 | 76.52 |

We can see the FP32 Acc@1 for Mobile-V2, Res-18, Reg-600, Mnas-2.0 in PD-Quant is slightly different from BRECQ, QDROP, NWQ and ours. Their **FP32 difference is no more than 0.2 percentage**. However, our DOMR is **at least 1 percentage better than others, at least 2 percentage better than PD-Quant**,  in W3A3 and W2A2. Thus DOMR’s performance is still robust and it is fair to say DOMR set up a new SOTA for PTQ.

Reviewer comment (Experiments):
It is better to study weight-only quantization (DJOS) and compare with previous methods to evaluate the effectiveness of DJOS.

Author response:
## 2.4 Weight-Only Quantization with DJOS
We can see DJOS outperforms existing PTQ SOTA in weight-only quantization.
|  | QDROP | PD-Quant | NWQ | DJOS |
| --- | --- | --- | --- | --- |
| W2A32-Res-18 | 66.02 | 66.66 | 67.11 | 67.83 |
| W3A32-Res-18 | 70.04 | 70.13 | 70.16 | 70.38 |

Reviewer comment (Experiments):
What's the performance of weight-only quantization using the proposed method?

Author response:
## 2.4 Weight-Only Quantization with DJOS
We can see DJOS outperforms existing PTQ SOTA in weight-only quantization.
|  | QDROP | PD-Quant | NWQ | DJOS |
| --- | --- | --- | --- | --- |
| W2A32-Res-18 | 66.02 | 66.66 | 67.11 | 67.83 |
| W3A32-Res-18 | 70.04 | 70.13 | 70.16 | 70.38 |

Reviewer comment (Experiments):
Which pretrained models are used as baseline?

Author response:
## 2.3 Baseline Pretrained Model
We borrowed FP32 pretrained model from BRECQ(https://github.com/yhhhli/BRECQ),  also the same as QDROP and NWQ. Thus our comparison is fair.  PD-Quant states different FP32 Acc@1. Their FP32 Acc@1 on their paper  are as follows:
| FP32 Acc@1| Mobile-V2 | Res-18 | Reg-600 | Mnas-2.0 |
| --- | --- | --- | --- | --- |
| DOMR(ours) | 72.49 | 71.08 | 73.71 | 76.68 |
| BRECQ | 72.49 | 71.08 | 73.71 | 76.68 |
| QDROP | 72.49 | 71.08 | 73.71 | 76.68 |
| NWQ | 72.49 | 71.06 | 73.71 | 76.68 |
| PD-Quant | 72.62 | 71.01 | 73.52 | 76.52 |

We can see the FP32 Acc@1 for Mobile-V2, Res-18, Reg-600, Mnas-2.0 in PD-Quant is slightly different from BRECQ, QDROP, NWQ and ours. Their **FP32 difference is no more than 0.2 percentage**. However, our DOMR is **at least 1 percentage better than others, at least 2 percentage better than PD-Quant**,  in W3A3 and W2A2. Thus DOMR’s performance is still robust and it is fair to say DOMR set up a new SOTA for PTQ.

Reviewer comment (Reproducibility):
Could you provide the detailed quantization algorithm?

Author response:
## 2.5 Detailed Algorithm can be seen in the supplementary PDF Sec-3.

Reviewer comment (Theory):
The authors base their Outlier-Migrated Reconstruction (OMR) method on the assumption that the activation function is ReLU. However, it's essential to discuss the applicability of OMR to models that use non-ReLU functions. For instance, MobileNetV3 employs h-swish, and ViT utilizes GeLU. A more comprehensive discussion about the adaptability of OMR to such activation functions would enhance the paper's coverage.

Author response:
## 1.2 OMR on ReLU, h-swish, GeLU and others.
Except for Conv-ReLU-Conv and positive nonlinear function like ReLU6/Sigmoid, **the core of OMR, migrating outliers into safe clipping range then compensating in the following layers, can be extented to other structures like Conv-Conv, Linear-Linear and nonlinear function like h-swish, GeLU.** 

$x_{int}^{min}(x_{int}^{max}),X_{clip}^{min}(X_{clip}^{max})$ denote the min(max) value(clipping bound) in integer and fp32 domain.  Activation fake quantization into B bits can be obtained by,
$$
\hat{x}=s_x\cdot x_{int}=s_x\cdot clip\\{{\lfloor\frac{x}{s_x}\rceil}, x_{int}^{min}, x_{int}^{max}\\}
$$
Detailed visualization is shown in **Sec4 of the re-polished supplementary PDF**.

- When activation function is **ReLU**
    - $x_{int}^{min}=0, x_{int}^{max}=2^B-1$ , $X_{clip}^{min}=0,X_{clip}^{max}=(2^B-1)*s_w$, The inner values is in range $[X_{clip}^{min}, X_{clip}^{max}]$. The outliers is in range $(X_{clip}^{max}, +\infty)$. We want to save outliers in $(X_{clip}^{max},2X_{clip}^{max}]$. Thus we can migrate $x$ by $X_{clip}^{max}-X_{clip}^{min}=(2^B-1)*s_w$ along the negative $x$-axis to save values in  $(X_{clip}^{max},2X_{clip}^{max}]$. 
- When activation function is **GeLU, h-swish** or **no activation function**, activation $x$ distributes in both negative and positive axis:
    - $x_{int}^{min}=-(2^{B-1}-1), x_{int}^{max}=2^{B-1}-1$ ,$X_{clip}^{min}=-(2^{B-1}-1)*s_w,X_{clip}^{max}=(2^{B-1}-1)*s_w$, The inner values is in range $[X_{clip}^{min}, X_{clip}^{max}]$. The outliers is in range $(-\infty, X_{clip}^{min})$ and  $(X_{clip}^{max}, +\infty)$. We want to save outliers in $[2X_{clip}^{min},X_{clip}^{min})$ and  $(X_{clip}^{max},2X_{clip}^{max}]$. Thus we can migrate $x$ by  $X_{clip}^{max}-X_{clip}^{min}$ along the negative $x$-axis to save values in $(X_{clip}^{max},2X_{clip}^{max}]$. We can migrate $x$ by $X_{clip}^{max}-X_{clip}^{min}$ along the positive $x$-axis to save values in $[2X_{clip}^{min},X_{clip}^{min})$.

Reviewer comment (Evaluation):
The paper mentions that OMR involves duplicating channels for both weights and activations. This duplication may introduce additional computational overhead, but the paper lacks an in-depth analysis of this aspect. Providing a more detailed examination of the computational costs could be better.

Author response:
## 1.3 Detail of Computational Costs
Our OMR does no need the Fake Quant process. It can be finished in **1 CPU-second** based on offline mathematical transformation. It introduces extra FLOPs during Real Quant.
- For k=0.2 or k=0.5, where we save 20% or 50% channles' outliers, if
FLOPs of original OMR-Solvalble structures is 10M, then FLOPs of
OMR-applied ones is 12M or 15M.
- Trade-off analysis on MobileNet-V2 between FLOPs/BOPs and its performance is as follows. We denote the FLOPs 327M and BOPs 1.8G of MobileNet-V2 on W2A2 as 1.0, on which of other methods is based. **Note** that "W2A3" can not run on 2-bit hardware while our **OMR can run on 2-bit hardware with W2A3 performance** as our paper PDF.
    | Mobile-V2 | W2A2 | FLOPs | BOPs | Run on 2bit Hardware |
    |-|-|-|-|-|
    | NWQ+Ori_Net | 26.42 | 1.0 | 1.0 | √ |
    | NWQ+Channel-plus -Net | 32.02 | 2.0 | 2.0 | √ |
    | **OMR_0.0+Ori-Net(DJOS)** | **31.43** | **1.0** | **1.0** | √ |
    | OMR_0.3+Ori-Net | 36.33 | 1.3 | 1.3 | √ |
    | **OMR_0.5+Ori-Net** | **39.35** | 1.5  | 1.5 | √ |
    | OMR_0.7+Ori-Net | 39.75 | 1.7 | 1.7 | √ |
    | OMR_1.0_Ori-Net | 41.65 | 2.0 | 2.0 | √ |

Reviewer comment (Evaluation):
The inclusion of an overall cost breakdown in Table 11 is appreciated. However, it would be even more informative if the authors could provide a specific breakdown of the runtime costs associated with Quant Time in the Fake Quantization process. This would offer a more granular view of the computational expenses involved in the proposed method, contributing to a deeper understanding of its practical implications.

Author response:
## 1.4 Runtime Costs in Fake-Quant
The specific breakdown of runtime costs of our DOMR(DJOS+OMR) in ResNet-18 fake quantization period is as follows：

1.4.1. Fold BN into its preceding convolution in FP32 model: 0.02 second.

1.4.2. Initialize the weight’s single quant-step $s_w$ with MSE minimization, decouple the single $s_w$ into $s_w,s^\prime_w$:  0.5 second

1.4.3. OMR’s sensitivity analysis on channels + choosing sensitive channels for OMR + weight adjustment of the current layer and the next layer: 0.42 minutes

1.4.4. Jointly optimize $s_w^\prime,s_x$, AdaRound param of weight $\alpha$ on 1024 random calibration images for 20K iterations: 44 minutes

All the time consumed on our DOMR fake quantization: 0.02s + 0.5s + 0.42 min + 44 min ≈ 44.43 min

Reviewer comment (Experiments):
It seems that using decoupled step sizes for weights can also generalize to quantization-aware training, have the authors experimented with this?

Author response:
## 1.5 DJOS on QAT(Quantization-Aware Training)
| ResNet18 | W3A3 | W2A2 |
|-|-|-|
| LSQ_QAT | 69.57±0.08 | 66.50±0.21 |
| LSQ_QAT+DJOS | 68.95±0.07 | 62.57±0.18 |

We re-implement LSQ-QAT W2A2 and W3A3 on ResNet-18. Experimental results over 3 runs can be seen above. We can see LSQ_QAT+DJOS performs worse than original LSQ_QAT. 

The reason is that model weight in QAT is learnable and its distribution changes over training process. Whether to learn model weights is also one of the biggest difference for PTQ and QAT.

Thus a fixed quant-step $s_w$ of DJOS can not be adaptive to a weight distribution that changes over training process. However, in PTQ, model weight is almost fixed, with at most one quant-step changes due to AdaRound. Therefore,  with limited calibration set and limited optimization time in PTQ, a fixed quant-step $s_w$ will perform better on an almost-fixed weight distribution.