Paper: $R^2$: Range Regularization for Model Compression and Quantization (ICLR-2024)

Reviewer comment (Experiments):
Many prior works have argued the impact of outliers on the quantization results, but this paper only references two from 2020 and 2021, and only references KURE from 2020 as the baseline for comparison, which lacks persuasiveness.

Author response:
There are a couple of prior works tackled outliers in model quantization. We mainly compared $R^2$ with DFQ, PACT, LSQ, and EWGS in our paper as we thought they are representative prior works for outlier handing. DFQ focused that each channel has different weight ranges so that per-layer quantization suffers from a channel which has a very wide weight range. DFQ resolves the issue by applying channel-wise scaling factor to make each channel have similar weight ranges. From our experiment with DFQ (in Table 2), it seems that enforcing to reduce weight range at pre-training by $R^2$ is much better than DFQ approach. PACT highlighted outliers in activation, not weight. PACT removes outliers in activation by clipping them and the clipping thresholds are gradually adjusted as they are learnable parameters during quantization. LSQ and EWGS take a weight clipping approach. They clips weights at its initialization stage before starting a fine-tuning phase for quantization to deal with weight outliers.

Also there are more recent works that consider outliers in model quantization as you mentioned such as [1] and [2]. [1] proposed activation smoothing to deal with outliers in activation of LLM. It applies smoothing factors for the outlier activation and applies another scaling factors for corresponding weights to compensate the outlier smoothing factors. It mainly focuses on activation outliers, not weight outliers. [2] proposed a similar concept of [1], but it also proposed token-wise clipping to deal with outliers. We may want to conduct more studies with those works in the context of activation outliers and LLM quantization/compression as one of our future works.

Reviewer comment (Experiments):
It is hard to prove that applying R2 can lead to higher accuracy after quantization than not applying it. First, only four quantization methods are discussed in the experiments, and they can not cover all the quantizers. Secondly, the existing experiments in the paper show that the results w/o R2 can also achieve higher model accuracy than that with R2 (Table 4).

Author response:
EWGS basically takes +- 3 sigma of weight distributions in a pre-trained model as a clip point for each layer. As you suggested, we conduct an ablation study varying clip thresholds from +- 3 sigma to +- 1 sigma for MobileNetV2 2bit quantization using EWGS. Please note that +- 3 sigma covers 99.7% and +- 1 sigma covers 68.3% of weights assuming normal distribution of weights. With $R^2$, EWGS can achieve **57.36%** top-1 accuracy on ImageNet while weight clipping does not show noticeable difference varying clipping thresholds.

* **MobileNet-V2 with 2bit weight/activation EWGS**
| Clip threshold | 3 sigma | 2 sigma | 1 sigma |
|-----|:----:|:----:|:----|
| Top-1 Acc.     |  53.93  |  54.01  |  53.87  |

We also would like to discuss about the concern that $R^2$ does not seem to improve quantization accuracy, especially for Table 4. As we discussed in the paper, we think that higher-bit quantization-aware training can cover the outlier as a higher-bit quantized model has enough weight representation power. Therefore, in 4bit QAT result, we can see with and without $R^2$ quantization shows almost the same quantization accuracy while 2bit QAT result clearly shows effectiveness of $R^2$ in Table 4. Please note that EWGS result in Table 4 seems somewhat marginal, but in Table 3, we can see that $R^2$ meaningfully improves EWGS accuracy for MobileNet-V1 and V2. So, we think again that weight clipping would not sufficient for handling outliers in model quantization.

For the concern that we only showed accuracy improvement by $R^2$ with a limited number of quantization methods. As you pointed, we conducted the comparison study with DFQ, AdaRound, PACT, LSQ, KURE, and EWGS. There are bunch of different quantization methods and it is very hard to show the effectiveness of $R^2$ with all of them. To do our best, we choose diverse quantization methods in our study such as PTQ (equalizing per-channel weight range (DFQ), adaptive quantization bin assignment (AdaRound)) and QAT (activation clipping (PACT), learnable step-size (LSQ), gradient scaling (EWGS), weight regularization for stable quantization (KURE)). Also, we added a comparison study with more recent quantization works as shown in our response for the first reviewer vWZn, PD-Quant and SQuant. In the all comparison study, we can see the clear  evidence that $R^2$ helps quantization methods better at low-bit.

Reviewer comment (Experiments):
I am interested in how the experimental results would differ if we applied the clip function instead of R2 regularization in model training.

Author response:
EWGS basically takes +- 3 sigma of weight distributions in a pre-trained model as a clip point for each layer. As you suggested, we conduct an ablation study varying clip thresholds from +- 3 sigma to +- 1 sigma for MobileNetV2 2bit quantization using EWGS. Please note that +- 3 sigma covers 99.7% and +- 1 sigma covers 68.3% of weights assuming normal distribution of weights. With $R^2$, EWGS can achieve **57.36%** top-1 accuracy on ImageNet while weight clipping does not show noticeable difference varying clipping thresholds.

* **MobileNet-V2 with 2bit weight/activation EWGS**
| Clip threshold | 3 sigma | 2 sigma | 1 sigma |
|-----|:----:|:----:|:----|
| Top-1 Acc.     |  53.93  |  54.01  |  53.87  |

Reviewer comment (Experiments):
Based solely on the experiments in the paper, it is challenging to discern whether the proposed $R^2$ method is more effective in terms of generalization compared to other regularization methods.

Author response:
Thank you for the helpful comments.

While regularization (like L2-regularization) is a popular technique to improve model generalization during training, its final generalization capability (indicated by the val/test accuracy) after model compression may not be sufficient. In $R^2$, we are demonstrating that carefully-crafted and targeted regularization techniques (like our smin-max) can be helpful for the generalization AFTER compression (again indicated by the val/test accuracy) as shown in Table 2 in Section 4.

Reviewer comment (Experiments):
The paper demonstrates the effectiveness of $R^2$ based on weight distribution, but it does not address activations. Since the experiments in the paper show a significant compression ratio for activations, with activations also sharing the same number of bits as weights, it seems essential to include an analysis or discussion regarding activations.

Author response:
As you pointed, activation is not in our scope. PACT addressed outliers in activation by applying learnable clipping thresholds on activation during quantization. The reason why we aggressively quantize activation as well in our study is that we want to show $R^2$ does not ruin activation quantization even it changes the shape of weight distribution. If $R^2$ negatively affected activation quantization, then we will not be able to get better accuracy for weight and activation quantization. In our study, $R^2$ helps quantization accuracy even at low-bit for weight and activation together, so we think $R^2$ does not hurt activation quantization.

Reviewer comment (Evaluation):
From the experiments in the paper alone, it's difficult to ascertain the effectiveness of the proposed $R^2$ method in terms of generalization. Are there any experimental data available to assess the Generalization Gap?

Author response:
Thank you for the helpful comments.

While regularization (like L2-regularization) is a popular technique to improve model generalization during training, its final generalization capability (indicated by the val/test accuracy) after model compression may not be sufficient. In $R^2$, we are demonstrating that carefully-crafted and targeted regularization techniques (like our smin-max) can be helpful for the generalization AFTER compression (again indicated by the val/test accuracy) as shown in Table 2 in Section 4.

Reviewer comment (Experiments):
The paper only presents results for small models like CNN or MobileBERT, primarily focusing on CNN results. Are there any results showcasing the application of the proposed method to larger models such as ResNet-101 or LLM (e.g., transformer architectures like GPT-J)? Is there experimental evidence showing the more general applicability of the proposed approach?

Author response:
To resolve your concern that we conducted comparison studies with $R^2$ using small models like MobileNet, Resnet18, and MobileBERT, we performed further comparison studies using larger models as you suggested. In the following tables, we can see that $R^2$ still shows significant accuracy improvement than without $R^2$ cases. From the large model quantization result in the tables and MobileBERT result in our paper, we believe that the $R^2$ would be effective for a large language model (LLM) as well. We are planning to apply $R^2$ in LLM quantization/compression with state-of-the-art techniques.


* **AdaRound 4bit weight / 8bit activation all-layer quantization**
|   Model  | W/O $R^2$ | R_inf | R_Margin | R_SMM |
|----------|:---------:|:-----:|:--------:|:------|
| Resnet50 |   70.51   | 71.00 |   74.38  | 72.55 |
| Resnet101|   67.30   | 75.45 |   75.03  | 73.37 |

* **SQuant 4bit weight / 8bit activation all-layer quantization**
|   Model  | W/O $R^2$ | R_inf | R_Margin | R_SMM |
|----------|:---------:|:-----:|:--------:|:------|
| Resnet50 |    70.87  | 72.65 |   71.57  | 72.83 |
| Resnet101|    68.34  | 73.11 |   74.98  | 74.63 |