Paper: Rotation Invariant Quantization for Model Compression (ICLR-2024)

Reviewer comment (Theory):
The central assumption, $\||w\|| = \||\hat{w}\|| + o(\||w\||)$, appears to be contentious. For instance, with fixed-bit quantization, the discrepancy between $\||w\||$ and $\||\hat{w}\||$ is proportional to $\||w\||$, or in other words, $O(\||w\||)$.

Author response:
[A]  The order of discrepancy depends on the rate regime that is considered. This paper considers the high-rate regime as mentioned explicitly throughout the paper.

As the reviewer pointed out, the quantization error in each layer is bounded by $\Vert \epsilon \Vert \leq \sqrt{n}\frac{\Delta}{2}$. Assuming that the range of $w$ is symmetric around zero for simplicity, then, $\Delta \approx \frac{2 max(w)}{2^R}$, where $R$ is the rate of quantization. Hence, the error $\Vert \epsilon\Vert \leq \sqrt{n}\frac{max(w)}{2^R} \approx \frac{\Vert w \Vert}{2^R}$, and thus, the $o(\cdot)$ is contentious. 

Yet, $O(\Vert w\Vert)$ does not reflect the high rate regime that is considered in the paper. In particular, when letting $R = O(\log^c(\Vert w \Vert))$ for $c>1$, then, we obtain the desired $o(\Vert w \Vert)$. 
Moreover, even when $R = O(\log(\Vert w \Vert))$, which results in a relaxed error of $O(1)$, still provides meaningful results. In this case, the resulting approximation error would be $O(1/\Vert w \Vert)$. 

We thank the reviewer for pointing this out and agree that the discussion above would improve the presentation and clarity of the lemma. In the revised version, we explicitly restrict the lemma to the high-rate regime, and we provide the above discussion in the proof of Lemma 1.

Reviewer comment (Theory):
The surrogate model feels somewhat contrived. Moreover, the method for deriving $\tilde{w}_{\ell}$ is not clear. For instance, are the angles $\theta_{\ell}$ specified? The notion of being "uniformly distributed on a cone" is ambiguous due to its unbounded norm. As a result, Theorem 1 requires a more comprehensive problem definition.

Author response:
[A]  Note that $\hat w_\ell$, and hence, its norm and  $\theta_\ell$ are specified by the quantization. The $\tilde w_\ell$ models the quantized vector $\hat w_\ell$. In particular, the surrogate model considers all the vectors that are $\theta_\ell$ away by using a random rotation matrix $U(\theta_\ell)$ (see Figure 3 for illustration). That is, $\tilde w_\ell$ is a realization of this random rotation. Note that the norm of $\tilde w_\ell$ is the same as $\hat w_\ell$ (and hence, bounded). Thus, any realization of  $\tilde w_\ell$ is uniformly distributed on the cone's ring. 

In the revised version we clarify this point after introducing the surrogate model.

Reviewer comment (Theory):
The link between distortion (related to weights) and deviation (pertaining to output) is unclear.

Author response:
The link between the distortion and the deviation is given in Proposition 1 proof. Specifically, we prove that both the distortion and the deviation scale with the quantization rate as $O(1/k^2)$. To clarify this link, Proposition 1 now explicitly states this connection in the revised version.

Reviewer comment (Evaluation):
Mixed precision is difficult to apply in the industrial scenarios. It usually requires the design of specialized chips to achieve a slight increase in inference speed, so I have doubts about the impact of the proposed method.

Author response:
[A] This study does not target any specific hardware and allows running on standard CPU/GPU with 8 bits after de-quantizing the weights. By doing this, we do not benefit from acceleration from the mixed precision (only from the 8-bit execution), and the main benefit is a reduced model size. 

One may consider a specialized hardware solution, but in practice, if all symbols are encoded with less than 8 bits, it allows running the model with 8-bit computation, rather than using a mixed-bit on dedicated hardware and achieving a slight increase in inference speed, as the reviewer mentioned. Accordingly, it is not necessary to restore the weights to full precision. Our compression scheme facilitates such quantization, as we point out in the Remark below Proposition 2.

Still, we would like to emphasize that our goal is to reach the smallest model size rather than accelerate the running time, and it should be the main takeaway from this study.

Reviewer comment (Experiments):
The experimental result lacks a comparison of inference speed of compressed model, between the proposed method and existing works.

Author response:
[A] As mentioned, the objective of this study is to minimize the model size under the cosine distance constraint. In Section A.8.2, we did perform full quantization (i.e., quantizing weights by RIQ and the activations by [Wu 20'](https://arxiv.org/pdf/2004.09602.pdf)), where the de-quantized weights were packed in 8-bits, allowing to further accelerate the inference. Nevertheless, the same speedup was obtained also by a simple uniform 8-bit quantization. So, the main benefit of RIQ is the reduced model size and not its acceleration.

Reviewer comment (Experiments):
The experimental result lacks a comparison of lightweight structure including separable convolution (like mobilenetv2) with existing methods like AdaRound or BRECQ.

Author response:
[A] Please note that Table 1 provides comparisons with the AdaRound and BERCQ on the Resnet50 model. 
RIQ is a powerful method for models with high redundancy and attains good compression results without further tuning. MobileNet_v2 is a relatively compact and efficient model (with a size of only 13MB), and hence, it cannot benefit significantly from RIQ compared to the BRECQ method, which applies further optimization for tuning the weights and requires a few hours to tune on GPU. 

As per the reviewer's request, we evaluated RIQ on mobilenet_v2 on ImageNet1k, and the results are given below:

| Method   | compression   | Accuracy(%) | Reference(%) | Drop (%) |
|----------|---------------|-------------|--------------|----------|
| BRECQ    | $\times 8$    | $71.66$     | $72.49$      | $0.83$   |
| AdaRound | $\times 8$    | $69.78$     | $72.49$      | $2.71$   |
| RIQ      | $\times 5.33$ | $71.19$     | $71.87$      | $0.68$   |

Reviewer comment (Writing):
I suggest the author provide more descriptions about the scenarios where the mixed-precision model can be applied. The advantages of mixed-precision models can be manifested in NLP-type structures.

Author response:
[A] We thank the reviewer for this suggestion. Indeed, NLP-type structures and Large-Language Models (LLMs) in particular, have high redundancies that RIQ can exploit. Indeed, the benefit of a reduced model size is twofold. First, the low memory that is required to store the model. Second, the lower loading and extracting time of the model weights may be significant in LLMs. For example, Llama2-70B requires 140GB to load on the device. Whereas, assuming $\times 4$ compression with RIQ (as demonstrated in Table 4 in the appendix for Llama-7B) would require only 35GB, which can fit a single GPU with 40GB memory (e.g., Nvidia A100).  

We added this motivation in the introduction of the revised version.

Reviewer comment (Experiments):
In order for post-training quantization to be practical, activations are quantized too. How activation quantization can be jointly done is not addressed.

Author response:
[A] We refer the reviewer to Section A.8.2, where we perform full quantization (i.e., quantizing weights by RIQ and the activations by [Wu 20'](https://arxiv.org/pdf/2004.09602.pdf)). To gain speed on GPU, the de-quantized weights were packed in 8 bits, allowing acceleration of the inference. This way, one can attain a small memory footprint of RIQ and still benefit from full quantization speedup.

Reviewer comment (Evaluation):
Experimental results did not show a definitive advantage over competing methods.

Author response:
[A] Table 1 shows that RIQ outperforms many of the methods, achieving a small accuracy drop. Note that there are other methods that also attain small degradation from the baseline, thus, it is hard to show a definitive advantage in this setting. 

However, as mentioned in Section A.8.1, RIQ is an efficient method for reducing the model size, and it optimizes models quickly. For example, compressing the ResNet-50 takes less than a minute on a CPU, while most of the methods are more complex and require a longer time to optimize.

Reviewer comment (Writing):
I found it a bit hard to understand how $k$ come into picture during the first readthrough of the paper. I feel like a little more work needed to introduce it.

Author response:
We agree with the reviewer and the following text was added to introduce $k$:

> The connection of $\Delta_\ell$ to $\Vert w_\ell \Vert$ in Lemma1 hints at the rotation-invariant nature of the optimization. To focus on rotation-invariant solutions, RIQ introduces a search parameter $k$ that maintains proportion with $\Vert w_\ell \Vert$, allowing efficient search over these solutions.  Specifically, when $\Delta_\ell(k) = \Vert w_\ell \Vert/k$  where $k$ to be optimized, the bin-width...

Reviewer comment (Experiments):
I would like to understand how good is the search parameters wrt a synthetically created problem where (say weights sampled from Gaussian/Laplacian), single, layer and etc. How much the heuristics during the search may (or may not) miss the optimal bin width?

Author response:
RIQ performs constrained optimization of parameter $k$, where the constraint is the deviation between outputs. The heuristic allows tolerance from the optimal deviation which is specified by the smallest step.  

As suggested by the reviewer, we tested a synthetic problem, where RIQ quantizes a single convolution layer with a Gaussian input tensor to examine the searching points and the possible miss (See results in [this Figure](https://anonymous.4open.science/r/riq-3CE1/conv2d_deviation.pdf)). In this example, the deviation constraint between the outputs is 0.005 and RIQ reached very near with a deviation value of 0.00499 (i.e., missed by 1e-5).

Reviewer comment (Novelty):
While I agree that in general setting search of $k$ is unbounded (as you write in the paper), practically speaking it is not the case: the weights are finite, and thus there are only certain number of $delta$-s to check. This has in fact been done in the work called "Optimal quantization using scaled codebook" (btw, you cite this paper but attribute it as QAT, which is not correct).

Author response:
Indeed, practically there's a bounded search space in which the optimization can be done.  In the revised text we rephrase the text below eq. (7):

> Even though the search of $k$ is unbounded in general, practically it is sufficient to search in bounded space since the weights' norm is finite (Idelbayev et al. 2021).  

Thank you for pointing out the incorrect citation of this work. We fixed it in the revised version.

Reviewer comment (Evaluation):
I believe all compression results are given after ANS encoding; providing the compression ratio before ANS would be of great value (most papers report results before any additional encodings).

Author response:
Indeed, the results are after applying ANS encoding, and we agree that there is great value in seeing the contribution of ANS to RIQ alone, and hence, we provide the rate before and after ANS for the ResNet50 model in Figure 4(a), from which we can infer the contributions of ANS to Table 1. 

Figure 4(a) shows that the entropy limit is 2-3 bits/symbol away from RIQ only (depends on the deviation requirement). This difference stems from the different representations' efficiency. In other words, the RIQ representation minimizes the entropy without attaining it, whereas the ANS encoding attains the entropy limit.

Please note that while other baselines did not apply further encoding, most of the baselines have used per-channel quantization, for which encoding may be less beneficial due to the encoding tables overhead.