Paper: End-to-End Neural Network Compression via $\frac{\ell_1}{\ell_2}$  Regularized Latency Surrogates (ICLR-2024)

Reviewer comment (Novelty):
But, such method is not novel enough, and similar idea (optimize the mask) has been widely discussed.

Author response:
$\textbf{Limited Novelty}$:  We would like to emphasize that introducing masks in the objective is not our contribution. Our main contribution is to address the challenges that arise in optimization of the resulting objective. Below we highlight our key contributions

> Our first contribution is to design a novel $\ell_1/\ell_2$ regularizer for optimizing the FLOP regularized objective. This helps us effectively compress models with batchnorm and layernorms. This is in contrast to prior works which use $\ell_1$ based FLOPs regularizer and as a result fail to work well in the presence of layer/batchnorms. Please see  Figure 2 in our paper for a comparison of our technique with $\ell_1$ norm based methods. 

> The second contribution is to provide a simple algorithm for solving this objective. Past works have mostly relied on SGD/Adam to solve the FLOPs regularized objective. However, these algorithms are extremely slow to converge to sparse solutions and often require post processing steps. In our work, we overcome this problem by using proximal/projected techniques that are proven to converge to sparse solutions [1].

Reviewer comment (Reproducibility):
One question that arises is: what is the relationship between model compression and transfer learning? Do they occur in the same pipeline? For instance, is model compression done within the transfer learning process? Or is model compression performed first, followed by transfer learning to enhance performance?

Author response:
$\textbf{Transfer learning and compression}$: We perform transfer learning and compression at the same time in our experiments, and add the FLOPs regularizer term to the cross-entropy loss during transfer learning.

Reviewer comment (Experiments):
The authors claim that the proposed method can be used with pruning, low-rank factorization, quantization etc. However, the quantization is not verified on the experiments.

Author response:
$\textbf{Quantization}$: We would like to note that we do have quantization results in the appendix (please see Figure 5 in the appendix). Due to space constraints, we didn’t place these experiments in the main paper.

Reviewer comment (Experiments):
The performance on different devices can be various, so I wonder is the proposed method still effective on other device?

Author response:
$\textbf{On device performance}$: In Appendix B.1 we describe how to take the device into account for model compression using our framework.  We develop a L1/L2 latency surrogate which takes the actual on-device latency into account. This helps directly optimize for on-device latency. In figure 6, the left plot presents the latency vs accuracy curves obtained using this technique for mobilenetv3 compression. The results show that working with latencies indeed helps us find models with better latencies than FLOPs regulrizer.

Reviewer comment (Novelty):
The main concern is the limited novelty of the work. The idea of making the FLOPs constraint differentiable is commonly used in many pruning works, such as using auxiliary masks. The positivity constraints are crucial for pruning models. However, as mentioned in the work, similar ideas have been studied in existing works.

Author response:
$\textbf{Limited Novelty}$:  We would like to emphasize that introducing masks in the objective is not our contribution. Our main contribution is to address the challenges that arise in optimization of the resulting objective. Below we highlight our key contributions

Our first contribution is to design a novel $\frac{\ell_1}{\ell_2}$ regularizer for optimizing the FLOP regularized objective. This helps us effectively compress models with batchnorm and layernorms. This is in contrast to prior works which use $\ell_1$ based FLOPs regularizer and as a result fail to work well in the presence of layer/batchnorms. Please see  Figure 2 in our paper for a comparison of our technique with $\ell_1$ norm based methods. We also compare against such techniques on a larger scale in figure 3(c), and empirically demonstrate our technique's better performance. 

The second contribution is to provide a simple algorithm for solving this objective. Past works have mostly relied on SGD/Adam to solve the FLOPs regularized objective. However, these algorithms are extremely slow to converge to sparse solutions and often require post processing steps, which include extra hyper-parameters for thresholding the sparse values.This limits their ease-of-use for practical deployment. In our work, we overcome this problem by using proximal/projected techniques that are proven to converge to sparse solutions [1].

Reviewer comment (Experiments):
It is recommended to evaluate the performance of the proposed method multiple times, as it is claimed to be more stable than other methods.

Author response:
$\textbf{Multiple evaluations of our method}$: We would like to note that the experiments we performed on BERT, MobilenetV3, EfficientNet are large scale and it is computationally expensive to repeat these experiments multiple times. That being said, we note that we tried our technique on diverse tasks (please refer to experiments section in the paper) and noticed stable performance/loss curves across all the tasks.

Reviewer comment (Experiments):
Direct comparison with advanced compression methods is also encouraged, and it would be better to show the training cost comparison since the proposed method is claimed to be fast.

Author response:
$\textbf{Comparison with advanced compression methods}$: We compare with state of the art compression methods on language and vision tasks, including recent works for compressing BERT, and compressing ResNet using pruning and NAS techniques. We also demonstrate that our method is much faster than state of the art NAS methods on mobileNet compression. In particular, we would like to highlight our results in figure 1, which compares our MobileNet pruning method against against TuNAS, an industry standard NAS algorithm which is widely deployed. We emphasize that MobileNetv3 and EfficientNet are highly optimized families of models, and our method can further compress them. We also compare against multiple recent state-of-the-art pruning and NAS works for ResNets in figure 3(a), and report much better performance. Finally, we also compare against recent state-of-the-art compression methods for BERT including [1] fine-tuning in figure 1, and report a better accuracy-FLOPs trade-off.

[1] - Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami.
A fast post-training pruning framework for transformers. arXiv preprint arXiv:2204.09656, 2022.