Paper: Meta Compression: Learning to compress Deep Neural Networks (ICLR-2024)

Reviewer comment (Novelty):
The novelty of this paper is limited. The proposed method is a simple machine learning problem, which is easy to prove in a certain hypothesis.

Author response:
We address a fundamental question in this paper - can we predict the compression accuracy tradeoff of a novel pre-trained model without actually compressing the model? We formulate this problem and devise a framework based on meta-learning that solves this problem, building upon strong theoretical foundations. Furthermore, we demonstrate that diffusion models can be leveraged to generate additional evaluation data resembling test distribution in a meta-learning context.

Reviewer comment (Experiments):
In the actual scenarios, the performance of a certain compression method is highly related to multiple hyper-parameters. This paper does not consider these factors.

Author response:
We consider two key hyperparameters for compression methods - target sparsity level and target quantization level. Fine-tuning hyperparameters such as learning rate, number of epochs, etc. were kept fixed and decided based on the values used for pretraining.

Reviewer comment (Experiments):
The pruning and quantization are usually implemented serially in practice. However, the proposed method considers them separately and then merges them together using a simple criterion.

Author response:
All our experiments involve applying pruning followed by quantization serially. There is no merging criterion. For instance, if the optimal strategy is to use only pruning to 90% sparsity, the meta predictor will suggest pruning to 90% sparsity and quantizing to 32bit precision, which is equivalent to no quantization. In addition, we would like to highlight that the proposed framework can easily be extended to any joint pruning+quantization specification, for e.g. Slim pruning + LSQ quantization, as already discussed in the paper.

Reviewer comment (Experiments):
The quantization and pruning methods tested in this paper are out-of-date. Also, other compression categories like low-rank matrix and tensor decomposition are not considered.

Author response:
The choice of compression methods selected in our experiments is based on a practical constraint - the availability of stable implementations that can compress a wide array of DNN architectures. Keeping this in mind, we considered all compression methods that could reliably compress all considered architectures with available implementations in the popular open-source model compression library - Neural Network Intelligence (NNI). We will consider additional compression methods in future work.

Reviewer comment (Evaluation):
According to Fig. 7, the proposed method does not work well on large-scale datasets.

Author response:
We agree that the performance gap between meta compression and exhaustive search is slightly higher in the case of ImageNet setup experiment, but this is mainly because fewer data points were used in the ImageNet setup experiment due to limited available compute in academic settings. However, we would like to highlight here that even in this case, the performance is significantly better than any static recommendation strategy using a single pruning and quantization method across all constraints.

Reviewer comment (Evaluation):
The training cost is still very high, i.e., 8 days. I don't think the proposed framework can save time a lot as compared to existing SOTA compression methods.

Author response:
While the cost of training the meta-prediction model is comparatively high, the main benefit of using the proposed approach is that obtaining recommendations for a new deployment scenario, consisting of previously unseen pretrained architectures and compression level constraints, is extremely cheap (order of a few milliseconds). Following this, the model can be compressed using the recommended compression method specification. Meta compression outperforms network architecture search approaches here in terms of compute as the latter require significantly more compute for every new deployment scenario.

Reviewer comment (Writing):
It's advisable to include citations in the introduction to enhance the document's comprehensibility. The absence of citations, except in the initial paragraph, can pose challenges for readers in understanding the content. Incorporating citation references in the introduction will provide valuable context and background information.

Author response:
We thank the reviewer for their valuable suggestions. We will take them into account for improving the introduction and the illustration in Figure 2a. Please find below the answers to your questions.

Reviewer comment (Presentation):
I find that the overall presentation of the work can be somewhat challenging to follow. For instance, in the description of the inner loop of the meta-learning process in Figure 2a, there appears to be a discrepancy where Metadata extraction lacks information from the compressed classifier, which contradicts the description in the main context.

Author response:
We thank the reviewer for their valuable suggestions. We will take them into account for improving the introduction and the illustration in Figure 2a. Please find below the answers to your questions.

Reviewer comment (Reproducibility):
Could you please offer further elaboration on the "META FEATURES"? Additional details about the specific features used would be beneficial for a clearer understanding.

Author response:
“_Could you please offer further elaboration on the "META FEATURES"? Additional details about the specific features used would be beneficial for a clearer understanding._”  

Additional details about meta-features and their preprocessing have been provided in Appendix B.1.

Reviewer comment (Experiments):
One part which is not very clear is around the choice of the accuracy predictor and not very clear how the experiments look like with respect to other design choices around different compression strategies which would make it better to understand the generalizability of the solution. Because of this choice it limits the applicability of the results as they are compared against only one compression constraint.

Author response:
_Choice of accuracy predictor_

Two architectures were considered for the accuracy prediction task - multi layer perceptron (MLP) and gradient boosted decision trees (GBDT), with hyperparameters such as depth of MLP and number of decision trees selected using cross validation. GBDT outperformed MLP in our experiments, which is also consistent with the findings of [1] where multiple performance prediction architectures for DNNs are considered. 
[1] White et al, “How Powerful are Performance Predictors in Neural Architecture Search?”, NeurIPS’21

_Choice of compression stragies and generalizability_

The choice of compression methods selected in our experiments is based on a practical constraint - the availability of stable implementations that can compress a wide array of DNN architectures. We found the open source library “neural network intelligence (NNI)” to align the closest with our requirements and used the compression methods implemented there for our experiments. Some methods such as movement pruning had to be discarded as their fine tuning diverged for several architectures leading to unreliable data. In summary, our experiments so far suggest that the proposed meta compression approach works well across all compression algorithms that can reliably compress a variety of DNN architectures.  

“_Applicability of the results as they are compared against only one compression constraint._”

While we evaluate and provide results for two key constraints in our analysis - compression constraint and accuracy constraint, additional constraints can also be easily incorporated into the proposed framework. For instance, a constraint on FLOPs/MACs can be implemented by limiting the search space to compression methods that reduce FLOPs(eg. Channel pruning), and predicting the performance of several pretrained models after compression using channel pruning methods to a level that satisfies the constraint.

Reviewer comment (Experiments):
It is not clear how the research generalized with more recent modeling techniques specifically around transformers, which would help to strengthen the paper and understand the generalizability of the provided approach.

Author response:
“_It is not clear how the research generalized with more recent modeling techniques specifically around transformers, which would help to strengthen the paper and understand the generalizability of the provided approach._”

We fully agree that using more recent models and compression methods would be beneficial. However, many of these (including transformers) require significant computing resources and are tailored for specific architectures, making the evaluation very challenging. We will consider such an evaluation as future work.

Reviewer comment (Experiments):
How would the result and analysis hold against newer architectures specifically around transformers?

Author response:
“_It is not clear how the research generalized with more recent modeling techniques specifically around transformers, which would help to strengthen the paper and understand the generalizability of the provided approach._”

We fully agree that using more recent models and compression methods would be beneficial. However, many of these (including transformers) require significant computing resources and are tailored for specific architectures, making the evaluation very challenging. We will consider such an evaluation as future work.

Reviewer comment (Novelty):
It was hard to contextualize this work for me. For example, how does this work compare to [1], which uses multi-objective Bayesian optimization to yield an entire Pareto-frontier for architecture + compression algorithm hyperparameters?

Author response:
Thank you for suggesting additional references. [1] is closer to network architecture search (NAS) methods cited in our paper, where Bayesian optimization is used to navigate the search space of multiple architectures and pruning levels, in contrast to reinforcement learning which is commonly used in NAS papers. It also has the same key drawback as other NAS methods - that several search iterations are needed for any new deployment scenario that corresponds to a new accuracy, compression level constraint. Note that each step of this search involves training a DNN architecture and evaluating its performance which is quite expensive. We take a different approach which involves learning from multiple data points across DNN architectures, compression methods, and compression levels, which requires more compute in the beginning to generate these data points, but makes the compression method recommendation for any new deployment scenario very cheap (few milliseconds).

Reviewer comment (Novelty):
What is the novelty of this work? It is neither a new meta learning algorithm nor compression algorithm, as far as I can tell. To me, it seems the main novelty is the application of meta learning to model compression. In this case, I would need to see experimental evidence that meta learning for model compression is better than existing approaches using Bayesian optimization [1], DNAS, OFA.

Author response:
As correctly pointed out by the reviewer, the novelty of our approach lies in devising a framework based on meta-learning to provide recommendations tailored for specific accuracy-resource constraints. Another novel insight is the successful demonstration of the feasibility of using diffusion models to generate additional data that resembles test distribution in a meta-learning context. We agree with the reviewer that providing additional results for comparison against the stated references will be useful, and we will consider it in the future.

Reviewer comment (Writing):
I was confused what exactly is the static erm strategy?

Author response:
The static ERM strategy involves selecting the single best pruning and quantization method combination that yields the highest performance on average across several accuracy/compression level constraints.