Paper: In defense of parameter sharing for model-compression (ICLR-2024)

Reviewer comment (Writing):
The writing of the paper lacks clarity and needs some major improvements. For example, the problem statement is vague --- “Is pruning the correct method to reduce model parameters?”. However, the authors do not define what are the “correct methods” for model compression. Are methods that exhibit better stability and Pareto-continuity considered as correct? Are there other correctness conditions?

Author response:
**"Is pruning the correct method to reduce model parameters?" What does correct mean?**

We are raising the question that, given an architecture and a parameter memory budget whether "pruning" is the best method to reduce parameters in the model to achieve the memory budget. We have reworded the sentence to read, "Is pruning the optimal method to reduce model parameters?", where optimality is measured in terms of model memory and quality trade-off.

Reviewer comment (Writing):
The paper uses unconvincing examples to support its arguments. For instance, the authors try to explain that pruning adversely affects model capacity, and use an example: "Consider pruning a n × d embedding table. If we prune beyond d×, we start getting degenerate zero embeddings." But what does "pruning beyond dx" even mean, and why is it not obvious that pruning reduces model capacity?

Author response:
**what does pruning beyond dx mean?**

Pruning beyond dx implies we are compressing more than a factor of $d\times$ ( where 2x means reducing the model to half the parameters ).

**why is not it obvious that pruning reduces model capacity**

The model capacity is reduced with any model compression technique. We argue that pruning affects the model capacity more adversely than other methods, such as RPS. The embedding table is a perfect example because compression with pruning leads to degenerate embeddings, which will not happen with RPS, even at very high compression. We have also added datasets in experiments on compressing embedding tables, corroborating this argument.

Reviewer comment (Experiments):
The paper is narrowly focused on comparing with pruning-based methods, which are only a subset of the model compression techniques. The paper does not justify why pruning methods are the most relevant baselines for the proposed method, which belongs to the parameter sharing paradigm.

Author response:
**why pruning baselines, why not other baselines such as low-rank, distillation, quantization, etc?**
In this paper, we talk about model compression of a given architecture at the start of training. Pruning is the most advanced and well-studied baseline in this setting.

1. Low-Rank: Low-rank methods for model compression are inferior to global-mask pruning. We have added a low-rank baseline to our new results on graphs to show this.

2. Distillation: In this setting, distillation is not the correct baseline as it needs first to train a big model and then use it to train smaller models.

3. Quantization: Quantization is popularly applied in model compression as either post-quantization or training-aware quantization - both of which are not compression-at-start techniques. To apply quantization at the start, we can train low-precision models, which can be used in conjunction with other techniques, such as pruning and RPS. Also, the compression range of quantization is very limited.

Reviewer comment (Experiments):
The paper ignores other important compression methods, such as quantization, distillation, and low rank factorization.

Author response:
**why pruning baselines, why not other baselines such as low-rank, distillation, quantization, etc?**
In this paper, we talk about model compression of a given architecture at the start of training. Pruning is the most advanced and well-studied baseline in this setting.

1. Low-Rank: Low-rank methods for model compression are inferior to global-mask pruning. We have added a low-rank baseline to our new results on graphs to show this.

2. Distillation: In this setting, distillation is not the correct baseline as it needs first to train a big model and then use it to train smaller models.

3. Quantization: Quantization is popularly applied in model compression as either post-quantization or training-aware quantization - both of which are not compression-at-start techniques. To apply quantization at the start, we can train low-precision models, which can be used in conjunction with other techniques, such as pruning and RPS. Also, the compression range of quantization is very limited.

Reviewer comment (Experiments):
The evaluation setups are weak. Evaluation is done on tiny datasets such as CIFAR10/100 and Tiny-ImageNet and very old architectures such as ResNet-20 and VGG-11 (10-year old), which raise questions on how well the observation from this work can be generalized to larger datasets, such as ImageNet, and more recent architectures, such as vision transformers.

Author response:
**Diversity of Datasets and models for evaluation**
1. We added more datasets from different domains : 
We understand the concern from reviewers about restricted domains of evaluation. To provide more diverse evidence, we have added two more domains of "Graphs with Attention-based Networks"  and "Deep learning Recommendation Models (DLRM) " in Appendix H. The DLRM models (~540M parameters) and the associated dataset are relatively large (40M samples). The observations are consistent in that RPS outperforms other methods. In fact, pruning is especially bad in DLRM models due to the pruning of embedding tables. Pruning of embedding tables at high compression (beyond d\times ) will give degenerate embeddings. 

2. Why not train vit on imagenet?
Creating a single data point by training a vit model on imagenet needs 300 GPU hours if using A100 GPUs. We do not have access to the kind of computing it would need to create Pareto curves for this experiment in a reasonable time.

Reviewer comment (Experiments):
How does STABLE-RPS compare with neural architecture search based methods? Can RPS still outperform search methods in different compression settings?

Author response:
**How does STABLE-RPS compare with neural architecture search-based methods? Can RPS still outperform search methods in different compression settings?**

At this stage, we cannot compare the two methods. NAS finds the best architecture given standard modules. STABLE-RPS and pruning methods find the reduced parameters for a given architecture. From works on pruning, it is clear that once a model is decided, it can be sparsified further and still keep matching accuracies in some cases. However, we still need to choose a good model to begin with, which is where NAS is useful. In the future, NAS can be combined with methods such as STABLE-RPS to obtain even better architectures. But, this direction is out of the scope of the current paper.

Reviewer comment (Theory):
Why does Theorem 3.2 involve cache line fetches? Cache line fetches are related to execution speed, not memory efficiency. The paper should explain how cache line fetches affect the memory consumption of the proposed method.

Author response:
**How does the proposed method affect latency? Why does Theorem 3.2 involve cache line fetches?**

 In theorem 3, we proved STABLE-RPS introduces minimal cache-inefficiency over ROAST mapping. We add latency numbers to the appendix section G. We measure the training time for different-sized matrices using different mappings in these experiments. We observe that STABLE-RPS mapping introduces a minimal increase in the training time of the model over ROAST.

Reviewer comment (Experiments):
Is STABLE-RPS compatible with parameter quantization (to achieve additional compression)?

Author response:
Yes Quantization can be used in conjunction with other methods including RPS and Pruning to obtain further compression.