Paper: Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR-2024)

Reviewer comment (Experiments):
More LLMs. Right now, it's only vicuna model (7B and 13B), having other architectures and perhaps bigger sizes (e.g. llama 70b) would add more evidence for the insights.

Author response:
Thank you for your suggestion and we strongly agree that extending our experiments beyond Vicuna will bolster our work. In that direction, we have included some additional experiments with LLaMa-2 as you suggested in Appendix A6 which obey our previous findings. For larger model size, we additionally provide some results for Vicuna-33B as follows which again aligns with our conclusion. We also have plans to run our experiments on llama-2 70B as pointed out by you, and include in our final draft (rebuttal duration doesn’t permit us to complete this experiment).

|Factoid-QA|Dense|10%|20%|30%|40%|50%|60%|
| ------------- |:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|
|Magnitude|76.92|76.72|71.73|59.44|46.75|42.46|3.10|
|SparseGPT|76.92|73.93|70.43|61.24|50.85|42.16|29.37|
|Wanda|76.92|73.63|72.73|65.03|50.65|52.05|39.36|

Reviewer comment (Evaluation):
Why do we see these performance degradations? Any way to explain why some methods work and some do not?

Author response:
Thank you for raising this question. We would like to bring attention to Junk DNA Hypotheis (JDH) recently proposed by https://arxiv.org/abs/2310.02277 . According to JDH, small-magnitude weights may appear "useless" for simple tasks and suitable for pruning (in most SoTA pruning methods), but actually encode crucial knowledge necessary for solving more difficult downstream tasks. Removing these seemingly insignificant weights can lead to irreversible knowledge forgetting and performance damage in difficult tasks. Their finding reveals  fresh insights into how LLMs encode knowledge in a task-sensitive manner and pruning them significantly hurt the performance. On the other hand, quantization doesn’t remove these small-magnitude weights completely, which can explain why quantization is more successful than pruning for LLMs. Another recent work, https://arxiv.org/abs/2307.02973 provides a detailed through analysis but wrt small-scale models like ResNet, EfficientNet, etc.

Reviewer comment (Experiments):
It would make the conclusions more robust and convincing if the evaluations use more than a single family of LLMs (i.e., Vicuna). Why not repeat these experiments with, e.g., Llama 2 and Falcon?

Author response:
Thank you for your suggestion and we strongly agree that extending our experiments beyond a single family of LLMs will bolster our work. In that direction, we have included some additional experiments with LLaMa-2 as you suggested in Appendix A6 (Factoid-QA, MCR-QA, ICR-QA) which obey our previous findings. For larger model size, we additionally provide some results for Vicuna-33B as follows which again aligns with our conclusion. We are running more experiments on larger LLMs which will be included in our final draft.

|Factoid-QA|Dense|10%|20%|30%|40%|50%|60%|
| ------------- |:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|
|Magnitude|76.92|76.72|71.73|59.44|46.75|42.46|3.10|
|SparseGPT|76.92|73.93|70.43|61.24|50.85|42.16|29.37|
|Wanda|76.92|73.63|72.73|65.03|50.65|52.05|39.36|

Reviewer comment (Experiments):
Regarding the observation that even 8-bit quantization has evident gaps with respect to uncompressed models, have the authors considered evaluating LLM.int8()? (https://arxiv.org/pdf/2208.07339.pdf)

Author response:
Thank you for your suggestion. As also pointed out by X7p4, we considered evaluating AWQ and LLM.int8() across our different task settings and we summarize our results on Vicuna-7B as in the following table. We observe that LLM.int8() despite its simplicity and ease-of-use, achieves better results than AWQ (8-bit), and GPTQ (8-bit) across all listed tasks. Note that we couldn’t complete our experiments on summarization and instruction following due to dependency and time constraints but we will update them in the final draft.

| | GPTQ | AWQ | LLM-int() | 
| ------------- |:-------------:|:-------------:|:-------------:|
|**Factoid-QA**|60.14%|60.31%|61.02%|
|**MCR-QA (MMLU)**|47.10%|47.18%|47.82%|
|**Retrieval Augmented QA**|75.55%|75.89%|75.91%|

Reviewer comment (Presentation):
It would help the reader to have a table summarizing all the tasks' performance over the different architectures, compression techniques and their resulting perplexity.

Author response:
Thank you again for your great suggestion and we agree that a table summarizing all task performance will be very helpful for the readers to quickly digest our experimental insights. We promise to include it in our final draft with an additional page.

Reviewer comment (Evaluation):
Does the authors have insights regarding why, in the paper's evaluation, quantization works better than pruning?

Author response:
To answer this question, we would like to bring attention to Junk DNA Hypothesis (JDH) recently proposed by https://arxiv.org/abs/2310.02277 . According to JDH, small-magnitude weights may appear "useless" for simple tasks and suitable for pruning (in most SoTA pruning methods), but actually encode crucial knowledge necessary for solving more difficult downstream tasks. Removing these seemingly insignificant weights can lead to irreversible knowledge forgetting and performance damage in difficult tasks. Their finding reveals fresh insights into how LLMs encode knowledge in a task-sensitive manner and pruning them significantly hurt the performance. On the other hand, quantization doesn’t remove these small-magnitude weights completely, which can explain why quantization is more friendly to pruning for LLMs. Another recent work, https://arxiv.org/abs/2307.02973 provides a detailed thorough analysis but wrt small-scale models like ResNet, EfficientNet, etc.

Reviewer comment (Experiments):
The authors say that SparseGPT is data-free. Is that true?

Author response:
Many thanks for pointing this out. We will correct it as data-free -> downstream data-free which means no downstream data is used to prune the models, rather only a small amount of calibration data is used for pruning.

Reviewer comment (Presentation):
Add a summarizing table to list dataset statistics.

Author response:
We also significantly value your suggestion to include the dataset statistics, and we will update our Appendix A5 to provide more details. We also have plans to release codes, experimental details, and host the datasets on Dropbox, which will help LLM compression researchers use LLM-KICK with ease for evaluating their novel algorithms.