Paper: Structural Pruning of Large Language Models via Neural Architecture Search (ICLR-2024)

Reviewer comment (Experiments):
**No experiments on LLM, but the topic of this paper is about LLM**. The title of the paper suggests a focus on Large Language Models, leading me to expect analyses or experiments involving LLMs like LLaMA, or at least T5-large, especially since the terms 'LLM' are predominantly used in the paper rather than 'language model' or 'pre-trained language model'. However, upon delving into the experimental section, it's surprising to find that the actual experiments exclusively involve **BERT**. There is no mention of LLMs in the experiment, nor is there any comparison or discussion on how the application of Neural Architecture Search might differ between LLMs and PLMs.

Author response:
To address their concerns, we updated our paper and replaced the term ‘large language model’ by ‘pre-trained language model’. Even though LLM with prompt-classification are increasingly popular, fine-tuning models such as BERT is still highly competitive on labeled datasets and often used in practice. We see the extension of our approach to LLM as a promising future work.

Reviewer comment (Experiments):
**More Baselines are needed**. The authors appear to have selected some baselines that may be relatively less challenging to outperform, such as Retraining-Free Pruning and self-defined baselines. Retraining-Free Pruning, focusing on rapid compression of BERT without retraining, isn't directly comparable with the methods in this paper, which involve around 12 hours of training (as indicated by 50,000 seconds in Figure 5) on the MNLI dataset. Moreover, the proposed method underperforms DistillBERT in more than half of the datasets (including MRPC, COLA, SST2, QNLI, MNLI). Other baseline methods are self-created, and there's a notable omission of any recent advancements in structural pruning methods for BERT in the past five years (e.g., DynaBERT, CoFi). Comparison with those methods are needed. Additionally, the related work section only references four papers on structural pruning for BERT, whereas the authors could have expanded their literature scope by referring to a broader survey[1] on this topic.

Author response:
We tried to benchmark against COFI using the official implementation but obtained inconclusive results compared to the original paper. We decided not to include these results yet to avoid an unfair comparison that would put COFI at a disadvantage. We reached out to the authors to understand this in detail and ensure a fair comparison;  if possible, will included the results of experiments for the camera ready version. However, we would like to stress that including additional structural pruning methods would not change the main argument of the paper: NAS allows for multi-objective optimization rather than constrained optimization to obtain the Pareto set of sub-networks. This captures the non-linear relationship between performance and compression ratio. For example, Figure 5 right shows that pruning the model on the MNLI by 50% leads to the same performance as pruning by 60%. Since we do not know this before, we either have to run the constraint optimization process (e.g. COFI or RFP) multiple times based on a grid of threshold values or stick to a sub-optimal solution based on an arbitrarily defined threshold. Multi-objective NAS on the other hand allows us to select the model without any additional overhead.

Reviewer comment (Reproducibility):
Could you clarify the process used to calculate the runtime for each technique as shown in Figure 5? The RFP (Retraining-Free Pruning) paper mentions that pruning takes merely 0.01 hour, approximately 36 seconds. However, in your study, this duration extends to about 20,000 seconds. What factors contribute to this substantial discrepancy in runtime measurements?

Author response:
Question regarding runtime: For all methods we compute the total runtime to obtain a Pareto set of solutions. This includes also fine-tuning the original model for RFP and running it multiple times with different thresholds.