Paper: LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning (ICLR-2024)

Reviewer comment (Writing):
This paper can benefit from better writing and presentation.

Author response:
Thanks for your suggestion. We have carefully checked and modified the equations and presentation, and reflected these changes in the revised paper.

Reviewer comment (Experiments):
The authors claim that the proposed method approximates the importance of the weights. Could you present any supporting experimental results on how accurate LoRA-guided importance approximations are? And, following up on this, does better approximation yield better performance eventually?

Author response:
A2. We have already provided the results and discussions in "how accurate LoRA-guided importance approximations are" in the alation study. We set the vanilla gradient-guided criterion in Eq. (3) as our baseline, since our criterion is an efficient approximation of vanilla criterion. As shown in Figure 4, the similarity between these two criteria is very high with Ratio = 10%, which indicates that our LoRA-guided criterion can accurately estimate the importance in an iterative way.

Reviewer comment (Reproducibility):
It seems ‘\hat{I}’ has the same dimension as the model weights W? Then, in terms of memory usage (at least if we strictly follow the algorithm 1), is the same as computing the gradients w.r.t. W?

Author response:
A4. Yes, $\hat{I}$ has the same dimension as the model weights $W \in R^{d\times k}$. However, we do not need to save $\hat{I}$ for all layers in each iteration. We can calculate $\hat{I}$ group by group. For example, we simultaneously calculate importance of Query, Key and Value in $g$-th Attention layer. Subsequently, we accumulate group importance $\mathcal{\hat{G}}_g \in R^{1\times heads}$  according to Eq. (4) and only  $\mathcal{\hat{G}}_g$  needs to be saved, where $heads << d\times k$.

Reviewer comment (Evaluation):
It would be helpful for authors to provide ‘theoretical FLOPs’ compared to dL/dW-based importance criterion.

Author response:
A5. We provide the theoretical FLOPs between LoRA-guided criterion and vanilla gradient-guided (dL/dW-based) criterion in the following table. The FLOPs computation rule can be referred to [1]. The table shows LoRA-guided criterion has lower theoretical FLOPs than Vanilla, as it does not compute pretrained weight gradients. We add this comparison in Table 8 in the revised paper.  

| Method      | FLOPs (G) | 
|-------------|-----------|
| LoRA-guided | 12881     |
| Vanilla     | 20298     |

[1] Scaling laws for neural language models. Arxiv 2020

Reviewer comment (Theory):
This paper lacks the motivation why the LoRA can considered as the guidance. Instead, the authors just directly show it can estimate the importance of each parameter. It is encouraged to illustrate why this criterion is created. More explanations and analysis are needed.

Author response:
A1. As explained in the introduction, our primary motivation for employing the LoRA-guided criterion is to reduce computational and memory demands during iterative pruning. As depicted in Figure 1(b), the conventional gradient-guided criterion depends on the gradients of tpre-trained weights, necessitating the computation and storage of gradients with a shape of $d\times k$. In stark contrast, the LoRA-guided criterion requires only the gradients of matrices $A$ and $B$  with shapes of $r\times k$ and $d\times r$, respectively, where $r$ is significantly smaller than $d$ and $k$ (often by a factor of a thousand, as in the case of LLaMA-7B with $d=4096$ and $r=8$). This substantial reduction in size translates to a corresponding reduction in computational load and memory usage. The efficacy of the LoRA-guided criterion, as demonstrated by our experimental results in Figure 4, is its capability to accurately gauge the importance of each weight group, mirroring the accuracy of the traditional gradient-guided approach but with much greater efficiency.

We have incorporated the analysis into the METHOD section in the revised paper.

Reviewer comment (Evaluation):
Upon examining Table 2, several concerns arise regarding the experimental results. Notably, when juxtaposed with the PPL of LLaMA-7B, there's a marked degradation in PPL. Even at a relatively modest 20% pruning rate, the PPL increase is evident. One has to question if there are scenarios where such a pronounced PPL drop would be deemed acceptable.

Author response:
A1. We thoroughly address this concern in General Response Q1. It is critical to highlight that our method significantly outperforms both LLM-Pruner and WANDA at token lengths of 2048 and 128, as shown in Table 2 and Table 7, respectively. In scenarios where a pronounced PPL drop is observed, it is often possible to mitigate this through the use of additional data to refine the pruning results.

Reviewer comment (Evaluation):
The 50% pruning rate appears to present substantial challenges. What is the overarching conclusion here? Is the implication that a 50% pruning rate might be overly ambitious? Alternatively, do the authors consider the observed performance drop at this rate to be inconsequential?

Author response:
A3.In the post-training structured pruning context, especially when limited calibration data is available, a 50% pruning rate indeed presents substantial challenges. However, we significantly outperform other structured pruning methods. To mitigate the PPL degradation in 50% ratio, we also iteratively prune LLMs on large-scale dataset. The details can be found in General Response Q1.

Reviewer comment (Evaluation):
For a more comprehensive understanding, it would be beneficial to have the PPL and other metrics as reported by contemporaneous studies.

Author response:
A4. We have duly noted the request for comprehensive metrics. In response, Table 2 has reported PPL and accuracy metrics for zero-shot task classification on the common sense reasoning dataset. To further validate the efficacy of LoRAPrune, we have supplemented these with results from the MMLU (5-shot) evaluations, offering a broader performance perspective consistent with contemporaneous studies. We include these results in Table 5 in the revised paper.

| Method               | MMLU(5-shot) | Ratio | 
|----------------------|--------------|-------|
| LLaMA-7b             | 36.81        | 0%    | 
| LLM-Pruner           | 28.34        | 20%   |
| __LoRAPrune (Ours)__ | 29.56        | 20%   |
| LLM-Pruner           | 26.12        | 50%   |
| __LoRAPrune (Ours)__ | 28.03        | 50%   |

Reviewer comment (Novelty):
To the best of my understanding, the novelty of this paper is limited. Specifically, there are some existing explorations on identifying the dependency during structural pruning to maximally preserve the performance after pruning, such as LLM-Pruner. In this paper, the key difference is that the authors propose to shift the computation of dependency from backbone weight in LLMs to LoRA adapters in LLMs. The author may want to further address the novelty here.

Author response:
A1. We propose the innovative LoRA-guided pruning criteria, which is informed by low-rank adaptation (LoRA) rather than relying on the gradients of pre-trained weights. This efficient criteria enables us to structurally prune 65B LLMs in a single GPU, improving the applicability of pruning on LLMs. We discuss the differences between LoRAPrune and LLM-Pruner in General Response Q2. Reviewers nuRr and RcVc agree with our novelty.

Reviewer comment (Novelty):
In Table 1, the authors claim that LLM-Pruner does not support tuning. However, as LLM-Pruner also uses a structural pruning technique, it is also compatible with LoRA adapters, which is indicated in the abstract and experiment sections in LLM-Pruner. The authors may want to further justify their claim.

Author response:
A2. We apologize for any confusion caused. To clarify, LLM-Pruner supports tuning with LoRA but does not support iterative pruning efficiently. Specifically, LLM-Pruner uses one-shot pruning and then fine-tunes the pruned models by LoRA. The reasons for LLM-Pruner's inefficiency with iterative pruning are discussed in our General Response Q2. The term "Fine-tune" as referenced in Table 1 pertains to the process of iterative pruning which involves simultaneous pruning and fine-tuning to regain performance. To clarify this process and avoid further misunderstanding, we have updated the terminology in Table 1 from "Fine-tune" to "Iterative Pruning" in our revised submission.

Reviewer comment (Experiments):
In Figure 3, the author missed an important baseline, LLM-Pruner. It would help the authors to better understand the performance of the proposed LoRAPruner by adding the LLM-Pruner baseline in the figure.

Author response:
A3. We acknowledge the significance of including LLM-Pruner as a baseline for larger models. Due to the limited timeframe of the rebuttal period, we provide results for the 50% pruning ratio. These results are detailed in the table below.

| Model     |Method | PPL   | Ratio | 
|-----------|-------|-------|----------------------|
| LLaMA-13B |  LLM-Pruner           |11.88 | 50%   |
|           |  __LoRAPrune (Ours)__ |9.84  | 50%   |
| LLaMA-30B | LLM-Pruner           |8.69  | 50%   | 
|           | __LoRAPrune (Ours)__ |7.27  | 50%   | 

The preliminary results, as shown in above table, indicate that LoRAPrune outperforms LLM-Pruner in both the LLaMA-13B and LLaMA-30B models at a 50% pruning ratio. This reinforces the scalability and effectiveness of our proposed LoRAPrune. We include these results in Figure 3 in the revised paper.