Paper: Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity (ICLR-2024)

Reviewer comment (Theory):
The fourth “cross-task difficulty” definition is flawed. The authors assumed the larger the positive normalized performance gap between humans and models, the more difficult the task is for the model. However, if both human and LLM perform very poor (but “comparably poor”) on one task, it could mean this task is very difficult, yet in your setting the “relative” gap might not be significant. Besides, as the authors also pointed out, different tasks might have different metrics so directly normalizing and comparing across tasks can be problematic too.

Author response:
- We appreciate your insightful critique of the “cross-task difficulty”. Your observation rightly points out a potential limitation in our methodology. Specifically, in cases where both humans and LLMs exhibit poor performance, the relative gap might not accurately reflect the true difficulty of the task. Nevertheless, the task pairs used in our paper were carefully selected by us, such that our assessment of task difficulty aligns well with intuitions derived from previous studies [1,2,3]. These studies suggest that COLA is more challenging than SST-2, STS-B is more difficult than QQP, and CSQA is more challenging than WinoGrande, respectively. Therefore, despite the limitation of our “cross-task difficulty” definition, we believe that our main conclusion remains valid within this context. We will highlight the current scope and limitations of the definition in the manuscript. We hope our work can inspire and encourage more efforts to accurately measure the cross-task difficulty.

Reviewer comment (Novelty):
It was known before that difficult tasks are more fragile for pre-trained model pruning, such as in Sparsity-May-Cry (ICLR 2023). This paper essentially delves deeper on top of this existing observation.

Author response:
- While our study is indeed inspired by Sparsity-May-Cry (SMC), as rightly pointed by the reviewer, we delve deeper to study the role of small-magnitude weights in LLMs in addition to identifying the importance of the knowledge they encapsulate for downstream task performance. Specifically, compared with SMC, the unique contributions of the current draft can be summarized as follows:
  1. Provided task criticality definitions: SMC estimates task difficulty mainly based on their ad-hoc results, while we provide more concrete measures (two large categories with four small categories) to identify task difficulty. Especially, the within task category is able to disentangle task difficulty from task type, providing a more convincing evaluation.
  2. Exploring the effect of data volume on pruning.
  3. Wider range of task choices: Small-scale: RoBERTa on various tasks from GLUE; Large-scale: billion-level models i.e., Vicuna-7B on Open-book and Close-book QA.
  4. Exploring More sparsity patterns: we explored structured N:M sparsity.
  5. Understand the importance of knowledge of small magnitude weights in the context of LLM fine-tuning.

Reviewer comment (Novelty):
Most findings are not that surprising to me, for example, the finding that the small weights in LLMs can be important, or not, and that depends on the task. Nevertheless, this needs to be proved and this paper does it well.

Author response:
We are grateful that you think our paper does well on verifying the Junk DNA hypothesis. While the importance of small-magnitude weights depending on task difficulty may not be very surprising, no previous work has comprehensively studied this point, likely due to the daunting challenges of quantifying task difficulty. While our paper does not introduce any new pruning method, we propose a novel way to quantify task difficulty including both within task and across task. Based on our approach, we provide a comprehensive understanding of the role of small-magnitude weights in LLMs, inspiring the exploration of directions such as task-complexity-dependent dynamic inference and network self-slimmable properties. As rightly highlighted by the **Reviewer Q2er**, quoting his/her comment:
  
  > "One of the main highlights of this paper is the authors' proposal of a method to quantitatively define NLP downstream task difficulty. While this is in general highly ambiguous, the authors proposed (1) Varying the Adequacy of Target Domain Data; (2) Majority v.s. Minority in Multi-Lingual Translation, which essentially extends the first setting to multi-domain learning; (3) QA with v.s. without available external Information, and (4) for different task types, as the disparity in performance between humans and models, normalized by human performance. The definition will be broadly useful for understanding LLM (both full and compressed) performance in fine granularity."

Reviewer comment (Evaluation):
The paper could be improved if the error margins of the results were evaluated or included in the figures. If this might cause a huge additional computational burden (does it?), at least some statistical analysis of the significance of the results would help.

Author response:
Thank you for your valuable suggestion. We would like to clarify that the results presented in our submission are averages derived from three different random seeds. We chose not to include the error margin in the plot because the values represented are not original; they are after normalization based on the mean value of the dense transfer.

  Based on your concerns, we take RoBERTa-Large on QNLI task as an example and conducted **20 runs** instead of our previous 3 runs. We report our mean and standard deviation as follows:

  |                          | Dense Transfer   | 7:8           | 6:8           | 5:8           | 4:8           | 3:8           | 2:8           | 1:8          |
  |--------------------------|------------------|---------------|---------------|---------------|---------------|---------------|---------------|--------------|
  | Sparse to Dense Transfer | 0.91 ± 0.11      | 0.91 ± 0.11   | 0.85 ± 0.17   | 0.91 ± 0.11   | 0.87 ± 0.14   | 0.54 ± 0.1    | 0.51 ± 0.021  | 0.5 ± 0.0053 |
  | Sparse Transfer          | 0.91 ± 0.11      | 0.91 ± 0.11   | 0.91 ± 0.11   | 0.91 ± 0.11   | 0.85 ± 0.17   | 0.86 ± 0.11   | 0.8 ± 0.094   | 0.62 ± 0.035 |
  | Dense Transfer with Freezing | 0.91 ± 0.11 | 0.89 ± 0.15   | 0.95 ± 0.0021 | 0.94 ± 0.019  | 0.92 ± 0.11   | 0.92 ± 0.11   | 0.92 ± 0.11   | 0.91 ± 0.11  |

  To illustrate the performance drop by pruning from dense transfer, we normalized the average results against dense transfer, as shown in the following table:

  |                          | Dense Transfer | 7:8  | 6:8  | 5:8  | 4:8  | 3:8  | 2:8  | 1:8  |
  |--------------------------|----------------|------|------|------|------|------|------|------|
  | Sparse to Dense Transfer | 1.00           | 1.00 | 0.93 | 1.00 | 0.96 | 0.59 | 0.56 | 0.55 |
  | Sparse Transfer          | 1.00           | 1.00 | 1.00 | 1.00 | 0.93 | 0.95 | 0.88 | 0.68 |
  | Dense Transfer with Freezing | 1.00      | 0.98 | 1.04 | 1.03 | 1.01 | 1.01 | 1.01 | 1.00 |

  We observed a repeating pattern that is consistent with Figure 6 in our original submission. Specifically, the methods Dense Transfer and Dense Transfer with Freezing significantly outperform the other two methods. Notably, removing small-magnitude weights from pre-trained models leads to a marked decrease in performance.

Reviewer comment (Evaluation):
How do you explain the ups and downs in the figures? Specifically, for example in Fig.6a, the sparse-transfer 3:8 has better result than 4:8 in QNLI, and in Fig. 6b., sparse to dense transfer in CSQA 30% is higher than 20%, etc. Might such ups-and-downs indicate the variance of the results are high, and therefore the results are statistically insignificant?

Author response:
This could be attributed to two factors. First, pruning may introduce noise since performance doesn't always show a consistent, monotonic decrease with increased weight pruning, similar results are observed in [1,2].  Secondly, results averaged from limited random seeds might intensify this noise. Interestingly, a similar pattern of fluctuation persists in N:M QNLI sparse transfer even after 20 random seeds are run, specifically sparse transfer 3:8 still outperforms 4:8 in QNLI as shown in the above table. 

[1] Jaiswal A, Liu S, Chen T, et al. The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter. NeurIPS 2023.

[2] Liu S, Chen T, Zhang Z, et al. Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!. ICLR 2023.

Reviewer comment (Evaluation):
This paper does not provide another LLM pruning method. As stated above it is mainly considered as a strength (with its simplicity and great clarity). However, it remains uncertain how the magnitude-based pruning approach would yield practical application value because (1) this vanilla pruning technique leads to a rapid decline in performance, and (2) unstructured sparsity is impractical for GPU implementation.

Author response:
-  We thank the reviewer for identifying the major strength of our work. As already highlighted by the reviewer, our intent was not to demonstrate a new pruning method, but to present a detailed analysis of the importance of the low magnitude weights from a task-centric angle. For that, we view magnitude pruning as a *quantitative and easily controllable tool* for investigating and understanding the role of small-magnitude weights within pre-trained LLMs. Additionally, we consider magnitude pruning to be a widely used post-training pruning technique since it does not necessarily require more sophisticated proxies such as gradient or Hessian. Also, as the models grow bigger, even fine-tuning for few epochs may not be feasible for a downstream user to evaluate such proxies. We believe magnitude-based proxy has value due to its ease of adaptation without any additional fine-tuning or proxy storage cost.
-  Please note, we have demonstrated the Junk DNA hypothesis both for unstructured and N:M sparsity. It may often be argued that having inference speed up for unstructured sparsity is difficult without proper hardware or compiler support. However, N:M sparsity can yield inference speed up ([Accelerating Inference with Sparsity Using Ampere and TensorRT](https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt/)).

Reviewer comment (Evaluation):
The majority of experiments indicate that pruning performance, even for moderately challenging tasks, begins to drop at medium sparsity (around 30-50%). This raises doubts about the potential for any acceleration in LLM inference speed resulting from such pruning techniques.

Author response:
- Regarding the reviewer's concern on the potential of pruning for LLMs, we want to highlight that our goal is not to demonstrate a pruning method that can yield speedup. Rather, we take the first step to analyze the impact of pruning from a task-centric point of view. More importantly, we take the first step to answer a rather overlooked question: *whether the low-magnitude weights are truly inconsequential artifacts for large-scale models?* 
- Nevertheless, apart from the detailed analysis on the importance of low-magnitude weights, we believe our discoveries should inspire exploration into potential avenues, such as task-complexity-dependent dynamic inference and network self-slimmable properties. Specifically, we can dynamically assign a different amount of weights according to specific prior knowledge of task difficulty, and then we can implement more precise and hardware-friendly methods to dynamically prune models, thereby facilitating efficient inference.

Reviewer comment (Experiments):
Have the authors examined their study topic for quantization?

Author response:
- We appreciate the reviewer's suggestion. As you suggested, we explored the Junk DNA Hypothesis in the field of quantization. We use GPTQ [1] to quantize the weights and generate the results with different target quantizations Bit-width. Specifically, we examined with Setting 3: With vs. Without Available External Information. We use Vicuna-7B as the base model and report the results in the following table, the values before the slash ("/") represent Exact Match Ratios, while the values after the slash are Normalized Against Pre-Quantization Performance.

  | Configuration      | Closed Book (hard task)  | Open Book (easy task)  |
  |--------------------|----------------------------|---------------------------|
  | Before Quantization| 63.7 / 1.0                 | 76.1 / 1.0                |
  | 8-bit              | 62.8 / 0.98                | 75.55 / 0.99              |
  | 4-bit              | 54.35 / 0.85               | 72.2 / 0.95               |
  | 2-bit              | 13.5 / 0.21                | 18.65 / 0.25              |


    Our findings demonstrate a notable trend: as the bit-width decreases, there is a more rapid decline in performance after quantization for hard tasks. This trend also indicates the connection between the Junk DNA hypothesis and quantization.

    [1] Frantar E, Ashkboos S, Hoefler T, et al. Gptq: Accurate post-training quantization for generative pre-trained transformers[J]. arXiv preprint arXiv:2210.17323, 2022.