Paper: Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICLR-2024)

Reviewer comment (Experiments):
The analysis of this paper on non-uniform sparsity is mostly focused on the whole Transformer block. However, less insights are provided as to which type of layer should be pruned more.

Author response:
**Q1:** Less insights are provided as to which type of layer should be pruned more.


- We thank you for raising this great question. As you suggested, we explored OWL in a per-layer basis. The comparison of per-layer and per-Transformer block is reported in the following tables. We report the sparsity ratio of 7 FC layers including q\_proj, k\_proj, v\_proj, o\_proj, gate\_proj, down\_proj, and up\_proj of layer 1, 2, 15, 30, and 31.


    *Table: OWL per-layer. 70% Sparsity, 86.285 Perplexity*

  | Layer | q\_proj | k\_proj | v\_proj | o\_proj | gate\_proj | down\_proj | up\_proj |
  |-------|---------|---------|---------|---------|------------|------------|----------|
  | 1     | 0.639   | 0.638   | **0.691** | 0.598   | **0.710**   | 0.696      | **0.713** |
  | 2     | 0.680   | 0.677   | **0.707** | 0.679   | **0.711**   | 0.703      | **0.713** |
  | 15    | 0.698   | 0.693   | **0.710** | 0.706   | **0.709**   | 0.703      | **0.712** |
  | 30    | 0.705   | 0.704   | **0.713** | 0.663   | **0.710**   | 0.657      | **0.712** |
  | 31    | 0.702   | 0.701   | **0.712** | 0.670   | **0.710**   | 0.621      | **0.711** |

  *Table: OWL per-Transformer block. 70% Sparsity, 24.55 Perplexity*

  | Layer | q\_proj | k\_proj | v\_proj | o\_proj | gate\_proj | down\_proj | up\_proj |
  |-------|---------|---------|---------|---------|------------|------------|----------|
  | 1     | 0.613   | 0.613   | 0.613   | 0.613   | 0.613      | 0.613      | 0.613    |
  | 2     | 0.641   | 0.641   | 0.641   | 0.641   | 0.641      | 0.641      | 0.641    |
  | 5     | 0.608   | 0.608   | 0.608   | 0.608   | 0.608      | 0.608      | 0.608    |
  | 30    | 0.760   | 0.760   | 0.760   | 0.760   | 0.760      | 0.760      | 0.760    |
  | 31    | 0.721   | 0.721   | 0.721   | 0.721   | 0.721      | 0.721      | 0.721    |


- We found that applying OWL in a per-layer manner leads to sub-optimal performance (Perplexity: 86.285 vs 24.55). We can see that applying OWL in per-layer manner will lead to uniform sparsity of certain layers across Transformer blocks (marked in bold), such as v_proj, gate_proj and up_proj, which might be undesirable for LLM pruning.

Reviewer comment (Reproducibility):
Unstructured sparsity may be less practical as it needs special hardware for acceleration.

Author response:
**Q2:** Unstructured sparsity may be less practical as it needs special hardware for acceleration.

- Thank you for raising this great question! In order to examine if the promise of our non-uniform layerwise sparsity holds in hardware-friendly regimes, we explore OWL in two hardware-friendly regimes, including:

  - **(1) N:M sparsity**: Following DominoSearch [1], we opt for mixed N:8 sparsity configuration. Instead of employing a uniform N value across all layers, we allow individual layers to possess distinct N values while maintaining the **same** parameter count. We adopt OWL to determine the optimal value of N for individual layers.
 
    *Table: Perplexity of Mixed N:M Sparsity (N refers to non-zero weights).*

    | Method         | Layerwise Sparsity | Structure | Perplexity |
    |----------------|--------------------|-----------|------------|
    | Wanda          | Uniform            | 4:8       | 8.57       |
    | Wanda          | OWL                | Mixed 4:8 | **8.55**   |
    | Wanda          | Uniform            | 3:8       | 42.56      |
    | Wanda          | OWL                | Mixed 3:8 | **21.49**  |
    | Wanda          | Uniform            | 2:8       | 2962.00    |
    | Wanda          | OWL                | Mixed 2:8 | **331.37** |

  - **(2) Structured pruning**: Instead of pruning weights, we follow the recent methodology introduced in LLM Pruner [2], wherein entire neurons and attention heads are removed. This action facilitates direct acceleration of pruned LLMs on GPUs or TPUs. We replace the uniform layerwise sparsity used by LLM pruner to a non-uniform layerwise sparsity using OWL.
  
      *Table: Perplexity of Structure Pruning with LLaMA-7B on WikiText2 and PTB.*

    | Dataset        | Pruning Method     | Layerwise Sparsity | 20%    | 40%    | 60%    | 80%    |
    |----------------|--------------------|--------------------|--------|--------|--------|--------|
    | Wikitext2      | LLM Pruner         | Uniform            | 19.09  | 30.39  | 90.017 | 1228.17|
    | Wikitext2      | LLM Pruner         | OWL                | **18.57** | **28.65** | **76.99** | **321.64** |
    | PTB            | LLM Pruner         | Uniform            | 29.51  | 66.90  | 192.06 | 1691.87|
    | PTB            | LLM Pruner         | OWL                | **28.82** | **53.22** | **150.16** | **502.07** |

- The ensuing results presented above corroborate the extension of OWL's advantages into more hardware-friendly regimes. Across diverse practical regimes, such as N:M sparsity and structured pruning, OWL consistently yields substantial performance enhancements over its uniform baseline. Notably, in scenarios like the 3:8 and 2:8 structured sparsity regimes, OWL demonstrates its efficacy by achieving perplexity reductions of 2x and 8x, respectively, surpassing the uniform baseline. And the OWL can also be applied to structured pruning to enhance the performance of the state-of-the-art LLM Pruner. We have updated the above results in Appendix A of our revision.

Reviewer comment (Experiments):
It would be good to discuss how the approach could be applied to non-uniform sparsity per FC layer rather than the entire Transformer block. In this way, we could understand better which type of layers in Transformer should be pruned more.

Author response:
**Q1:** Less insights are provided as to which type of layer should be pruned more.


- We thank you for raising this great question. As you suggested, we explored OWL in a per-layer basis. The comparison of per-layer and per-Transformer block is reported in the following tables. We report the sparsity ratio of 7 FC layers including q\_proj, k\_proj, v\_proj, o\_proj, gate\_proj, down\_proj, and up\_proj of layer 1, 2, 15, 30, and 31.


    *Table: OWL per-layer. 70% Sparsity, 86.285 Perplexity*

  | Layer | q\_proj | k\_proj | v\_proj | o\_proj | gate\_proj | down\_proj | up\_proj |
  |-------|---------|---------|---------|---------|------------|------------|----------|
  | 1     | 0.639   | 0.638   | **0.691** | 0.598   | **0.710**   | 0.696      | **0.713** |
  | 2     | 0.680   | 0.677   | **0.707** | 0.679   | **0.711**   | 0.703      | **0.713** |
  | 15    | 0.698   | 0.693   | **0.710** | 0.706   | **0.709**   | 0.703      | **0.712** |
  | 30    | 0.705   | 0.704   | **0.713** | 0.663   | **0.710**   | 0.657      | **0.712** |
  | 31    | 0.702   | 0.701   | **0.712** | 0.670   | **0.710**   | 0.621      | **0.711** |

  *Table: OWL per-Transformer block. 70% Sparsity, 24.55 Perplexity*

  | Layer | q\_proj | k\_proj | v\_proj | o\_proj | gate\_proj | down\_proj | up\_proj |
  |-------|---------|---------|---------|---------|------------|------------|----------|
  | 1     | 0.613   | 0.613   | 0.613   | 0.613   | 0.613      | 0.613      | 0.613    |
  | 2     | 0.641   | 0.641   | 0.641   | 0.641   | 0.641      | 0.641      | 0.641    |
  | 5     | 0.608   | 0.608   | 0.608   | 0.608   | 0.608      | 0.608      | 0.608    |
  | 30    | 0.760   | 0.760   | 0.760   | 0.760   | 0.760      | 0.760      | 0.760    |
  | 31    | 0.721   | 0.721   | 0.721   | 0.721   | 0.721      | 0.721      | 0.721    |


- We found that applying OWL in a per-layer manner leads to sub-optimal performance (Perplexity: 86.285 vs 24.55). We can see that applying OWL in per-layer manner will lead to uniform sparsity of certain layers across Transformer blocks (marked in bold), such as v_proj, gate_proj and up_proj, which might be undesirable for LLM pruning.

Reviewer comment (Experiments):
While it is not the focus of this work, I am interesting as to how would OWL perform on other networks, e.g. ConvNet or Vision Transformer. Adding such analysis would be beneficial as to understand the effect of outliers on pruning LLMs.

Author response:
**Q4:** While it is not the focus of this work, I am interested in how OWL would perform on other networks, e.g., ConvNet or Vision Transformer. Adding such analysis would be beneficial to understand the effect of outliers on pruning LLMs.

- Thanks for your question. As you requested, we have applied OWL to ConvNeXt-Base and DeiT-Base and evaluated on ImageNet-1K. The pruning metric is Wanda and we compare OWL with uniform layerwise sparsity.

- Our findings reveal that OWL enhances the accuracy of sparse DeiT models in contrast to Wanda. However, for ConvNeXt models, it seems that OWL does not necessarily bring benefits to ConvNeXt (neither increase nor degrade accuracy). Overall, it seems that the performance improvement of OWL on vision models is not as pronounced as observed in Language Model Models (LLMs).

- Our hypothesis is that the phenomenon of outliers is not particularly evident in vision models. According to [3], outliers in LLMs are causally related to high-frequency tokens in pre-training data. These high-frequency tokens are more prevalent in textual datasets but are relatively scarce and challenging to identify within vision datasets. Hence, the phenomenon of outliers, crucial in OWL's effectiveness, may not be as evidently present or impactful within the domain of vision models, contributing to the differing performance improvements between LLMs and vision models.

  *Table: Accuracy of Sparse ConvNeXt-Base on ImageNet-1K.*

  | Sparsity | 50%   | 60%   | 70%   | 80%   |
  |----------|-------|-------|-------|-------|
  | Wanda    | 82.72 | 80.55 | 68.18 | 6.44  |
  | OWL + Wanda | 82.76 | 80.53 | 68.28 | 6.32  |

  *Table: Accuracy of Sparse DeiT-Base on ImageNet-1K.*
  
  | Sparsity | 50%   | 60%   | 70%   | 80%   |
  |----------|-------|-------|-------|-------|
  | Wanda    | 78.23 | 71.14 | 49.20 | 6.86  |
  | OWL + Wanda | 78.40 | 71.76 | 51.24 | 7.98  |

Reviewer comment (Experiments):
While structured sparsity enforces a fixed sparsity ratio 50%, i think OWL could be potentially applied for this type of sparsity. Instead of a dynamic sparsity ratio, we could choose to either prune a FC layer with 2:4/4:8 sparsity or skip this FC layer entirely. I think such analysis would make this work more comprehensive.

Author response:
**Q5:** While structured sparsity enforces a fixed sparsity ratio 50%, I think OWL could be potentially applied for this type of sparsity. Instead of a dynamic sparsity ratio, we could choose to either prune a FC layer with 2:4/4:8 sparsity or skip this FC layer entirely. I think such analysis would make this work more comprehensive.

- Thanks for providing such an excellent idea. Indeed, OWL has the potential to enhance 2:4 sparsity. We follow your idea and conduct experiments to evaluate it. Specifically, we use OWL to determine which layers should be skipped as dense and setting the rest layers to 2:4 sparsity. We report the results in the following table. Clearly, OWL provides a good indicator to choose which layer to skip than other baselines such as random skipping and L1 norm of weight.

    *Table: 2:4 Structured Sparsity*

  | Skipping metric | no skip | 2       | 5       | 10     | 15     |
  |-----------------|---------|---------|---------|--------|--------|
  | Random          | 11.53   | 11.521  | 10.403  | 9.196  | 8.335  |
  | L1 norm         | 11.53   | 11.823  | 11.171  | 10.425 | 9.261  |
  | OWL             | 11.53   | **9.559** | **8.599** | **7.79** | **7.049** |
 
    *Table: 4:8 Structured Sparsity*

  | Skipping metric | no skip | 2       | 5       | 10     | 15     |
  |-----------------|---------|---------|---------|--------|--------|
  | Random          | 8.566   | 8.668   | 8.25    | 7.672  | 7.187  |
  | L1 norm         | 8.566   | 8.709   | 8.358   | 7.98   | 7.453  |
  | OWL             | 8.566   | **7.922** | **7.488** | **7.091** | **6.653** |

Reviewer comment (Writing):
Some essential concepts are confusingly defined in the text: What exactly is $D_i$ (in LOD)? I presume that it is the ratio of "outlier-ish" weights, but how can it be over 1 in Figure 1 left?

Author response:
**Q1: Clarity (Medium).** Some essential concepts are confusingly defined in the text: What exactly is (in LOD)? I presume that it is the ratio of "outlier-ish" weights, but how can it be over 1 in Figure 1 left?

- In the context of LOD, $D_i$ specifically denotes the outlier ratio within layer $i$. Regarding Figure 1, we want to clarify that the values depicted are indeed represented as percentages, as explicitly stated in the legend: 'Figure 1: Layerwise Outlier Distribution (LOD) (**%**) of dense LLaMA-7B, 13B, and 30B.' We appreciate the opportunity to offer this clarification.

Reviewer comment (Theory):
The overall motivation of the proposed algorithm seems to come from the so-called "empirical studies" in section 3.2., which attempt to connect the notion of LOD (layerwise outlier distribution) with the layerwise sparsity. It is very difficult for me, however, to follow how the conclusions they make can be logically deduced from their observations (it seems to be circular logic).

Author response:
**Empirical Study 1.** 

- The Wanda metric is essentially the  product  of the sum of input features and weight magnitude. Since the magnitude of feature outliers in LLMs is significantly larger than the others (more than 100x larger) while the magnitude of weights  does not exhibit such outliers [1], "outliers in terms of Wanda score'' are fairly the weighs that are connected to those outlier features, thus playing an crucial role to preserve outliers. 

- Moreover, the consistent performance enhancements observed in LOD across a wide range of domains including unstructured pruning, N:M sparsity, structured pruning, and mixed-precision quantization further solidify the importance of LOD in LLMs.

Reviewer comment (Theory):
In Empirical Study 1, the empirical observation is that LOD is non-uniform on dense Llamas. From this fact, the paper jumps to the conclusion that "employing uniform pruning across all layers would inevitably disrupt the outlier structure in layers." I find this point very difficult to follow, unless the authors have a way to prove that the LOD itself, for some arbitrary $M$, is an informative criterion for pruning. But is it really so? Why is the ratio between the weight and the layerwise average important, from the first place? Perhaps the authors are resorting to some circular logic.

Author response:
**Q2: Empirical Study 1.** The empirical observation is that LOD is non-uniform on dense Llamas. From this fact, the paper jumps to the conclusion that "employing uniform pruning across all layers would inevitably disrupt the outlier structure in layers." I find this point very difficult to follow, unless the authors have a way to prove that the LOD itself, for some arbitrary , is an informative criterion for pruning. But is it really so? Why is the ratio between the weight and the layerwise average important, from the first place? Perhaps the authors are resorting to some circular logic.

- Thanks for your comments. We would like to clarify that LOD is indeed a very informative criterion for pruning. LOD measures the outlier ratio of elements in metric $\mathbf{A} =\parallel \mathbf{X} \parallel _{2}\cdot |\mathbf{W}|$whose values are much larger than the averaged value. $\mathbf{A}$ essentially serves as the pruning metric used by Wanda [1], which is the current SOTA pruning approach for LLMs (**note that we have also clearly mentioned this fact in Section 3.2**). Therefore, **the crucial role of LOD as an informative criterion for pruning has already been exhibited by previous work.**


- Furthermore, the significance of outliers has been extensively explored in prior literature concerning language models [2,3] and LLM quantization [4,5,6]. The pivotal role of outliers to LLMs is directly illustrated by a quotation from LLM.int8() [4]: "Setting these outlier feature dimensions to zero decreases top-1 attention softmax probability mass by more than 20% and degrades validation perplexity by 600-1000% despite them only making up about 0.1% of all input features."

- In addition, our ablation study in Table 5 demonstrates that reversing the LOD metric (i.e., removing more weights in layers containing a higher concentration of outliers) significantly impairs performance (i.e., ranking as the second-worst among various pruning metrics). This further highlights the informativeness of LOD as an indicator for determining layerwise sparsity.

- In summary, we hold a high level of confidence that our conclusion does not rely on circular reasoning but rather is a scientific study that draws inspiration from previous works and our conclusion is thoroughly supported with solid empirical results.

Reviewer comment (Evaluation):
In Empirical Study 2, I do not see how having higher LOD for non-zero weights is equal to "preserving more outliers." The reason is twofold: First, the outliers after pruning may not necessarily be identical to the outliers before pruning. Second, the pruning also changes the average.

Author response:
**Q3: Empirical Study 2.** I do not see how having higher LOD for non-zero weights is equal to "preserving more outliers." The reason is twofold: First, the outliers after pruning may not necessarily be identical to the outliers before pruning. Second, the pruning also changes the average.

- Thank you for raising this important question. To clarify, the primary objective of Empirical Study II is to investigate the impact of the current LLM pruning technique on outliers. Hence, we anticipate that various pruning methodologies might yield distinct effects on outliers post-pruning. As you rightly mentioned, the outliers observed after pruning may not necessarily align with those observed before the pruning process. Nothing is wrong with this point.

- However, your second point is indeed valid and we were also aware of this during our analysis. In order to avoid the influence of pruning on average values, we maintained consistency by utilizing the pre-pruning average value to measure the outlier ratio after pruning. To illustrate, if the average value before pruning is 0.1, we utilized this same value of 0.1 to identify outlier weights after pruning. Subsequently, the number of outliers after pruning was divided by the total number of weights in the layer (including both zero and non-zero weights) to calculate the updated outlier ratio. This approach helps avoid the impact of pruning on average values, ensuring a precise evaluation of alterations in the outlier ratio. We have made our approach more clear in our revision.

Reviewer comment (Evaluation):
In Empirical Study 3, I cannot find a clear logical conclusion that can be made from this observation. How does this motivate the algorithm?

Author response:
**Q4: Empirical Study 3.** I cannot find a clear logical conclusion that can be made from this observation. How does this motivate the algorithm?

- The goal of Empirical Study III is to study the effect of pruning granularity on LLMs, a crucial facet of network pruning. It is a common belief in pruning vision models that global pruning typically leads to better performance than other types of constraint pruning granularity such as layer-wise pruning. However, some recent work such as SparseGPT and Wanda show that constraint pruning granularities like uniform layerwise pruning and output-balanced pruning are more preferable to LLM pruning. To solve this mystery, we conduct this empirical study to thoroughly investigate the effect of different pruning granularity on pruning performance. Our investigation confirms that pruning with coarser granularity results in diminished performance. In general, we observe a consistent trend of improved perplexity as the pruning granularity becomes finer, transitioning from global layerwise sparsity to uniform layerwise sparsity at the macro level, and from output-imbalanced sparsity to output-balanced sparsity at the micro level. This finding motivates us that the variance of our non-uniform layerwise sparsity cannot be too aggressive. Consequently, as we mentioned in Section 3.3, **"Additionally, we introduce a hyperparameter $\lambda$ which constrains the layerwise sparsity to fall within a small range, i.e., $S_i \in [S-\lambda, S+\lambda]$ ... This constraint is inspired by the insights gained from “Empirical Study III” which highlight the detrimental impact of overly aggressive layerwise sparsity, akin to global pruning, on sparse LLMs."**.

Reviewer comment (Experiments):
Uniform 50% sparsity is practically more useful than non-uniform sparsity at similar global sparsity level, because of the hardwares that support 2:4 sparsity.

Author response:
**Q6: Practicality (Major)** Uniform 50% sparsity is practically more useful than non-uniform sparsity at similar global sparsity level, because of the hardwares that support 2:4 sparsity.

- Yes, we totally agree with you that structured 2:4 sparsity is very useful in practice where uniform layerwise sparsity naturally fits this regime. To assess the potential of our non-uniform layerwise sparsity in hardware-friendly scenarios, we investigate OWL across three distinct hardware-friendly regimes:

    - **(1) N:M sparsity**: Following DominoSearch [9], we opt for mixed N:8 sparsity configuration. Instead of employing a uniform N value across all layers, we allow individual layers to possess distinct N values while maintaining the **same** parameter count. We adopt OWL to determine the optimal value of N for individual layers.
    
       *Table: Perplexity of Mixed N:M Sparity (N refers to non-zero weights)*
       
      | Method         | Laywewise Sparsity | Structure | Perplexity |
      |----------------|--------------------|-----------|------------|
      | Wanda          | Uniform            | 4:8       | 8.57       |
      | Wanda          | OWL                | Mixed 4:8 | **8.55**   |
      |                |                    |           |            |
      | Wanda          | Uniform            | 3:8       | 42.56      |
      | Wanda          | OWL                | Mixed 3:8 | **21.49**  |
      |                |                    |           |            |
      | Wanda          | Uniform            | 2:8       | 2962.00    |
      | Wanda          | OWL                | Mixed 2:8 | **331.37** |

    - **(2) Structured pruning**: Instead of pruning weights, we follow the recent methodology introduced in LLM Pruner [16], wherein entire neurons and attention heads are removed. This action facilitates direct acceleration of pruned LLMs on GPUs or TPUs. We replace the uniform layerwise sparsity used by LLM pruner to a non-uniform layerwise sparsity using OWL.
    
        *Table: Perplexity of Structure Pruning with LLaMA-7B on WikiText2 and PTB.*

      | Dataset    | Pruning Method | Layerwise Sparsity | 20%    | 40%    | 60%    | 80%    |
      |------------|----------------|--------------------|--------|--------|--------|--------|
      | Wikitext2  | LLM Pruner     | Uniform            | 19.09  | 30.39  | 90.017 | 1228.17|
      | Wikitext2  | LLM Pruner     | OWL                | **18.57** | **28.65** | **76.99** | **321.64** |
      | PTB        | LLM Pruner     | Uniform            | 29.51  | 66.90  | 192.06 | 1691.87|
      | PTB        | LLM Pruner     | OWL                | **28.82** | **53.22** | **150.16** | **502.07** |

    - **(3) Mixed-precision quantization**: Leveraging our non-uniform layerwise sparsity, we aim to enhance mixed-precision quantization by assigning higher precision to layers exhibiting more outliers. Following the approach outlined in [17], we utilize OWL to assign different bit precision to different layers, thereby facilitating a mixed-precision quantization strategy.Our baseline here is selecting with random and L1 norm of weights.
    
        *Table : Perplexity of Mixed-Precision Quantization with LLaMA-7B on WikiText2.*
    
      | Method              | Precision       | Perplexity   |
      |---------------------|-----------------|--------------|
      | Same Bit-width      | 2 Bit           | 104151.84    |
      | Same Bit-width      | 3 Bit           | 25.82        |
      | Same Bit-width      | 4 Bit           | 6.29         |
      |                     |                 |              |
      | Select with random  | Mixed 3/4 Bit   | 12.04        |
      | Select with L1 Norm | Mixed 3/4 Bit   | 14.61        |
      | Select with OWL     | Mixed 3/4 Bit   | **9.09**     |
      |                     |                 |              |
      | Select with random  | Mixed 2/3/4 Bit | 11455.54     |
      | Select with L1 Norm | Mixed 2/3/4 Bit | 13959.422    |
      | Select with OWL     | Mixed 2/3/4 Bit | **190.28**   |
      |                     |                 |              |
      | Select with random  | Mixed 2/4 Bit   | 14817.12     |
      | Select with L1 Norm | Mixed 2/4 Bit   | 33670.214    |
      | Select with OWL     | Mixed 2/4 Bit   | **7505.60**  |
    

    The ensuing results presented above corroborate the extension of OWL's advantages into more hardware-friendly regimes. Across diverse practical regimes, such as N:M sparsity, structured pruning, and mixed-precision quantization, OWL consistently yields substantial performance enhancements. Notably, in scenarios like the 3:8 and 2:8 structured sparsity regimes, OWL demonstrates its efficacy by achieving perplexity reductions of 2x and 8x, respectively, surpassing the uniform baseline.

Reviewer comment (Experiments):
It is quite confusing why authors compare with ERK instead of ER. ERK is something that is used for convolutional layers. Does Llama have any?

Author response:
**Q7: ERK (Minor)** It is quite confusing why authors compare with ERK instead of ER. ERK is something that is used for convolutional layers. Does Llama have any?

- Thanks for your knowledgeable comment. Yes, ERK is originally designed for convolutional layers. For linear layers, it will automatically scale back to ER. Therefore, we actually adopt ER for this comparison. We have fixed this in our revision.

Reviewer comment (Novelty):
Citing Frankle & Carbin (2019) for the global threshold is simply wrong (section 5.1.). Look at their figure 1, the last row.

Author response:
**Q8: Global (Minor)** Citing Frankle  Carbin (2019) for the global threshold is simply wrong (section 5.1.). Look at their figure 1, the last row.

- As indicated in Frankle and Carbin (2019), they employed uniform sparsity, referred to as 'layer-wise pruning,' for smaller models such as LeNet and Conv-2/4/6. Conversely, for larger models like VGG-19 and Resnet-18, they implemented global pruning, as documented in Section 4 on page 7 and Appendix I.1 on page 39.

- Furthermore, upon reviewing Figure 1, specifically the last row, we did not identify any content related to a global threshold. The last row of Figure 1 is 'sparse networks (average of ten trials). Solid lines are winning tickets (average of five trials).' If you are referring to a different aspect, could you please specify the particular section or figure?

Reviewer comment (Writing):
Sometimes, the paper says "outlier-weighted" instead of "outlier-weighed."

Author response:
**Q9: Typos (Minor)** Sometimes, the paper says "outlier-weighted" instead of "outlier-weighed." Section 2, paragraph 2, 3rd line from bottom, "inefficacy" should be something else.

- Thanks for these great, detailed comments. We have fixed them and polished our paper in our revision.

Reviewer comment (Writing):
Section 2, paragraph 2, 3rd line from bottom, "inefficacy" should be something else.

Author response:
**Q9: Typos (Minor)** Sometimes, the paper says "outlier-weighted" instead of "outlier-weighed." Section 2, paragraph 2, 3rd line from bottom, "inefficacy" should be something else.

- Thanks for these great, detailed comments. We have fixed them and polished our paper in our revision.