Original: S PARSITY M AY C RY : L ET U S F AIL (C URRENT ) S PARSE N EURAL N ETWORKS T OGETHER !
Revised: S PARSITY M AY C RY : L ET U S F AIL (C URRENT ) S PARSE N EURAL N ETWORKS T OGETHER !

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Sparse Neural Networks (SNNs) have received voluminous attention predominantly due to growing computational and memory footprints of consistently exploding parameter count in large-scale models.
Revised: Sparse Neural Networks (SNNs) have received voluminous attention predominantly due to growing computational and memory footprints of consistently exploding parameter count in large-scale models.

Original: Similar to their dense counterparts, recent
Revised: Similar to their dense counterparts, recent

Original: SNNs generalize just as well and are equipped with numerous favorable benefits (e.g., low complexity, high scalability, and robustness), sometimes even better than the original dense networks.
Revised: SNNs generalize just as well and are equipped with numerous favorable benefits (e.g., low complexity, high scalability, and robustness), sometimes even better than the original dense networks.

Original: As research effort is focused on developing increasingly sophisticated sparse algorithms, it is startling that a comprehensive benchmark to evaluate the effectiveness of these algorithms has been highly overlooked.
Revised: As research effort is focused on developing increasingly sophisticated sparse algorithms, it is startling that a comprehensive benchmark to evaluate the effectiveness of these algorithms has been highly overlooked.

Original: In absence of a carefully crafted evaluation benchmark, most if not all, sparse algorithms are evaluated against fairly simple and naive tasks (eg.
Revised: In absence of a carefully crafted evaluation benchmark, most if not all, sparse algorithms are evaluated against fairly simple and naive tasks (eg.

Original: CIFAR-10/100,
Revised: CIFAR-10/100,

Original: ImageNet, GLUE, etc.), which can potentially camouflage many advantages as well unexpected predicaments of SNNs.
Revised: ImageNet, GLUE, etc.), which can potentially camouflage many advantages as well unexpected predicaments of SNNs.

Original: In pursuit of a more general evaluation and unveiling the true potential of sparse algorithms, we introduce “ Sparsity MayCry ”
Revised: In pursuit of a more general evaluation and unveiling the true potential of sparse algorithms, we introduce “ Sparsity May

Original:  Benchmark (SMC-Bench) , a collection of carefully curated 4 diverse tasks with 12 datasets, that accounts for capturing a wide-range of domain-specific  knowledge.
Revised: Cry ” Benchmark (SMC-Bench) , a collection of carefully-curated 4 diverse tasks with 10 datasets, that accounts for capturing a wide range of domain-specific and sophisticated knowledge.

Original: Our systemic evaluation of  representativeSOTA sparse algorithms reveals an important obscured observation:all of the SOTA sparse algorithms bluntly fail to perform on SMC-Bench , sometimes at significantly trivial sparsity as low as 5% , which sought immediate attention of  sparsity  community to reconsider the highly proclaimed benefits of SNNs.
Revised: Our systemic evaluation of the most representative sparse algorithms reveals an important obscured observation: the state-of-the-art magnitude- and/or gradient-based sparse algorithms seemingly fail to perform on SMC-Bench when applied out-of-the-box, sometimes at significantly trivial sparsity as low as 5% . The observations seek the immediate attention of the sparsity research community to reconsider the highly proclaimed benefits of SNNs.

Revised: We further conduct a thorough investigation into the reasons for the failure of common SNNs.

Revised: Our analysis points out that such failure is intimately related to the “lazy regime” of large model training, which hints us with stronger pruning recipes that alleviate the failure on SMC-Bench (though still more or less suffering).

Original: By incorporating these well-thought and diverse tasks, SMC-Bench is designed to favor and encourage the development of highly generalizable sparse algorithms.
Revised: By incorporating these well-thought and diverse tasks, SMC-Bench is designed to favor and encourage the development of more scalable and generalizable sparse algorithms.

Original: Weplan to open-source SMC-Benchevaluation suite toencourage sparsity researchers and assist them in building next-generation sparse algorithms with the potential to generalize on complex and practical tasks.
Revised: We open-source SMC-Bench to assist researchers in building next-generation sparse algorithms that scale and generalize: https://github.com/VITA-Group/SMC-Bench .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Sparse Neural Networks (SNNs) are no stranger to the deep learning community, but recently they have received stupendous attention in the era of transformers (eg. BERT, GPT, ViT, CLIP, etc.), when the parameter count is frequently measured in billions rather than millions.
Revised: Sparse Neural Networks (SNNs) are no stranger to the deep learning community (Liu & Wang, 2023), but recently they have received stupendous attention in the era of transformers (eg. BERT, GPT, ViT, CLIP, etc.), when the parameter count is frequently measured in billions rather than millions.

Original: Due to the consistent efforts of sparsity researchers, SNNs have ushered enormous breakthroughs and can generalize just as well as original dense networks, and it is feasible to procure them after training (Frankle & Carbin, 2019;
Revised: Due to the consistent efforts of sparsity researchers, SNNs have ushered enormous breakthroughs and can generalize just as well as original dense networks, and it is feasible to procure them after training (Frankle & Carbin, 2019;

Original: Sanh et al., 2020;
Revised: Sanh et al., 2020;

Original: Chen et al., 2020; Frankle et al., 2020), during training (Zhu & Gupta, 2017; Gale et al., 2019;
Revised: Chen et al., 2020; Frankle et al., 2020), during training (Zhu & Gupta, 2017; Gale et al., 2019;

Original: Liu et al., 2021b), and even before training (Mocanu et al., 2018; Lee et al., 2019; Liu et al., 2022) their dense counterparts using pruning.
Revised: Liu et al., 2021b), and even before training (Mocanu et al., 2018; Lee et al., 2019; Liu et al., 2022) their dense counterparts using pruning.

Original: Apart from wellknown efficiencyand performance benefits, surprisingly, SNNs also enjoy auxiliary benefits such as adversarial robustness (Guo et al., 2018; ¨Ozdenizci & Legenstein, 2021), out-of-distribution generalization (Zhang et al., 2021; Diffenderfer et al., 2021), and uncertainty estimation (Liu et al., 2021a), etc.
Revised: Apart from well-known efficiency benefits, surprisingly, SNNs also enjoy auxiliary benefits such as adversarial robustness (Guo et al., 2018; ¨Ozdenizci & Legenstein, 2021; Chen et al., 2022), out-of-distribution generalization (Zhang et al., 2021; Diffenderfer et al., 2021), and uncertainty estimation (Liu et al., 2021a), etc.

Original: Despite the multi-dimensional success of numerous sparse algorithms, startlingly, our extensive survey across  100 recent SNNs papers within 2015-2022 unveils multiple daunting issues regarding evaluation datasets and protocols blindly followed within the sparsity community, that may significantly impede future progress if left unacknowledged.
Revised: Despite the multi-dimensional success of numerous sparse algorithms, startlingly, our extensive survey across over 100 recent SNN papers within 2015-2022 unveils multiple daunting issues regarding evaluation datasets and protocols blindly followed within the sparsity community, that may significantly impede future progress if left unacknowledged.

Original: Issues with current evaluation paradigm: Firstly , the vast majority of current work on SNNs is narrowly evaluated , i.e., only targeting a single or a few tasks (usually on image classification and sometimes on language understanding) on which SNNs have already proven their proficiency (Gale et al., 2019; Frankle & Carbin, 2019).
Revised: Issues with current evaluation paradigm: Firstly , the vast majority of current work on SNNs is narrowly evaluated , i.e., only targeting a single or a few tasks (usually on image classification and sometimes on language understanding) on which SNNs have already proven their proficiency (Gale et al., 2019; Frankle & Carbin, 2019).

Original: Surprisingly, 79 papers out of our carefully selected 100 papers on SNNs, evaluate sparse models merely on a single task, where 72 out of them evaluateon image classification.
Revised: Surprisingly, 79 papers out of our carefully selected 100 papers on SNNs, evaluate sparse models merely on a single task, where 72 out of them evaluate image classification.

Original: Secondly , people are obsessed with evaluating SNNs on well-understood datasets, including but not limited to MNIST (LeCun, 1998) (26 papers), CIFAR-10/100 (Krizhevsky et al., 2009) (59 and 37 papers, respectively), ImageNet (Deng et al., 2009) (62 papers), and GLUE (Wang et al., 2018)
Revised: Secondly , people are obsessed with evaluating SNNs on well-understood datasets, including but not limited to MNIST (LeCun, 1998) (26 papers), CIFAR-10/100 (Krizhevsky et al., 2009) (59 and 37 papers, respectively), ImageNet (Deng et al., 2009) (62 papers), and GLUE (Wang et al., 2018)

Original: (9 papers), where deep neural networks have already exceeded the human-equivalent performance (refer to Appendix D for more details).
Revised: (9 papers), where deep neural networks have already exceeded the human-equivalent performance (refer to Appendix D for more details).

Original: For instance, even though ImageNet has been consideredas a rather challenging task over years, very high accuracy ( > 90%) has been reported many times (Yu et al., 2022; Wortsman et al., 2022; Zhai et al., 2022).
Revised: For instance, even though ImageNet has been considered a rather challenging task over years, very high accuracy ( > 90%) has been reported many times (Yu et al., 2022; Wortsman et al., 2022; Zhai et al., 2022).

Original: Such relatively restrictedand simple evaluations will limit the development of sparse neural networks and are  ill-suited to identify new and unexpected capabilities of SNNs.
Revised: Such relatively restricted evaluations with “nearly saturated performance” limit the scope of sparse neural networks and are potentially ill-suited to identify new and unexpected capabilities of SNNs.

Original: To this end, addressing the aforementioned limitations of current SNNs evaluation protocols is urgently required and is important for the machine learning community, considering the critical role of SNNs in industry and academia.
Revised: Addressing the aforementioned limitations of current SNN evaluation protocols is a pressing need for the community.

Original: In this work, we assemble a large-scale, fairly arduous and diverse benchmark for sparse neural networks - “ Sparsity May Cry ” Benchmark (or briefly SMCBench ).
Revised: To this end, we assemble a large-scale, fairly arduous, and diverse benchmark for sparse neural networks - “ Sparsity May Cry ” Benchmark (or briefly SMC-Bench ).

Original: Specifically, we consider a broad set of tasks including complex reasoning, multilingual translation, and protein prediction , whose content spans multiple domains, requiring a vast amount of commonsense knowledge, solid mathematical and scientific background to solve.
Revised: Specifically, we consider a broad set of tasks including complex reasoning, multilingual translation, and protein prediction , whose content spans multiple domains. Those tasks require a vast amount of commonsense knowledge, solid mathematical and scientific backgrounds to solve even for humans.

Original: Note that none of the datasets in SMC-Bench has been created from scratch for the benchmark, we rely on pre-existing datasets as they have beenimplicitly agreed by researchers as challenging, interesting, and of high practical value.
Revised: Note that none of the datasets in SMC-Bench was created from scratch for the benchmark, we rely on pre-existing datasets as they have been agreed by researchers as challenging, interesting, and of high practical value.

Original: We rigorously measure the performance ofthe most state-of-the-art sparse subnetworks with varying sparsity levels produced by the off-the-shelf SOTA pruning and sparse training approaches  on SMC-Bench, to understand the potential of SNNs to generalize on complex and practical tasks.
Revised: We rigorously measure the performance of state-of-the-art (SOTA) pruning and sparse training approaches (in their most common, basic settings) on SMC-Bench, to understand the potential of SNNs to scale and generalize.

Original: In summary, our key observations and contributions can be unfolded as:
Revised: Our key observations and contributions can be unfolded as:

Original: • We presenta brand-new “ Sparsity May Cry ” Benchmark , to re-define the evaluation protocols for sparse neural networks and facilitate a comprehensive assessment of SOTA sparse algorithms.
Revised: • We present “ Sparsity May Cry ” Benchmark, to re-define the evaluation protocols for sparse neural networks and facilitate a comprehensive assessment of SOTA sparse algorithms.

Original: The premise of SMC-bench is to develop a suite of large-scale, challenging, realistic, and diverse tasks and datasets that can empower the rigorous advancements in thesparsity community.
Revised: The premise of SMC-bench is to develop a suite of large-scale, challenging, realistic, and diverse tasks and datasets that can empower the rigorous advancements in the community.

Original: • SMC-Bench unveils a critical and startling observation - all of the SOTA sparse algorithms bluntly fail to perform on SMC-Bench, sometimes at significantly trivial sparsity e.g., 5% .
Revised: • SMC-Bench unveils a critical and startling observation - all of the SOTA sparse algorithms seem to fail on SMC-Bench “out-of-the-box”, sometimes at significantly trivial sparsity e.g., 5% .

Original: Note that the failure does not appear specific to one sparsification approach but unanimously across all approaches we evaluated.
Revised: Note that the failure does not appear specific to one sparsification approach but unanimously across all approaches we evaluated.

Original: This observation alarmingly demands the attention of the sparsity community to reconsider the highly proclaimed benefits of SNNs.
Revised: This observation alarmingly demands the attention of the sparsity community to reconsider the highly proclaimed benefits of SNNs.

Original: • We conducted extensive experiments across representative SNNs produced by various SOTA pruning and sparse training approaches on SMC-Bench, and we summarize our findings: ❶ Model prunability is intimately related to task difficulty: models trained on difficult tasks suffer more from pruning compared to easier tasks; ❷ The success of beforetraining sparsification (sparse training or pruning at initialization) is hard to generalize in more complex scenarios; ❸ Iterative magnitude pruning (IMP) does not necessarily generalize better than one-shot pruning (OMP) or during-training pruning; ❹ Despite performance difference, different magnitude-based pruning approaches lead to extremely similar layerwise sprasities on SMC-Bench.
Revised: • We conduct extensive experiments across representative SNNs produced by various SOTA pruning and sparse training approaches on SMC-Bench, and we summarize our findings: ❶ Model prunability is intimately related to task difficulty: models trained on difficult tasks suffer more from pruning compared to easier tasks; ❷ The success of before-training sparsification (sparse training or pruning at initialization) is hard to generalize in more complex scenarios; ❸ Iterative magnitude pruning (IMP) does not necessarily generalize better than one-shot pruning (OMP) or during-training pruning; ❹ Despite performance difference, different magnitude-based pruning approaches lead to extremely similar layerwise sprasities. • We further carry out a comprehensive investigation into the potential causes of SNN failures on SMC-Bench.

Revised: Our analysis suggests that the failure of the existing sparse algorithms might be due to the “lazy regime” dynamics emerging in sufficiently overparameterized models (Chizat et al., 2019; Malladi et al., 2022).

Revised: Inspired by this finding, we hypothesize and confirm that the second-order pruning approaches, i.e., oBERT (Kurtic et al., 2022), are more reliable pruning approaches for SMC-Bench, which yield relatively more promising performance on SMC-Bench in Appendix C.

Original:  R ELATED W ORK 2. A DVANCES IN S PARSE N EURAL N ETWORKS
Revised:  R ELATED W ORK 2. A DVANCES IN S PARSE N EURAL N ETWORKS

Original: Post-Training.
Revised: Post-Training.

Original: SNNs refer to a neural network where a certain portion of its components (e g., weights, neurons, filters, and attention heads) have exactly zero values.
Revised: SNNs refer to a neural network where a certain portion of its components (e.g., weights, neurons, filters, and attention heads) have exactly zero values.

Original: The initial purpose of SNNs is retrospectively to accelerate model at inference time (a.k.a., post-training sparsification; Mozer &
Revised: The initial purpose of SNNs is retrospectively to accelerate model at inference time (a.k.a., post-training sparsification; Mozer &

Original: Smolensky (1989); LeCun et al.
Revised: Smolensky (1989); LeCun et al.

Original: Thanks to the over-parameterization property of deep neural networks, we can dramatically prune deep neural networks to smaller sizes with marginal loss of performance.
Revised: Thanks to the over-parameterization property of deep neural networks, we can dramatically prune deep neural networks to smaller sizes with marginal loss of performance.

Original: Post-training sparsification has been well studied and results in various mature criteria that can be generally categorized into zero-order methods (magnitude-based; Han et al.
Revised: Post-training sparsification has been well studied and results in various mature criteria that can be generally categorized into zero-order methods (magnitude-based; Han et al.

Original: , first-order methods (gradient-based; Molchanov et al. (2016); Sanh et al.
Revised: , first-order methods (gradient-based; Molchanov et al. (2016); Sanh et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2021)), and second-order methods (Hessian- LeCun 
Revised: (2021)), and second-order methods (Hessian-based; LeCun et al.

Original:  Hassibi & 
Revised: (1990); Hassibi & Stork (1992);

Original: Dong 
Revised: Dong et al.

Original: Second-order sparsification usually achieves higher performance than the other two but is also more expensive due to the full Hessian calculation.
Revised: Second-order sparsification usually achieves higher performance than the other two but is also more expensive due to the full Hessian calculation.

Original: Fortunately, many approaches have been proposed to efficiently approximate Hessian (Zeng & Urtasun, 2018;
Revised: Fortunately, many approaches have been proposed to efficiently approximate Hessian (Zeng & Urtasun, 2018;

Original: Wang et al., 2019; Singh &
Revised: Wang et al., 2019; Singh &

Original: Alistarh, 2020).
Revised: Alistarh, 2020).

Original: The Lottery Ticket Hypothesis (LTH) adopts iterative magnitude pruning (IMP) on fully trained networks and finds a subnetwork at initialization that can be re-trained in isolation to match the original dense networks.
Revised: The Lottery Ticket Hypothesis (LTH) adopts iterative magnitude pruning (IMP) on fully trained networks and finds a subnetwork at initialization that can be re-trained in isolation to match the original dense networks.

Original: Renda et al.
Revised: Renda et al.

Original: (2020) further found that instead of re-training with the initial weights, re-training with the final weights achieves better performance.
Revised: (2020) further found that instead of re-training with the initial weights, re-training with the final weights achieves better performance.

Revised: With the rise of large language models (LLMs), newer post-training pruning methods have emerged which aim to improve the affordability of these models (Sanh et al., 2020; Chen et al., 2020; Zafrir et al., 2021;

Revised: Kurtic et al., 2022; Xu et al., 2021; Lagunas et al., 2021; Zhang et al., 2022; Frantar et al., 2021).

Original: During-Training.
Revised: During-Training.

Original: During-training sparsification (Finnoff et al., 1993) is a cheaper option, compared to sparsify a fully converged model.
Revised: During-training sparsification (Finnoff et al., 1993) is a cheaper option, compared to sparsify a fully converged model.

Original: Approaches of during-training sparsification usually train a dense network for some time and then gradually sparsify the network with some schedules and end up with a sparse model with target sparsities.
Revised: Approaches of during-training sparsification usually train a dense network for some time and then gradually sparsify the network with some schedules and end up with a sparse model with target sparsities.

Original: Zhu & Gupta (2017); Gale et al.
Revised: Zhu & Gupta (2017); Gale et al.

Original: ; Lin et al.
Revised: ; Lin et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2021b) are highlight approaches that gradually prune networks during training and meanwhile allow the pruned weights to be reactivated in case of inaccurate pruning.
Revised: (2021b) are highlight approaches that gradually prune networks during training and meanwhile allow the pruned weights to be reactivated in case of inaccurate pruning.

Original: Another direction of during-training sparsification is adding sparsifying penalties such as (grouped) L 0 and
Revised: Another direction of during-training sparsification is adding sparsifying penalties such as (grouped) L 0 and

Original: L 1 norm to the loss function, which will punish the unimportant weights to zero, leading to sparse weights (Louizos et al., 2018; Luo & Wu, 2020; Savarese et al., 2020).
Revised: L 1 norm to the loss function, which will punish the unimportant weights to zero, leading to sparse weights (Louizos et al., 2018; Luo & Wu, 2020; Savarese et al., 2020).

Original: Prior-Training.
Revised: Prior-Training.

Original: Recently, foundation models (Brown et al., 2020; Chowdhery et al., 2022; Ramesh et al., 2022) have demonstrated promising quantitative improvement and new qualitative capabilities with increasing scale (Zhang et al., 2020b).
Revised: Recently, foundation models (Brown et al., 2020; Chowdhery et al., 2022; Ramesh et al., 2022) have demonstrated promising quantitative improvement and new qualitative capabilities with increasing scale (Zhang et al., 2020b).

Original: Along with the scaling of model size and data size, the training resources of these foundation models also get outrageous.
Revised: Along with the scaling of model size and data size, the training resources of these foundation models also get outrageous.

Original: To accelerate training, we need to sparsify models before training.
Revised: To accelerate training, we need to sparsify models before training.

Original: LTH unveils the possibility to find SNNs at initialization that can match their dense counterparts, even though it uses post-training pruning to find them.
Revised: LTH unveils the possibility to find SNNs at initialization that can match their dense counterparts, even though it uses post-training pruning to find them.

Original: At the same time, sparse training (Mocanu et al., 2018; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Evci et al., 2020; Kundu et al., 2020) was proposed that can train a randomly-initialized sparse neural network from scratch while dynamically optimizing the sparse connectivity with promising performance.
Revised: At the same time, sparse training (Mocanu et al., 2018; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Evci et al., 2020; Liu et al., 2021c; Schwarz et al., 2021) was proposed that can train a randomlyinitialized sparse neural network from scratch while dynamically optimizing the sparse connectivity with promising performance.

Original: Instead of randomly initializing sparse networks, one iteration (Lee et al., 2019; Wang et al., 2020) or a few iterations (Tanaka et al., 2020; de Jorge et al., 2021) of training can be utilized to guide the search for sparse networks before training.
Revised: Instead of randomly initializing sparse networks, one iteration (Lee et al., 2019; Wang et al., 2020) or a few iterations (Tanaka et al., 2020; de Jorge et al., 2021) of training can be utilized to guide the search for sparse networks before training.

Original: 2. B ENCHMARKING IN S PARSE N EURAL N ETWORKS
Revised: 2. B ENCHMARKING IN S PARSE N EURAL N ETWORKS

Original: Gale et al.
Revised: Gale et al.

Original: (2019) rigorously evaluated variational dropout (Molchanov et al., 2017), l 0 regularizaion (Louizos et al., 2018), and GMP (Zhu & Gupta, 2017) on two large-scale tasks.
Revised: (2019) rigorously evaluated variational dropout (Molchanov et al., 2017), l 0 regularizaion (Louizos et al., 2018), and GMP (Zhu & Gupta, 2017) on two large-scale tasks.

Original: They demonstrated that the appealing results of variational dropout and l 0 regularization can not generalize to large-scale tasks whereas simple magnitude pruning performs surprisingly well.
Revised: They demonstrated that the appealing performance advantages of variational dropout and l 0 regularization cannot generalize to large-scale tasks whereas simple magnitude pruning performs surprisingly well.

Original: Liu et al.
Revised: Liu et al.

Original: (2018) examined two pipelines: training from scratch and fine-tuning, concluding that fine-tuning a pruned model only gives comparable or worse performance than training from scratch.
Revised: (2018) examined two pipelines: training from scratch and fine-tuning, concluding that fine-tuning a pruned model only gives comparable or worse performance than training from scratch.

Original: Blalock et al.
Revised: Blalock et al.

Original: (2020) provided a comprehensive literature review on SNNs and found that pruning papers rarely make direct and controlled comparisons.
Revised: (2020) provided a comprehensive literature review on SNNs and found that pruning papers rarely make direct and controlled comparisons.

Original: Frankle et al.
Revised: Frankle et al.

Original: (2021) assessed the efficacy of various pruning at initialization approaches and attribute their inferior performance to their insensitivity to weight shuffling and re-initialization.
Revised: (2021) assessed the efficacy of various pruning at initialization approaches and attribute their inferior performance to their insensitivity to weight shuffling and re-initialization.

Original: Liu et al.
Revised: Liu et al.

Original: re
Revised: re

Original: -evaluated the performance of  random pruning before training and found that sparsely training a randomly pruned network from scratch can  match the performance of its dense equivalent.
Revised: -evaluated the performance of various random pruning before training and found that sparsely training a randomly pruned network from scratch can surprisingly match the performance of its dense equivalent.

Original: These papers shed light on the behavior of SNNs and discover important research problems for future work.
Revised: These papers shed light on the behavior of SNNs and discover important research problems for future work.

Original:  SMC
Revised:  SMC-B ENCH

Original: [Equation]

Original: SMC-Bench isa benchmark specifically crafted for the evaluation of sparse neural networks.
Revised: SMC-Bench is crafted for evaluating if all proclaimed benefits of SNNs can “scale and generalize”.

Original: It consists of 4 diverse and difficult tasks, including commonsense reasoning, arithmetic reasoning, multilingual translation, and protein prediction, with 10 datasets collected from prior work and opensource GitHub repositories.
Revised: It consists of 4 diverse and difficult tasks, including commonsense reasoning, arithmetic reasoning, multilingual translation, and protein prediction, with 10 datasets collected from prior work and opensource GitHub repositories.

Original: To investigate if there is a strong correlation between model prunability and task difficulty, we choose multiple datasets with different degrees of difficulty.
Revised: To investigate if there is a strong correlation between model prunability and task difficulty, we choose multiple datasets with different degrees of difficulty.

Original: 3. C OMMONSENSE R EASONING
Revised: 3. C OMMONSENSE R EASONING

Original: Commonsense reasoning task asks commonsense questions about the world involving complex semantics that often require rich common sense and background knowledge.
Revised: Commonsense reasoning task asks commonsense questions about the world involving complex semantics that often require rich common sense and background knowledge.

Original: We consider three commonly used datasets for commonsense reasoning.
Revised: We consider three commonly used datasets for commonsense reasoning.

Original: (1) RACE (Lai et al., 2017) contains near 28,000 passages and 100,000 questions collected from the English exams for Chinese students in middle (RACE-M) and high school (RACE-H).
Revised: (1) RACE (Lai et al., 2017) contains near 28,000 passages and 100,000 questions collected from the English exams for Chinese students in middle (RACE-M) and high school (RACE-H).

Original: (2) WinoGrande (Sakaguchi et al., 2021) is a modified version of the Winograd Schema Challenge (WSC) benchmark (Levesque et al., 2012) with enhanced scale and hardness, containing 44k problems.
Revised: (2) WinoGrande (Sakaguchi et al., 2021) is a modified version of the Winograd Schema Challenge (WSC) benchmark (Levesque et al., 2012) with enhanced scale and hardness, containing 44k problems.

Original: (3) Commonsense QA (CSQA) (Talmor et al., 2018) is a challenging dataset containing 12,247 multiple-choice questions where one source concept and three target concepts are first extracted from ConceptNet (Speer et al., 2017) based on which the Crowd-works are asked to author multiple-choice questions with two additional distractors.
Revised: (3) Commonsense QA (CSQA) (Talmor et al., 2018) is a challenging dataset containing 12,247 multiple-choice questions where one source concept and three target concepts are first extracted from ConceptNet (Speer et al., 2017) based on which the Crowd-works are asked to author multiple-choice questions with two additional distractors.

Original: In general, CommonsenseQA is harder than WinoGrande and RACE, with ceiling human performance of 89%, 94%, and 95%, respectively.
Revised: In general, CSQA is harder than WinoGrande and RACE, with ceiling human performance of 89%, 94%, and 95%, respectively.

Original: 3. A RITHMETIC R EASONING
Revised: 3. A RITHMETIC R EASONING

Original: Arithmetic reasoning poses a question of a math problem and the model is asked to generate a mathematical equation that can be evaluated to get the answer.
Revised: Arithmetic reasoning poses a question of a math problem and the model is asked to generate a mathematical equation that can be evaluated to get the answer.

Original: We consider the following three math word problem datasets: (1) the widely used MAWPS benchmark (Koncel-Kedziorski et al., 2016) composed of 2,373 problems; (2) the arithmetic subset of ASDiv (Miao et al., 2021) - ASDiv-A with 1,218 math problems; (3) the more challenging SVAMP (Patel et al., 2021) dataset which is created by applying complex types of variations to the samples from ASDiv-A.
Revised: We consider the following three math word problem datasets: (1) the widely used MAWPS benchmark (Koncel-Kedziorski et al., 2016) composed of 2,373 problems; (2) the arithmetic subset of ASDiv (Miao et al., 2021) - ASDiv-A with 1,218 math problems; (3) the more challenging SVAMP (Patel et al., 2021) dataset which is created by applying complex types of variations to the samples from ASDiv-A.

Original: The task difficulty monotonically increases from MAWPS to ASDiv-A, and to SVAMP.
Revised: The task difficulty monotonically increases from MAWPS to ASDiv-A, and to SVAMP.

Original: 3. P ROTEIN T HERMOSTABILITY P REDICTION
Revised: 3. P ROTEIN T HERMOSTABILITY P REDICTION

Original: Maintaining a stable 3D structure is an essential pre-condition for protein to function correctly in biological phenomena.
Revised: Maintaining a stable 3D structure is an essential pre-condition for protein to function correctly in biological phenomena.

Original: Numerous efforts are devoted to modeling and predicting protein’s stability against pH, salinity, and temperature.
Revised: Numerous efforts are devoted to modeling and predicting protein’s stability against pH, salinity, and temperature.

Original: We consider the tasks of protein thermostability prediction on two representative datasets: (1) HotProtein (Anonymous, 2022) is  a large-scale, standardized protein benchmark with organism-level temperature annotations, which contains 182 K protein sequences and 3 K folded structure from 230 different species.
Revised: We consider the tasks of protein thermostability prediction on two representative datasets: (1) HotProtein (Chen et al., 2023) is recently proposed as a large-scale, standardized protein benchmark with organism-level temperature annotations, which contains 182 K protein sequences and 3 K folded structure from 230 different species.

Original: Three dataset variants, HP-S ,HP-S 2 C 5 , and
Revised: Three dataset variants, HP-S ,

Original: HP-S 2 C  2 , are adopted to examine sequence- and structure-based methods, respectively.
Revised: HP-S 2 C 5 , and HP-S 2 C 2 , are adopted to examine sequence- and structure-based methods, respectively.

Original: HP-S has { 6 , 390 , 3 , 4946 , 30 , 333 , 79 , 087 , 31 , 549 } protein sequences from five categories of { Cryophilic , Psychrophilic , Mesophilic , Thermophilic , Hyperthermophilic } ; HP-S 2 C 5 has both sequences and structures for { 73 , 387 , 195 , 196 , 189 } proteins from the same five classes ordered from Cryophilic to Hyperthermophilic ; HP-S 2 C 2 has both sequences and structures for { 1 , 026 , 939 } proteins from { “hot” ( ≥ 45 ◦ C ), “cold” ( < 45 ◦ C ) } two classes.
Revised: HP-S has { 6 , 390 , 3 , 4946 , 30 , 333 , 79 , 087 , 31 , 549 } protein sequences from five categories of { Cryophilic , Psychrophilic , Mesophilic , Thermophilic , Hyperthermophilic } ; HP-S 2 C 5 has both sequences and structures for { 73 , 387 , 195 , 196 , 189 } proteins from the same five classes ordered from Cryophilic to Hyperthermophilic ; HP-S 2 C 2 has both sequences and structures for { 1 , 026 , 939 } proteins from { “hot” ( ≥ 45 ◦ C ), “cold” ( < 45 ◦ C ) } two classes.

Original: (2) Meltome Atlas (Jarzab et al., 2020) is another challenging test bed for protein’s thermostability.
Revised: (2) Meltome Atlas (Jarzab et al., 2020) is another challenging test bed for protein’s thermostability.

Original: It has { 7 , 902 , 15 , 833 , 10 , 518 } protein sequences from three of the five aforementioned classes, from Mesophilic to Hyperthermophilic .
Revised: It has { 7 , 902 , 15 , 833 , 10 , 518 } protein sequences from three of the five aforementioned classes, from Mesophilic to Hyperthermophilic .

Original: All samples are annotated with their melting temperature.
Revised: All samples are annotated with their melting temperature.

Original: 3. M ULTILINGUAL T RANSLATION
Revised: 3. M ULTILINGUAL T RANSLATION

Original: Multilingual translation processes multiple languages using a single language model and requires the model to have the ability to perform translation across languages.
Revised: Multilingual translation processes multiple languages using a single language model and requires the model to have the ability to perform translation across languages.

Original: We follow Liu et al.
Revised: We follow Liu et al.

Original: ; Tang et al.
Revised: ; Tang et al.

Original: and choose 10 English-centric language pairs (Fr, Cs, De, Gu, Ja, My, Ro, Ru, Vi, Zh ↔
Revised: and choose 10 English-centric language pairs (Fr, Cs, De, Gu, Ja, My, Ro, Ru, Vi, Zh ↔

Original: En) from an open source parallel corpus - OPUS (OPU, 2020).
Revised: En) from an open source parallel corpus - OPUS (OPU, 2020).

Original: We follow Arivazhagan et al.
Revised: We follow Arivazhagan et al.

Original: and use pivot data through English to create 3 Many-to-Many multilingual translation fine-tuning settings including 2 -to- 2 (Fr, Cs), 5 -to- 5 (Fr, Cs, De, Gi, Ja), and 10 -to- 10 .
Revised: and use pivot data through English to create 3 Many-to-Many multilingual translation fine-tuning settings including 2 -to- 2 (Fr, Cs), 5 -to- 5 (Fr, Cs, De, Gi, Ja), and 10 -to- 10 .

Original:  E VALUATION ON SMC-B ENCH
Revised:  E VALUATION ON SMC-B ENCH

Original: Models.
Revised: Models.

Original: Despite we are aware that performing few-shot prompting on large-scale pre-training language models with billions of parameters is able to solve these tasks (Wei et al., 2022; Srivastava et al., 2022), we choose to focus on fine-tuning or training with pre-trained mid-scale models with millions of parameters, to improve the accessibility of our Benchmark.
Revised: Despite we are aware that performing few-shot prompting on large-scale pre-training language models with billions of parameters is able to solve these tasks (Wei et al., 2022; Srivastava et al., 2022), we choose to focus on fine-tuning or training with pre-trained mid-scale models with millions of parameters, to improve the accessibility of our Benchmark.

Original: Specifically, we choose to finetune the popular RoBERTa (Liu et al., 2019) for commonsense reasoning; to fine-tune mBART (Liu et al., 2020) for multilingual translation; to train GTS (Xie & Sun, 2019) and Graph2Tree (Zhang et al., 2020a) with RoBERTa’s pre-trained embedding for arithmetic reasoning; to fine-tune Transformerbased (Vaswani et al., 2017) for protein property prediction.We sparsify all layers including embedding layers for all models.
Revised: Specifically, we choose to finetune the popular RoBERTa (Liu et al., 2019) for commonsense reasoning; to fine-tune mBART (Liu et al., 2020) for multilingual translation; to train GTS (Xie & Sun, 2019) and Graph2Tree (Zhang et al., 2020a) with RoBERTa’s pre-trained embedding for arithmetic reasoning; to fine-tune Transformerbased (Vaswani et al., 2017) for protein property prediction.

Original: All the detailed hyperparameters are shared in Appendix A.
Revised: See Appendix A for full details.

Original: Sparse Neural Networks.
Revised: Sparse Neural Networks.

Original: We choose the most representative  approaches from post-training, duringtraining, and prior-training sparsification approaches, and evaluate them on SMC-Bench.
Revised: We select the most representative magnitude- and/or gradient-based approaches where the prune operation is performed before, during, or after training.

Original: Formally, given a dense network θ l ∈ R d l with a dimension of d l in each layer l ∈ { 1 , . . .
Revised: Formally, given a dense network θ l ∈ R d l with a dimension of d l in each layer l ∈ { 1 , . . .

Original: , L } , pruning generates binary masks m l ∈ { 0 , 1 } d l yielding sparse neural networks with sparse weights θ l ⊙ m l .
Revised: , L } , pruning generates binary masks m l ∈ { 0 , 1 } d l yielding sparse neural networks with sparse weights θ l ⊙ m l .

Original: The sparsity level is the fraction of the weights that are zero-valued, calculated as s = 1 − (cid:80) l m l (cid:80) l d l .
Revised: The sparsity level is the fraction of the weights that are zero-valued, calculated as s = 1 − (cid:80) l m l (cid:80) l d l .

Original: The prune operation can be performed before training, during training, or after training.
Revised: Following a mainstream convention in many sparse training papers (Frankle & Carbin, 2019; Gale et al., 2019; Evci et al., 2020; Lee et al., 2019; Liu et al., 2021c), we sparsify most layers in the model including embedding layers and classifier heads, and we do not apply advanced techniques such as Iterative Learning Rate Rewinding (Renda et al., 2020) and Knowledge Distillation (Hinton et al., 2015) in our main evaluations, even if we observe that they help to alleviate accuracy drops as in Appendix C.

Original: • Lottery Ticket Hypothesis (LTH) (Frankle & Carbin, 2019) is a strong post-training pruning baseline that iteratively adopts magnitude pruning after training to produce binary masks and re-train together with weights from step t .
Revised: • Lottery Ticket Hypothesis (LTH) (Frankle & Carbin, 2019) is a strong post-training pruning baseline that iteratively adopts magnitude pruning after training to produce binary masks and re-train together with weights from step t .

Original: We set t = 0 in this paper, since rewinding to early training does not notably improve the performance of Transformer models (e.g., BERT) for downstream tasks (Chen et al., 2020).
Revised: We set t = 0 in this paper, since rewinding to early training does not notably improve the performance of Transformer models (e.g., BERT) for downstream tasks (Chen et al., 2020).

Original: The pruning rate of each IMP is set as 20%.
Revised: The pruning rate of each IMP is set as 20%.

Original: Magnitude After Training is a strong baseline for prune after training, which has demonstrated strong results in various regimes.
Revised: Magnitude After Training is a strong baseline for prune after training, which has demonstrated strong results in various regimes.

Original: After training or fine-tuning models on the specific task, we prune the model with one-shot magnitude pruning and re-train it with the full learning rate schedule from the beginning (Renda et al., 2020), dubbed “OMP (After)” in our experiments.
Revised: After training or fine-tuning models on the specific task, we prune the model with one-shot magnitude pruning and re-train it with the full learning rate schedule from the beginning, dubbed “OMP (After)” in our experiments.

Original: • Random After Training (Mittal et al., 2019) is the most naive baseline for post-training pruning.
Revised: • Random After Training (Mittal et al., 2019) is the most naive baseline for post-training pruning.

Original: It uniformly samples a random score s l ∈ Uniform (0 , 1) for each weight and prunes the weights with the lowest scores.
Revised: It uniformly samples a random score s l ∈ Uniform (0 , 1) for each weight and prunes the weights with the lowest scores.

Original: After pruning, we also re-trainthe sparse networks with the entire training schedule (Renda et al., 2020).
Revised: After pruning, we also re-train with the entire training schedule.

Original: • Gradual Magnitude Pruning (GMP) (Zhu & Gupta, 2017; Gale et al., 2019) gradually sparsifies networks during training according to a pre-defined sparsification schedule with sorting-based weight thresholding.
Revised: • Gradual Magnitude Pruning (GMP) (Zhu & Gupta, 2017; Gale et al., 2019) gradually sparsifies networks during training according to a pre-defined sparsification schedule with sorting-based weight thresholding.

Original: The starting and the ending iteration of the gradual sparsification process are set as 10% and 80% of the entire training iterations.
Revised: The starting and the ending iteration of the gradual sparsification process are set as 10% and 80% of the entire training iterations.

Original: The frequency of sparsification steps is tuned among 500, 1000, and 4000, depending on the specific tasks.
Revised: The frequency of sparsification steps is tuned among 500, 1000, and 4000, depending on the specific tasks.

Original: While we are aware of the advanced gradual pruning methods - movement pruning (Sanh et al., 2020), it usually exceeds GMP only at high sparsities (e.g., > 90%), which is interesting but not within the scope of this paper.
Revised: While we are aware of the advanced gradual pruning methods - movement pruning (Sanh et al., 2020), it usually exceeds GMP only at high sparsities (e.g., > 90%), which is interesting but not within the scope of this paper.

Original: • Magnitude Before Training (Frankle et al., 2021) simply removes weights with the lowest magnitude at initialization.
Revised: • Magnitude Before Training (Frankle et al., 2021) simply removes weights with the lowest magnitude at initialization.

Original: Since we inherit weights from pre-trained models, the initial weights actually refer to the weights that are learned on the pre-trained tasks.
Revised: Since we inherit weights from pre-trained models, the initial weights actually refer to the weights that are learned on the pre-trained tasks.

Original: We abbreviate this approach to “OMP (Before)” as we use one-shot magnitude pruning.
Revised: We abbreviate this approach to “OMP (Before)” as we use one-shot magnitude pruning.

Original: • Random Before Training (Liu et al., 2022) is the most naive baseline for prior-training pruning.
Revised: • Random Before Training (Liu et al., 2022) is the most naive baseline for prior-training pruning.

Original: We randomly sample scores for each weight and removes the weights with the lowest scores.
Revised: We randomly sample scores for each weight and removes the weights with the lowest scores.

Original: Different from Random After Training, the pruning operation is performed before fine-tuning models on the downstream tasks.
Revised: Different from Random After Training, the pruning operation is performed before fine-tuning.

Original: • SNIP
Revised: • SNIP

Original: (Lee et al., 2019) is a prior-training pruning technique that globally removes weights with the lowest connection sensitivity score | g ⊙ w | .
Revised: (Lee et al., 2019) is a prior-training pruning technique that globally removes weights with the lowest connection sensitivity score | g ⊙ w | .

Original: SNIP is a strong baseline that consistently performs well among various prior-training approaches (Frankle et al., 2021).
Revised: SNIP is a strong baseline that consistently performs well among various prior-training approaches (Frankle et al., 2021).

Original: • Rigging the Lottery (RigL)
Revised: • Rigging the Lottery (RigL)

Original: (Evci et al., 2020) is a leading sparse training method that updates the topology of sparse neural networks during training via a prune-and-grow scheme.
Revised: (Evci et al., 2020) is a leading sparse training method that updates the topology of sparse neural networks during training via a prune-and-grow scheme.

Original: To evaluate its effectiveness on downstream fine-tuning, we combine RigL with the other three prior-training methods.
Revised: To evaluate its effectiveness on downstream fine-tuning, we combine RigL with the other three prior-training methods.

Original: The update interval of RigL is set the same as the ones used for updating sparsity in GMP, following Liu et al.
Revised: The update interval of RigL is set the same as the ones used for updating sparsity in GMP, following Liu et al.

Original: (2021b).
Revised: (2021b).

Original: 4. C OMMONSENSE R EASONING
Revised: [Figure] [Figure]

Original: Implementation Details.
Revised: Implementation Details.

Original: We follow the training settings of sequence modeling toolkit Fairseq (Ott et al., 2019) and fine-tune the pre-trained RoBERTa on our datasets with a standard cross-entropy loss.
Revised: We follow the training settings of sequence modeling toolkit Fairseq (Ott et al., 2019) and fine-tune the pre-trained RoBERTa on our datasets with a standard cross-entropy loss.

Original: Specifically for each question, we also construct five inputs, one for each of the five candidate answer choices.
Revised: Specifically for each question [Figure] we also construct five inputs, one for each of the five candidate answer choices.

Original: Each input is constructed by concatenating the question and candidate answer together.
Revised: Each input is constructed by concatenating the question and candidate answer together.

Original: We then encode each input and pass the resulting “[CLS]” representations through a classifier to predict the correct answer.
Revised: We then encode each input and pass the resulting “[CLS]” representations through a classifier to predict the correct answer.

Original: All models are trained with the Adam (Kingma & Ba, 2014) optimizer with a learning rate of 1 × 10 − 5 using an A100 GPU.
Revised: All models are trained with the Adam (Kingma & Ba, 2014) optimizer with a learning rate of 1 × 10 − 5 using an A100 GPU.

Original: For CSQA, we train the model for 3000 steps with a linear warmup of 150 steps and a batch size of 16 .
Revised: For CSQA, we train the model for 3000 steps with a linear warmup of 150 steps and a batch size of 16 .

Original: The dropout rate is set as 0.1.
Revised: The dropout rate is set as 0.1.

Original: This gives us a test accuracy of 77 .
Revised: This gives us a test accuracy of 77 .

Original: 3% with dense RoBERTa.
Revised: 3% with dense RoBERTa.

Original: For RACE, we train each model for 3 epochs with a batch size of 16 .
Revised: For RACE, we train each model for 3 epochs with a batch size of 16 .

Original: This gives us 86 .
Revised: This gives us 86 .

Original: 6% and 82 .
Revised: 6% and 82 .

Original: 0% dense accuracy on RACE (H) and RACE (M), matching the ones reported in Fairseq ( 86 .
Revised: 0% dense accuracy on RACE (H) and RACE (M), matching the ones reported in Fairseq ( 86 .

Original: 5% and 81 .
Revised: 5% and 81 .

Original: Models on WinoGrande are trained for 23 , 750 steps with 2 , 735 warmup steps and 32 batch size, reaching a 76 . 3% accuracy.
Revised: Models on WinoGrande are trained for 23 , 750 steps with 2 , 735 warmup steps and 32 batch size, reaching a 76 . 3% accuracy.

Original: Results and Analyses.
Revised: Results and Analyses.

Original: The results of various sparse neural networks are demonstrated in Figure 1.
Revised: The results of various sparse neural networks are demonstrated in Figure 1.

Original: We summarize  main observations below:
Revised: We summarize our main observations below:

Original: [Figure] 1 All sparse algorithms  fail to find matching SNNs at trivial sparsities, even low as 36%.
Revised: 1 All sparse algorithms seemingly fail to find matching SNNs, even at trivial sparsities such as 36%.

Original: While several methods maintain the dense performance at 20% sparsity, their performance starts to drop significantly after that, and will undergo a catastrophic failure as the sparsity continues increasing.
Revised: While several methods maintain the dense performance at 20% sparsity, their performance starts to drop significantly after that, and will undergo a catastrophic failure as the sparsity continues increasing.

Original: It is difficult even for the top-performing LTH to maintain the matching performance after the 2 rd IMP iteration.
Revised: It is difficult even for the top-performing LTH to maintain the matching performance after the 2 rd IMP iteration.

Original: This is in stark contrast with the behavior of SNNs on the image classification task, where LTH can gracefully preserve the matching performance even at very extreme sparsities ( > 95% on CIFAR-10/100and > 80% (Yin et al., 2022)  on ImageNet (Renda et al., 2020)).
Revised: This is in stark contrast with the behavior of SNNs on the image classification task, where LTH can gracefully preserve the matching performance even at very extreme sparsities ( > 95% on CIFAR-10/100 (Yin et al., 2022) and > 80% on ImageNet (Renda et al., 2020)).

Original: 2 The quality of SNNs on harder tasks suffers more from sparsity.
Revised: The quality of SNNs on harder tasks suffers more from sparsity.

Original: Models trained on the hardest task, CSQA, undergo a larger accuracy loss at the same sparsity than the other two datasets.
Revised: Models trained on the hardest task, CSQA, undergo a larger accuracy loss at the same sparsity than the other two datasets.

Original: For instance, all the SNNs on CSQA suffer from a catastrophic accuracy drop (up to 74%) and become no better than the random prediction at 59% sparsity.
Revised: For instance, all the SNNs on CSQA suffer from a catastrophic accuracy drop (up to 74%) and become no better than the random prediction at 59% sparsity.

Original: Meanwhile, when trained on WinoGrande and RACE at 59% sparsity, two sparse algorithms (LTH and GMP) can maintain relatively good performance with a smaller performance loss (i.e., 3% ∼ 10%).
Revised: Meanwhile, when trained on WinoGrande and RACE at 59% sparsity, two sparse algorithms (LTH and GMP) can maintain relatively good performance with a smaller performance loss (i.e., 3% ∼ 10%).

Original: 3 Post-training pruning consistently outperforms prior-training pruning.
Revised: 3 Post-training pruning consistently outperforms prior-training pruning.

Original: LTH achieves the best performance across datasets, GMP performs well, and OMP (After) follows behind.
Revised: LTH achieves the best performance across datasets, GMP performs well, and OMP (After) follows behind.

Original: However, priortraining pruning achieves worse performance.
Revised: However, prior- training pruning achieves worse performance.

Original: OMP (Before) performs closely behind OMP (After), whereas SNIP performs no better than the naive random pruning.
Revised: OMP (Before) performs closely behind OMP (After), whereas SNIP performs no better than the naive random pruning.

Original: After digging deeper into the case of SNIP, we find SNIP aggressively prunes the embedding layers to more than 99% sparsity even with a mild overall sparsity of 20%.
Revised: After digging deeper into the case of SNIP, we find SNIP aggressively prunes the embedding layers to more than 99% sparsity even with a mild overall sparsity of 20%.

Original: Surprisingly, the leading dynamic sparsity approach RigL does not bring significant gains to these prior-training approaches, and sometimes even hurts the performance.
Revised: Surprisingly, the leading dynamic sparsity approach RigL does not bring significant gains to these prior-training approaches, and sometimes even hurts the performance.

Original: 4. A RITHMETIC R EASONING
Revised: A RITHMETIC R EASONING

Original: [Figure]

Original: Implementation Details.
Revised: Implementation Details.

Original: We follow SVAMP (Patel et al., 2021) and choose the two top-performing tree-based models for arithmetic reasoning: GTS (Xie & Sun, 2019) and Graph2Tree (Zhang et al., 2020a).
Revised: We follow SVAMP (Patel et al., 2021) and choose the two top-performing tree-based models for arithmetic reasoning: GTS (Xie & Sun, 2019) and Graph2Tree (Zhang et al., 2020a).

Original: Graph2Tree in general performs slightly better than GTS.
Revised: Graph2Tree in general performs slightly better than GTS.

Original: GTS adopts LSTM to encode input sequences and a tree-based Decoder to generate questions.
Revised: GTS adopts LSTM to encode input sequences and a tree-based Decoder to generate questions.

Original: Graph2Tree uses a graph transformer to learn the latent quantity representations from data, and a tree structure decoder to generate a solution expression tree.
Revised: Graph2Tree uses a graph transformer to learn the latent quantity representations from data, and a tree structure decoder to generate a solution expression tree.

Original: We follow exactly the training settings of Patel et al.
Revised: We follow exactly the training settings of Patel et al.

Original: The embedding weights are inherited from THE pre-trained RoBERTa.
Revised: The embedding weights are inherited from THE pre-trained RoBERTa.

Original: All models are trained with Adam for 50 epochs.
Revised: All models are trained with Adam for 50 epochs.

Original:  On MAWPS and ASDiv-A, models are trained with the training data and then evaluated on 5-fold cross-validation based on pre-assigned splits.
Revised: [Figure] On MAWPS and ASDiv-A, models are trained with the training data and then evaluated on 5-fold cross-validation based on pre-assigned splits.

Original: For SVAMP, we train the models on a combination of
Revised: For SVAMP, we train the models on a combination of

Original: MAWPS and ASDiv-A and test them on SVAMP, following Patel et al.
Revised: MAWPS and ASDiv-A and test them on SVAMP, following Patel et al.

Original: Results and Analyses.
Revised: Results and Analyses.

Original: The performance on arithmetic reasoning is reported in Figure 2.
Revised: The performance on arithmetic reasoning is reported in Figure 2.

Original: The overall performance trend is very similar to the commonsense reasoning: SNNs can only match the dense performance when the sparsity level is lower than 48.8% with the exception of Graph2Tree on the relatively simple MAWPS dataset whose failing sparsity is 59% ; SNNs are prone to sacrifice more performance on harder datasets (i.e., ASDiv-A and SVAMP) than the easier MAWPS dataset; prior-training methods perform no differently than random pruning.
Revised: The overall performance trend is very similar to the commonsense reasoning: SNNs can only match the dense performance when the sparsity level is lower than 48.8% with the exception of Graph2Tree on the relatively simple MAWPS dataset whose failing sparsity is 59% ; SNNs are prone to sacrifice more performance on harder datasets (i.e., ASDiv-A and SVAMP) than the easier MAWPS dataset; prior-training methods perform no differently than random pruning.

Original: Moreover, we want to highlight that LTH surprisingly reaches lower accuracy than OMP and GMP at high sparsity levels, indicating that iterative magnitude pruning may not necessarily generalize better than on more complex tasks.
Revised: Moreover, we want to highlight that LTH surprisingly reaches lower accuracy than OMP and GMP at high sparsity levels, indicating that iterative magnitude pruning may not necessarily generalize better than on more complex tasks.

Original: Moreover, Magnitude Before Training (OMG (Before)) consistently causes severe layer collapse in the non-embedding layers, leading to zero accuracies.
Revised: Moreover, Magnitude Before Training (OMG (Before)) consistently causes severe layer collapse in the non-embedding layers, leading to zero accuracies.

Original: Since including the results of OMG (Before) will significantly dilute the performance difference of different sparsification methods, we choose to report it in Appendix B.
Revised: Since including the results of OMG (Before) will significantly dilute the performance difference of different sparsification methods, we choose to report it in Appendix B.

Original: 4. P ROTEIN T HERMAL S TABILITY P REDICTION 4.3. S EQUENCE -B ASED M ODELS
Revised: 4. P ROTEIN T HERMAL S TABILITY P REDICTION 4.3. S EQUENCE -B ASED M ODELS

Original: Implementation Details.
Revised: Implementation Details.

Original: We examine two classic sequence-based approaches in protein property prediction, i.e. , TAPE (Rao et al., 2019) and ESM-1B (Rives et al., 2021).
Revised: We examine two classic sequence-based approaches in protein property prediction, i.e. , TAPE (Rao et al., 2019) and ESM-1B (Rives et al., 2021).

Original: For TAPE, we fine-tune it from the official pre-training (Rao et al., 2019) for 4 epochs with an initial learning rate of 1 × 10 −and a linear decay scheduler together with 100 warm-up steps.
Revised: For TAPE, we fine-tune it from the official pre-training (Rao et al., 2019) for 4 epochs with an initial learning rate of 1 × 10 −and a linear decay scheduler together with 100 warm-up steps.

Original: As for ESM-1B (Rives et al., 2021), starting from the official pre-trained checkpoint, we fine-tune the backbone with a learning rate of 1 × 10 − 6 and the linear classification head on top of the backbone with a learning rate of 2 × 10 − 2 .
Revised: As for ESM-1B (Rives et al., 2021), starting from the official pre-trained checkpoint, we fine-tune the backbone with a learning rate of 1 × 10 − 6 and the linear classification head on top of the backbone with a learning rate of 2 × 10 − 2 .

Original: The learning rate schedulers used for both backbone and linear head are OneCycle (Smith & Topin, 2019) decay schedulers.
Revised: The learning rate schedulers used for both backbone and linear head are OneCycle (Smith & Topin, 2019) decay schedulers.

Original: The training batch size is 2 for Meltome Atlas and 3 for HotProtein (HP-S).
Revised: The training batch size is 2 for Meltome Atlas and 3 for HotProtein (HP-S).

Original: Classification accuracy on test sets is collected to measure the model performance.
Revised: Classification accuracy on test sets is collected to measure the model performance.

Original: Results and Analyses.
Revised: Results and Analyses.

Original: In this section, we examine diverse sparse neural networks of sequencebased models ( i.e. , transformers) on protein thermostability prediction benchmarks.
Revised: In this section, we examine diverse sparse neural networks of sequencebased models ( i.e. , transformers) on protein thermostability prediction benchmarks.

Original: ESM-1B (Rives et al., 2021), a SOTA approach in protein property modeling, is evaluated on both HotProtein (HP-S) and Meltome Atlas datasets.
Revised: ESM-1B (Rives et al., 2021), a SOTA approach in protein property modeling, is evaluated on both HotProtein (HP-S) and Meltome Atlas datasets.

Original: [Table]
Revised: TAPE (

Original: TAPE (Rao et al., 2019) is a classic transformer-based model adopted on HotProtein (HP-S).
Revised: Rao et al., 2019) is a classic transformer-based model adopted on HotProtein (HP-S).

Original: Extensive results of both static and dynamic sparsifications are collected in Figure 3.
Revised: Extensive results of both static and dynamic sparsifications are collected in Figure 3.

Original: We observe that: ❶ For ESM-1B, all extracted sparse neural networks incur significant performance degradation whenever the sparsity level is larger than 20% .
Revised: We observe that: ❶ For ESM-1B, all extracted sparse neural networks incur significant performance degradation whenever the sparsity level is larger than 20% .

Original: Note that here we only sparsify the fully connected layers in multi-head self-attentions & feed-forward networks of each  transformer layer, and leave all other modules unpruned.
Revised: Note that here we only sparsify the fully connected layers in multi-head self-attentions & feed-forward networks of each [Figure] transformer layer, and leave all other modules unpruned.

Original: Even under this loose condition, ESM-1B still fails after pruning on both HP-S and Meltome Atlas, which indicates the low parameter redundancy in ESM-1B for modeling protein thermal stability.
Revised: Even under this loose condition, ESM1B still fails after pruning on both HP-S and Meltome Atlas, which indicates the low parameter redundancy in ESM-1B for modeling protein thermal stability.

Original: ❷
Revised: ❷

Original: In general, static and dynamic pruning algorithms achieve similar performance with ESM-1B. SNIP (Before) and SNIP + RIGL (Before) deliver relatively better accuracies, especially for the high sparsity ( ≥ 48 .
Revised: In general, static and dynamic pruning algorithms achieve similar performance with ESM-1B. SNIP (Before) and SNIP + RIGL (Before) deliver relatively better accuracies, especially for the high sparsity ( ≥ 48 .

Original: 80% ) subnetworks on HP-S. ❸
Revised: 80% ) subnetworks on HP-S. ❸

Original: As for the worse backbone TAPE compared with ESM-1B, magnitude-based prunings like LTH (After), OMP (After), and GMP (During) show satisfactory results before 59% sparsity.
Revised: As for the worse backbone TAPE compared with ESM-1B, magnitude-based prunings like LTH (After), OMP (After) [Figure] and GMP (During) show satisfactory results before 59% sparsity.

Original:  Furthermore, we conduct a more fine-grained pruning schedule to investigate the tolerance of ESM1B against sparsification.
Revised: [Table] Furthermore, we conduct a more fine-grained pruning schedule to investigate the tolerance of ESM-1B against sparsification.

Original: In detail, we prune 5% weights with OMP (after) on different modules in ESM-1B and present the obtained accuracies in Table 1.
Revised: In detail, we prune 5% weights with OMP (after) on different modules in ESM-1B and present the obtained accuracies in Table 1.

Original: Q, K, V, O, and FFN represent the fully[Figure] connected layer in the query, key, value, & output of the self-attention module and feed-forward networks, respectively.
Revised: Q, K, V, O, and FFN represent the fully connected layer in the query, key, value, & output of the self-attention module and feed-forward networks, respectively.

Original: The results show that whatever modules we select, 5% sparsity damages the ESM-1B performance of protein thermostability prediction on HP-S 2 C2.
Revised: The results show that whatever modules we select, 5% sparsity damages the ESM-1B performance of protein thermostability prediction on HP-S 2 C2.

Original: 4.3. S TRUCTURE -B ASED M ODELS
Revised: 4.3. S TRUCTURE -B ASED M ODELS

Original: Implementation Details.
Revised: Implementation Details.

Original: We further consider a representative structure-based algorithm for thermostability prediction, i.e. , ESM-IF1 (Hsu et al., 2022).
Revised: We further consider a representative structure-based algorithm for thermostability prediction, i.e. , ESM-IF1 (Hsu et al., 2022).

Original: Specifically, for ESM-IF1, we train the backbone and its linear head with learning rates of 1 × 10 − 4 and 2 × 10 − 2 , respectively.
Revised: Specifically, for ESM-IF1, we train the backbone and its linear head with learning rates of 1 × 10 − 4 and 2 × 10 − 2 , respectively.

Original: A batch size of 4 is used for both models on HotProtein (HP-S 2 C 5 ).
Revised: A batch size of 4 is used for both models on HotProtein (HP-S 2 C 5 ).

Original: Classification testing accuracy is reported to reflect the model performance.
Revised: Classification testing accuracy is reported to reflect the model performance.

Original: Results and Analyses.
Revised: Results and Analyses.

Original: In this section, we study structure-based models and their sparse variants on HotProtein (HP-S 2 C 5 ).
Revised: In this section, we study structure-based models and their sparse variants on HotProtein (HP-S 2 C 5 ).

Original: ESM-IF1 (Hsu et al., 2022), a recent SOTA approach, is chosen for benchmarking.
Revised: ESM-IF1 (Hsu et al., 2022), a recent SOTA approach, is chosen for benchmarking.

Original: It takes the 3D structure of proteins as input and predicts its thermal stable temperature.
Revised: It takes the 3D structure of proteins as input and predicts its thermal stable temperature.

Original: As shown in Figure 3, ESM-IF1 produces inferior sparse neural networks with all pruning mechanisms of both static and dynamic, where OMP (after) and GMP (During) present relatively better accuracies.
Revised: As shown in Figure 3, ESM-IF1 produces inferior sparse neural networks with all pruning mechanisms of both static and dynamic, where OMP (after) and GMP (During) present relatively better accuracies.

Original: 4. M ULTILINGUAL T RANSLATION
Revised: 4. M ULTILINGUAL T RANSLATION

Original: Implementation Details.
Revised: Implementation Details.

Original: We choose the official multilingual model mBART 1 (Liu et al., 2020), which was originally pre-trained on 25 languages using masked language modeling (MLM), following the fine-tuning setting of Tang et al.
Revised: We choose the official multilingual model mBART 1 (Liu et al., 2020), which was originally pre-trained on 25 languages using masked language modeling (MLM), following the fine-tuning setting of Tang et al.

Original: We first choose 10 languages from the language pools used for MLM pre-training; create three sub-groups containing 2 , 5 , 10 languages; and fine-tune mBART on each sub-group, referring to 2 -to- 2 , 5 -to- 5 , and 10 -to- 10 multilingual translation finetuning, respectively.
Revised: We first choose 10 languages from the language pools used for MLM pre-training; create three sub-groups containing 2 , 5 , 10 languages; and fine-tune mBART on each sub-group, referring to 2 -to- 2 , 5 -to- 5 , and 10 -to- 10 multilingual translation finetuning, respectively.

Original: During inference, we report the averaged BLEU (Tang et al., 2020; Liu et al., 2020) scores of bi-directional translation across 10 languages to measure the translation performance.
Revised: During inference, we report the averaged BLEU (Tang et al., 2020; Liu et al., 2020) scores of bi-directional translation across 10 languages to measure the translation performance.

Original: Hence, the task difficulty monotonically decreases from 2 -to- 2 to 5 -to- 5 , and to 10 -to- 10 fine-tuning as more languages are involved during training.
Revised: Hence, the task difficulty monotonically decreases from 2 -to- 2 to 5 -to- 5 , and to 10 -to- 10 fine-tuning as more languages are involved during training.

Original: The default fine-tuning configurations in Tang et al.
Revised: The default fine-tuning configurations in Tang et al.

Original: (2020) are adopted for 40 K iterations with an Adam optimizer and a learning rate of 1 × 10 − 6 .
Revised: (2020) are adopted for 40 K iterations with an Adam optimizer and a learning rate of 1 × 10 − 6 .

Original: Results and Analyses.
Revised: Results and Analyses.

Original: Intuitively, fewer languages involved during fine-tuning leads to a more difficult translation for all languages.
Revised: Intuitively, fewer languages involved during fine-tuning leads to a more difficult translation for all languages.

Original: As demonstrated in Figure 4, several consistent conclusions can be drawn: ❶ Besides OMP (After) and
Revised: As demonstrated in Figure 4, several consistent conclusions can be drawn: ❶ Besides OMP (After) and

Original: LTH (After), all other produced sparse subnetworks perform worse than the dense baseline when the sparsity is larger than 20% .
Revised: LTH (After), all other produced sparse subnetworks perform worse than the dense baseline when the sparsity is larger than 20% .

Original: The BLEU scores of OMP (After) and LTH (After) also decline and fail to match at ≥ 20% , ≥ 48 .
Revised: The BLEU scores of OMP (After) and LTH (After) also decline and fail to match at ≥ 20% , ≥ 48 .

Original: 8% , ≥ 59% sparsity levels for 2 -to- 2 , 5 -to- 5 , and 10 -to- 10 fine-tuning, respectively.
Revised: 8% , ≥ 59% sparsity levels for 2 -to- 2 , 5 -to- 5 , and 10 -to- 10 fine-tuning, respectively.

Original: ❷ Magnitude-based sparsifications like OMP, LTH, and GMP are comparably robust across all three translation settings, while other pruning methods have negligible advantages compared to random pruning.
Revised: ❷ Magnitude-based sparsifications like OMP, LTH, and GMP are comparably robust across all three translation settings, while other pruning methods have negligible advantages compared to random pruning.

Original: ❸ While the overall tendency of SNNs is quite consistent across different tasks, the prunability of mBART increases as more languages are involved during fine-tuning.
Revised: ❸ While the overall tendency of SNNs is quite consistent across different tasks, the prunability of mBART increases as more languages are involved during fine-tuning.

Original: It seems that multilingual translation has already been a challenging task for pruning, and involving more languages in inference causes extra obstacles.
Revised: It seems that multilingual translation has already been a challenging task for pruning, and involving more languages in inference causes extra obstacles.

Original: This is why in the hardest scenario of fine-tuning on 2 -to- 2 and evaluating with 10 languages, all sparse subnetworks suffer from substantial performance degradation.
Revised: This is why in the hardest scenario of fine-tuning on 2 -to- 2 and evaluating with 10 languages, all sparse subnetworks suffer from substantial performance degradation.

Original: T HE R EASON W HY SNN S F AIL

Original: This section investigates the potential reasons that SNNs fail to have matching results on SMC-Bench.

Original: [Figure]
Revised: [Figure]

Original: Pruning embedding layers or not? The pre-trained embeddings play a crucial role in the dense performance of SMC-Bench.
Revised: We conduct a thorough investigation into the reasons why most SNNs struggle to perform on SMCBench.

Original: For instance, the performance difference on ASDiv-A with and without RoBERTa’s pre-trained embeddings can be up to 21.4% as reported in Patel et al. To study whether the unsatisfactory performance of SNNs strongly correlates to the pruning of pre-trained embeddings, we conduct experiments where the embedding layers are kept dense.
Revised: Our analysis identifies two possible causes for the failure: (1) the “lazy regime” in LLMs, and (2) the specific model components that are pruned.

Original: We can see from Figure 5 while that pruning without embeddings unanimously brings large benefits to all approaches, it does not qualitatively change the fact that all these SOTA methods fail to match the dense performance at very naive sparsities. Moreover, since SNIP always aggressively prunes embedding layers, pruning without embeddings largely improves its performance (still behind duringand post-training pruning). Therefore, we believe the sparsification of embedding layers is not the root cause for the failure  of SNNs.
Revised: Based on these findings, we discover a set of stronger pruning recipes that alleviates (though still more or less suffering from) the failure on SMC-Bench, by breaking down the performance of the state-of-the-art BERT-pruning frameworkoBERT (Kurtic et al., 2022) on SMC-Bench (note that most models evaluated in this paper are also

Original: [Figure]
Revised: BERT-based).

Original: “w/o E” refers to pruning without embedding layers.
Revised: Due to the limited space, we present our full investigation in Appendix C, and briefly present our sanity check of layer collapse below.

Original: Does layer collapse occurs unexpectedly?
Revised: Does layer collapse occur unexpectedly on SMC-Bench?

Original: Layer collapse is another potential source of failure that blocks  information flow, leading to a catastrophic performance drop.
Revised: Layer collapse is the most common cause that blocks the information flow (signal propagation) of sparse neural networks, leading to a catastrophic performance drop (Tanaka et al., 2020).

Original: To check if layer collapse occurs unexpectedly, we plot the layerwise sparsity ratios of various sparse models in Appendix C.
Revised: We plot the layerwise sparsity ratios of various sparse models in Appendix C.1.

Original: The overall sparsity level is 36%, 67%, and 83%.

Original: We do not observe severe layer collapse across methods except for SNIP which specifically removes nearly entire embedding layers.
Revised: We do not observe severe layer collapse across methods except for SNIP which specifically removes nearly entire embedding layers.

Original: More importantly, we want to highlight a surprising observation: layerwise sparsities of different magnitude-based pruning approaches (i.e., IMP, OMP, and GMP) are extremely similar, all overlapped on one line, despite the significant performance gap  (up to 42.3%); small differences only start to appear until reaching very late layers (e.g., classification heads) (see Appendix C for more details).
Revised: However, we do observe an unexpected phenomenon: layerwise sparsities of different magnitude-based pruning approaches (i.e., IMP, OMP, and GMP) are extremely similar, all overlapped on one line, despite the significant performance gap among them (up to 42.3%); small differences only start to appear until reaching very deep layers (e.g., classification heads) (see Appendix C.1 for more details).

Original: This abnormal behavior indicates that the weight change before, during, and after fine-tuning is extremely small  and  all magnitude pruning approaches converge surprisingly close in the parameter space (Neyshabur et al., 2020).
Revised: This abnormal behavior is highly correlated with the “lazy regime” (Neyshabur et al., 2020; Malladi et al., 2022) where the model stays in the same basin during fine-tuning with fairly small weight changes, and hence all magnitudebased pruning approaches, before, during, and after fine-tuning, tend to collapse to the same solution.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Given the enormous breakthroughs and the fruitful results that sparse neural networks have achieved in numerous fields, it is necessary to rethink the sufficiency of current evaluation protocols and introduce more difficult and diverse benchmarks to explore the limitation of sparse neural networks.
Revised: Given the enormous breakthroughs and the fruitful results that sparse neural networks have achieved in numerous fields, it is necessary to rethink the sufficiency of current evaluation protocols and introduce more difficult and diverse benchmarks to explore the limitation of sparse neural networks.

Original: In this paper, we assemble a large-scale, challenging, and more diverse benchmark, SMC-Bench.
Revised: In this paper, we assemble a large-scale, challenging, and more diverse benchmark, SMC-Bench.

Original: After a careful evaluation across various leading sparsifications, we confirm that SMC-Bench is beyond the capability of current SOTA sparse algorithms.
Revised: Through our thorough evaluation across various leading sparsifications, we confirm that SMC-Bench notably challenges the capability of most magnitude- or/and gradient-based sparse algorithms.

Original: We further dig deeper into the behavior of SNNs, and observe several surprising phenomena that are absent in the current evaluation.
Revised: We further dig deeper into the behavior of SNNs, and observe several surprising phenomena that are absent in the current evaluation.

Original: However, we do not identify a root cause for the failureof SNNs on SMC-Bench. 
Revised: Our analysis points out that such failure is intimately related to the “lazy regime”, which leads us to a suite of strong pruning recipes that alleviates (yet still more or less suffering from) the failure on SMC-Bench. Our subsequent effort will focus on exploring stronger sparse training algorithms that can scale and generalize on SMC-Bench, and meanwhile will consider the training costs of different sparse algorithms for a more holistic picture of their efficiency benefits. A CKNOWLEDGEMENT

Original: In the hope to facilitate research endeavors to build next-generation sparse algorithms with  the potential to generalize on complex and practical tasks, we plan to open-source SMC-Bench.
Revised: We thank Dan Alistarh and Eldar Kurtic for the extremely helpful discussions about the implementation details of oBERT as well as our benchmark’s claims; and Zhangheng Li for helping run extra experiments with oBERT. S. Liu and Z. Wang are in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML). Part of this work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. NWO2021.060, EINF-2694 and EINF-2943/L

Original: A S UMMARY OF T ASKS , M ODELS , D ATASETS , AND T RAINING
Revised: A S UMMARY OF T ASKS , M ODELS , D ATASETS , AND T RAINING

Original: We summary the combinations of models and configurations that we used to evaluate SNNs on
Revised: We summarize the combinations of models and configurations that we used to evaluate SNNs on

Original: SMC-Bench.
Revised: SMC-Bench.

Original: [Table]
Revised: [Table]

Original: We strictly follow the training configurations reported in the original source to replicate the results of each task.
Revised: We strictly follow the training configurations reported in the original source to replicate the results of each task.

Original: The hyperparameters and configurations used for each model in this paper are shared below.
Revised: The hyperparameters and configurations used for each model in this paper are shared below.

Original: A. C OMMONSENSE R EASONING
Revised: A. C OMMONSENSE R EASONING

Original: [Table]
Revised: [Table]

Original: A. A RITHMETIC R EASONING
Revised: A. A RITHMETIC R EASONING

Original: [Table]
Revised: [Table]

Original: A. P ROTEIN T HERMOSTABILITY P REDICTION
Revised: A. P ROTEIN T HERMOSTABILITY P REDICTION

Original: [Table]
Revised: [Table]

Original: A. M ULTILINGUAL T RANSLATION
Revised: A. M ULTILINGUAL T RANSLATION

Original: [Table]
Revised: [Table]

Original: B R ESULTS OF A RITHMETIC R EASONING WITH M AGNITUDE BEFORE T RAINING
Revised: B R ESULTS OF A RITHMETIC R EASONING WITH M AGNITUDE BEFORE T RAINING

Original: In this appendix
Revised: In this appendix

Original: [Figure]
Revised: [Figure]

Original: we report the performance of SNNs on arithmetic reasoning including Magnitude before Training (OMP (Before)).
Revised: we report the performance of SNNs on arithmetic reasoning including Magnitude before Training (OMP (Before)).

Original: We can clearly observe that the accuracy of magnitude pruning before training dramatically falls from 80% to nearly 0% when sparsity is larger than 36%.
Revised: We can clearly observe that the accuracy of magnitude pruning before training dramatically falls from 80% to nearly 0% when sparsity is larger than 36%.

Original: After checking the corresponding layerwise sparsity, we find that OMP (Before) completely removes all the weights from non-embedding and non-encoder layers, leading to severe layer collapse
Revised: After checking the corresponding layerwise sparsity, we find that OMP (Before) completely removes all the weights from non-embedding and non-encoder layers, leading to severe layer collapse

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: C A N I NVESTIGATION OF W HY SNN S F AIL ON SMC-B ENCH

Revised: In this section, we conduct a full investigation, attempting to open the box for the potential causes of

Revised: SNN failures on SMC-Bench.

Revised: Our analysis reveals two possible causes: (1) the “lazy regime” in finetuning LLMs, and (2) the model components to prune.

Revised: Due to the “lazy regime” phenomenon (Chizat et al., 2019; Malladi et al., 2022), commonly used pruning techniques that rely on magnitude and gradient can be very uninformative.

Revised: Therefore, we turn to the latest strong second-order pruning framework - oBERT (Kurtic et al., 2022), which utilizes inverse-Hessian approximations to guide pruning decisions.

Revised: We choose RoBERTa.large on CSQA to conduct this investigation.

Revised: The roadmap of our full investigation is presented below.

Revised: • Layerwise sparsity ratios of various magnitude-based pruning methods are strikingly similar, suggesting that “lazy regime” may occur during fine-tuning.

Revised: In this regime, the most common pruning criteria such as magnitude and gradient can be rather unreliable.

Revised: • Second-order pruning approaches like oBERT provide more faithful signals than magnitudes and gradients for LLM pruning, and hence achieve significantly higher accuracy at high sparsities.

Original: C L AYERWISE S PARSITY R ATIOS 
Revised: C. L AYERWISE S PARSITY R ATIOS ON SMC-B ENCH

Original: In this appendix section, we plot the per-layer sparsity ratios discovered by each sparsification method at threeoverall sparsity levels: 36%, 64%, and 83%.
Revised: To check if severe layer collapse occurs on SMC-Bench, we plot the per-layer sparsity ratios discovered by various sparsification approaches at three sparsity levels: 36%, 64%, and 83%.

Original: Layers are ordered from input to output on the X axis.
Revised: Layers are ordered from input to output on the X-axis.

Original: We  report the layerwise sparsity of commonsense reasoning with RoBERTa on CSQA, WinoGrande, and RACE in Figure 7, 8, and 9, respectively, and arithmetic reasoning with GTS on SVAMP in Figure 10.
Revised: We respectively report the layerwise sparsity of commonsense reasoning with RoBERTa on CSQA and RACE in Figure 6 and 7, and arithmetic reasoning with GTS on SVAMP in Figure 8. We summarize our main findings here.

Original:  Layerwise sparsities of magnitude-based pruning approaches are extremely similar.
Revised: ❶ Layerwise sparsities of magnitude-based pruning approaches are extremely similar.

Original: IMP, OMP, and GMP that rely on weight magnitude for pruning share an extremely similar set of layerwise sparsities.
Revised: IMP, OMP, and GMP that rely on weight magnitude for pruning share an extremely similar set of layerwise sparsities.

Original: Especially, sparsity values of magnitude pruning on commonsense reasoning are completely identical, all overlapped on blue lines, except for the tiny difference in classification heads.
Revised: Especially, sparsity values of magnitude pruning on commonsense reasoning are completely identical, all overlapped on blue lines, except for the tiny difference in classification heads.

Original: This phenomenon indicates that weights of RoBERTa excluding classifiers remain rather stable during commonsense reasoning fine-tuning so that all the magnitude pruning variants (both before and after) discover the same sparsity pattern.
Revised: This phenomenon indicates that weights of RoBERTa excluding classifiers remain rather stable during commonsense reasoning fine-tuning so that all the magnitude pruning variants (both before and after) discover the same sparsity pattern.

Original: The sparsity difference of arithmetic reasoning is more distinguishable than commonsense reasoning.
Revised: The sparsity difference of arithmetic reasoning is more distinguishable than commonsense reasoning.

Original: Still, sparsities in the encoder (pre-trained RoBERTa) of IMP, GMP, and OMP (After) largely overlap.
Revised: Still, sparsities in the encoder (pre-trained RoBERTa) of IMP, GMP, and OMP (After) largely overlap.

Original: Until reaching the later layers, they start to identify their own sparsity patterns.
Revised: Until reaching the later layers, the sparsity ratios of different approaches start to be distinct.

Original:  SNIP tends toaggressively prune all the weights in embedding layers.
Revised: ❷ SNIP tends to prune all the weights in embedding layers aggressively.

Original: Even at the mild 36% sparsity, SNIP prunes weights of embedding layers to 99.7% sparsity, which may explain why SNIP struggles on SMC-Bench.
Revised: Even at the mild 36% sparsity, SNIP prunes weights of embedding layers to 99.7% sparsity, which may explain why SNIP struggles on SMC-Bench.

Original:  OMP (Before) suffers from layer collapse on arithmetic reasoning.
Revised: ❸ OMP (Before) suffers from layer collapse on arithmetic reasoning.

Original: We empirically find that OMP (Before) leads to completely empty late layers when sparsity is larger than 36%, indicating the limitation of only considering pruning with  magnitude before fine-tuning or training.
Revised: We empirically find that OMP (Before) leads to completely empty deep layers when sparsity is larger than 36%, indicating the limitation of only considering pruning with the magnitude before fine-tuning or re-training.

Original: L a y e r w i s e
Revised: The near-identical layerwise sparsity ratios across various magnitude-based methods remind us of the “lazy training” regime (Chizat et al., 2019; Malladi et al., 2022) which was revealed to occur during the fine-tuning of LLMs.

Original: Sp a r s i t y
Revised: Under this regime, weight changes during fine-tuning are negligible, hence non-informative and more “noisy”.

Original: RoBERTa
Revised: Consequently, various magnitude-based pruning approaches, regardless of their timing, all tend to converge to the same solution.

Original: [Table]
Revised: [Table]

Original: L a y e r w i s e Sp a r
Revised: [Table]

Original: s i
Revised: [Table]

Original: t y

Original: [Table]
Revised: [Table]

Original: e m b e d _ t o k e n s .
Revised: e m b e d _ t o k e n s .

Original: w e i g h t
Revised: w e i g h t

Original: e m b e d _ p o s i
Revised: e m b e d _ p o s i

Original: t i o n s .
Revised: t i o n s .

Original: w e i g h t
Revised: w e i g h t

Original: L . 1 .
Revised: L . 1 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 2 .
Revised: L . 2 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 3 .
Revised: L . 3 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 4 .
Revised: L . 4 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 5 .
Revised: L . 5 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 6 .
Revised: L . 6 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 7 .
Revised: L . 7 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 8 .
Revised: L . 8 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 9 .
Revised: L . 9 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 10 .
Revised: L . 10 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 11 .
Revised: L . 11 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 12 .
Revised: L . 12 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 13 .
Revised: L . 13 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 14 .
Revised: L . 14 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 15 .
Revised: L . 15 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 16 .
Revised: L . 16 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 17 .
Revised: L . 17 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 18 .
Revised: L . 18 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 19 .
Revised: L . 19 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 20 .
Revised: L . 20 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 21 .
Revised: L . 21 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 22 .
Revised: L . 22 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t c l a ss i f
Revised: o j _ w e i g h t c l a ss i f

Original: i c a t i o n _
Revised: i c a t i o n _

Original: h e a d s0. [Figure] [Figure] L a y e r w i s e
Revised: h e a d s

Original: [Figure]
Revised: [Figure]

Original:  ar s i t y 
Revised: L a y e r

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: e m b e d _ t o k e n s .

Original: w e i g h t

Original: e m b e d _ p o s i

Original: t i o n s .

Original: w e i g h t

Original: L . 1 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 2 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 3 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 4 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 5 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 6 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 7 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 8 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 9 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 10 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 11 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 12 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 13 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 14 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 15 .

Original: i n _ p r o j _ w e i g h t L . 16 . i n _ p r
Revised: i s

Original: o j _ w e i g h t

Original: L . 17 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 18 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 19 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 20 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 21 .

Original: i n _ p r

Original: o j _ w e i g h t

Original: L . 22 .

Original: i n _ p r

Original: o j _ w e i g h

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: L a y e r w i s e

Original: [Figure]
Revised: [Figure]

Original: a r s it y [Figure] [Table] [Table]
Revised: i

Original: [Table]

Original: L a y e r w i s e

Original: [Table]
Revised: [Figure]

Original: a
Revised: y

Original: [Table]
Revised: [Figure]

Original: i t y

Original: [Table]
Revised: [Figure]

Original: L a y e r w i s e

Original: Sp a r s i t y
Revised: y

Original: [Table]
Revised: [Figure]

Original: e m b e d _ t o k e n s .
Revised: m b e d _ t o k e n s .

Original: w e i g h t
Revised: w e i g h t

Original: e m b e d _ p o s i
Revised: e m b e d _ p o s i

Original: t i o n s .
Revised: t i o n s .

Original: w e i g h t
Revised: w e i g h t

Original: L . 1 .
Revised: L . 1 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 2 .
Revised: L . 2 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 3 .
Revised: L . 3 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 4 .
Revised: L . 4 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 5 .
Revised: L . 5 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 6 .
Revised: L . 6 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 7 .
Revised: L . 7 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 8 .
Revised: L . 8 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 9 .
Revised: L . 9 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 10 .
Revised: L . 10 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 11 .
Revised: L . 11 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 12 .
Revised: L . 12 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 13 .
Revised: L . 13 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 14 .
Revised: L . 14 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 15 .
Revised: L . 15 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 16 .
Revised: L . 16 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 17 .
Revised: L . 17 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 18 .
Revised: L . 18 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 19 .
Revised: L . 19 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 20 .
Revised: L . 20 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 21 .
Revised: L . 21 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g h t
Revised: o j _ w e i g h t

Original: L . 22 .
Revised: L . 22 .

Original: i n _ p r
Revised: i n _ p r

Original: o j _ w e i g ht
Revised: o j _ w e i g h

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: L a y e r w i s e
Revised: [Table]

Original: [Figure]
Revised: [Table]

Original: a r s i t y

Original: [Figure] [Figure]
Revised: [Table] [Table]

Original: m b e dd i n g s. wo r d _ e mb e dd 
Revised: e dd i n g s w _ e m dd i n g s w i

Original: i n g s . w e i gh t
Revised: g

Original:  e m b e dd i n g s .
Revised: h e m b e dd i n g s .

Original: p o s i
Revised: p o s i

Original: t i o n _
Revised: t i o n _

Original: e m be dd i n g s .
Revised: e m b dd i n g s w e

Original: w e i g h t
Revised: i g h t

Original: e m b e dd i n g s .
Revised: e m b e dd i n g s .

Original: t o k e n _
Revised: t o k e n _

Original: t
Revised: t

Original: y p
Revised: y p

Original: e _

Original: e m be dd i n g s. w e i g h t e n c o d e r .
Revised: e m b dd i n g s w e i g h t e n c o d e r .

Original: l a y e r . 0 .
Revised: l a y e r . 0 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 0 .
Revised: l a y e r . 0 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Revised: w e i g h t e n c o d e r .

Revised: l a y e r . 0 .

Revised: a tt e n t i o n .

Original:  w e i g h t e n c o d e r .
Revised: s e l f . v a l u e . w e i g h t e n c o d e r .

Original: l a y e r . 0 .
Revised: l a y e r . 0 .

Original: a tt e n t i o n . 
Revised: a tt e n t i o n . o u t p u t . d e n s e . w e i g h t e n c o d e r .

Original: s

Original: e l f .

Original: v a l u e .

Original: w e i g h t e n c o d e r .

Original: l a y e r . 0 .
Revised: l a y e r . 0 .

Original: a tt e n t io n. o u tp u t . d e n s e . w e  i g h t en c o d e r .
Revised: i n t e r m e d i a t e .

Original: l a y e r . 0 .

Original: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 0 .
Revised: l a y e r . 0 .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 1 .
Revised: l a y e r . 1 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 1 .
Revised: l a y e r . 1 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 1 .
Revised: l a y e r . 1 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 1 .
Revised: l a y e r . 1 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 2 .
Revised: l a y e r . 2 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 2 .
Revised: l a y e r . 2 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 2 .
Revised: l a y e r . 2 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 2 .
Revised: l a y e r . 2 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 3 .
Revised: l a y e r . 3 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 3 .
Revised: l a y e r . 3 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 3 .
Revised: l a y e r . 3 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 3 .
Revised: l a y e r . 3 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 4 .
Revised: l a y e r . 4 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 4 .
Revised: l a y e r . 4 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 4 .
Revised: l a y e r . 4 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 4 .
Revised: l a y e r . 4 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 4 .
Revised: l a y e r . 4 .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 4 .
Revised: l a y e r . 4 .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 6 .
Revised: l a y e r . 6 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 6 .
Revised: l a y e r . 6 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 6 .
Revised: l a y e r . 6 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 6 .
Revised: l a y e r . 6 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 8 .
Revised: l a y e r . 8 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 8 .
Revised: l a y e r . 8 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Revised: w e i g h t e n c o d e r .

Revised: l a y e r . 8 .

Revised: a tt e n t i o n .

Original:  w e i g h t e n c o d e r .
Revised: s e l f . v a l u e . w e i g h t e n c o d e r .

Original: l a y e r . 8 .
Revised: l a y e r . 8 .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: o u t p u t .

Original: e l f . v a l u e . w e i g h t e n c o d e r . l a y er . 8 .
Revised: d e n s e .

Original: a tt e n t i o n . o u t p u t . d e n s e . w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 8 .
Revised: l a y e r . 8 .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r . 8 .
Revised: l a y e r . 8 .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Revised: v a l u e .

Revised: w e i g h t e n c o d e r .

Revised: l a y e r .

Revised: a tt e n t i o n .

Revised: o u t p u t .

Revised: d e n s e .

Revised: w e i g h t e n c o d e r .

Revised: l a y e r .

Revised: i n t e r m e d i a t e .

Revised: d e n s e .

Revised: w e i g h t e n c o d e r .

Revised: l a y e r .

Revised: o u t p u t .

Revised: d e n s e .

Revised: w e i g h t e n c o d e r .

Revised: l a y e r .

Original: v a l u e . w e i g h t e n c o d e r  .
Revised: a tt e n t i o n . s e l f . q u e r y .

Original: l a y e r .

Original: a tt e n t i o n .

Original: o u t p u t .

Original: d e n s e . w e i g h t e n c o d e r .l a y e r . i n t e r m e d i a t e .
Revised: w e i g h t e n c o d e r .

Original: d e n s e .

Original: w e i g h t e n c o d e r .

Original: l a y e r .
Revised: l a y e r .

Original: o u t p u t . d e n s e . w e i g h t e n c o d e r . l a y e r .. a tt e n t i o n .
Revised: a tt e n t i o n .

Original: s e l f .

Original: q u e r y .

Original: w e i g h t e n c o d e r .

Original: l a y e r .. a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .. a tt e n t i o n .
Revised: l a y e r . a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .. a tt e n t i o n .
Revised: l a y e r . 1 0 . a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r ..
Revised: l a y e r .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .. o u t p u t .
Revised: l a y e r . o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .. a tt e n t i o n .
Revised: l a y e r . a tt e n t i o n .

Original: s e l f .
Revised: s e l f .

Original: q u e r y .
Revised: q u e r y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .. a tt e n t i o n .
Revised: l a y e r . a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: k e y .
Revised: k e y .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .. a tt e n t i o n .
Revised: l a y e r . a tt e n t i o n .

Original: s
Revised: s

Original: e l f .
Revised: e l f .

Original: v a l u e .
Revised: v a l u e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .. a tt e n t i o n .
Revised: l a y e r . a tt e n t i o n .

Original: o u t p u t .
Revised: o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r ..
Revised: l a y e r .

Original: i n t e r m e d i a t e .
Revised: i n t e r m e d i a t e .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t e n c o d e r .
Revised: w e i g h t e n c o d e r .

Original: l a y e r .. o u t p u t .
Revised: l a y e r . o u t p u t .

Original: d e n s e .
Revised: d e n s e .

Original: w e i g h t r nn .
Revised: w e i g h t r nn .

Original: w e i g h t
Revised: w e i g h t

Original: i h _ lr nn .
Revised: i h _ l 0 r nn .

Original: w e i g h t
Revised: w e i g h t

Original: hh _ lr nn .
Revised: hh _ lr nn .

Original: w e i g h t
Revised: w e i g h t

Original: i h _ l 0 _ r e v e r s e r nn .
Revised: i h _ l 0 _ r e v e r s e r nn .

Original: w e i g h t
Revised: w e i g h t

Original: hh _ l_ r e v e r s e r nn .
Revised: hh _ l 0 _ r e v e r s e r nn .

Original: w e i g h t
Revised: w e i g h t

Original: i h _ lr nn .
Revised: i h _ l 1 r nn .

Original: w e i g h t
Revised: w e i g h t

Original: hh _ lr nn .
Revised: hh _ lr nn .

Original: w e i g h t
Revised: w e i g h t

Original: i h _ l 1 _ r e v e r s e r nn .
Revised: i h _ l 1 _ r e v e r s e r nn .

Original: w e i g h t
Revised: w e i g h t

Original: hh _ l_ r e v e r s e c
Revised: hh _ l 1 _ r e v e r s e c

Original: o n c a t _ l .
Revised: o n c a t _ l .

Original: w e i g h t c
Revised: w e i g h t c

Original: o n c a t _ r .
Revised: o n c a t _ r .

Original: w e i g h t c
Revised: w e i g h t c

Original: o n c a t _ l g .
Revised: o n c a t _ l g .

Original: w e i g h t c
Revised: w e i g h t c

Original: o n c a t _ r g .
Revised: o n c a t _ r g .

Original: w e i g h t
Revised: w e i g h t

Original: o p s .
Revised: o p s .

Original: w e i g h t a tt n .
Revised: w e i g h t a tt n .

Original: a tt n .
Revised: a tt n .

Original: w e i g h t a tt n .
Revised: w e i g h t a tt n .

Original: s c o r e .
Revised: s c o r e .

Original: w e i g h t s c o r e .
Revised: w e i g h t s c o r e .

Original: a tt n .
Revised: a tt n .

Original: w e i g h t s c o r e .
Revised: w e i g h t s c o r e .

Original: s c o r e .
Revised: s c o r e .

Original: w e i g h t
Revised: w e i g h t

Original: e m b e dd i n g s .
Revised: e m b e dd i n g s .

Original: w e i g h t g e n e r a t e _ l . w e i g h t g e n e r a t e _ r .
Revised: w e i g h t g e n e r a t e _ l . w e i g h t g e n e r a t e _ r .

Original: w e i g h t g e n e r a t e _ l g . w e i g h t g e n e r a t e _ r g .
Revised: w e i g h t g e n e r a t e _ l g . w e i g h t g e n e r a t e _ r g .

Original: w e i g h t  0.
Revised: w e i g h t 0.0 0.

Original: L a y e r w i s e 
Revised: L a y e r w i s e Sp a r s i t

Original: [Figure]
Revised: [Figure]

Revised: C. E VALUATION OF O BERT ON SMC-B ENCH

Revised: Based on our conjecture that magtitudes/gradients become unreliable for pruning LLMs, we hypothesize that the second-order pruning approaches with approximated Hessian matrix would be more accurate options.

Revised: To verify our conjecture, we turn to the latest stronger second-order pruning framework, oBERT (Kurtic et al., 2022).

Original:  a r s i t y
Revised: Specifically, we follow Kurtic et al. and replace the magnitude pruning criterion of the LTH framework with the second-order oBERT criterion; adopt Learning Rate Rewinding (LRR) (Renda et al., 2020) and Knowledge Distillation (KD) (Hinton et al., 2015) during each pruning iteration; and keep the embeddings and classification heads dense. Figure 9 support our hypothesis and demonstrates that oBERT notably outperforms the zero-order and first-order sparsification approaches, and substantially improves the accuracy to a competitive level.

Revised: More importantly, oBERT produces a completely different layerwise sparsity pattern from magnitudebased pruning approaches, which is consistent with the patterns that are commonly observed in sparse computer vision models: deeper layers tend to have higher sparsities than lower layers (Evci et al., 2020; Kusupati et al., 2020; Tanaka et al., 2020; Liu et al., 2021b).

Original: [Figure]
Revised: [Figure]

Revised: So far, our investigation has discovered that the pruning recipe used in (Kurtic et al., 2022) can remarkably improve the LLM pruning performance on SMC-Bench.

Original: at sparsity levels ∈ [36%, 67%, 83%].
Revised: Nevertheless, such a well-tuned pruning recipe is both time- and resource-intensive (9 × more fine-tuning time, besides Hessian matrix [Table] t L . 22 . f c

Revised: [Figure]

Revised: w e i g h

Revised: [Figure]

Revised: g

Revised: [Figure]

Revised: p r o

Revised: [Figure]

Revised: w e i g h t

Revised: [Figure]

Original: D S UMMARY OF E VALUATION T ASKS AND D ATASETS IN 100 P APERS
Revised: s e l f _ a tt n . q _ p r o j . w e i g h t L . 23 . s

Revised: e l f _ a tt n .

Revised: o u t _ p r o j .

Revised: w e i g h t L . 23 .

Revised: f c 1 .

Revised: w e i g h t L . 23 .

Revised: f c 2 .

Revised: w e i g h

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Figure]

Original: E L IMITATION
Revised: L a [Figure]

Original: The main goal of our benchmark is not to provide an apple-to-apple comparison among different sparse algorithms in terms of training efficiency, but to highlight the limitations of the commonly-used evaluation protocols and assemble a new  challenging one. Hence, the training costs of different sparsification approaches evaluated in this paper vary a lot from prior-training to post-training.
Revised: approximation); and even so, the strongest SNNs still fall short of their dense counterpart by around 10% accuracy at sparsities between 60% − 80% , in contrast to “normal” SNNs that easily match their dense models on CIFAR, ImageNet, or GLUE. Therefore, the main claim of our paper still holds, that is, SMC-Bench indeed provides a new benchmark that is way more challenging for SOTA sparse algorithms, than existing testbeds. D S UMMARY OF E VALUATION T ASKS AND D ATASETS IN 100 P APERS

Original: Practical designs would need to consider the training efficiency as well, but it is out of the scope of our current work.
Revised: [Table]