Original: T HE U NREASONABLE E FFECTIVENESS OF R ANDOM P RUNING : R ETURN OF THE M OST N AIVE B ASELINE FOR S PARSE T RAINING
Revised: T HE U NREASONABLE E FFECTIVENESS OF R ANDOM P RUNING : R ETURN OF THE M OST N AIVE B ASELINE FOR S PARSE T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Random pruning is arguably the “ﬁrst child” of neural network pruning, and it has been thought by many as the most naive and uncompetitive  pruning baseline.
Revised: Random pruning is arguably the most naive way to attain sparsity in neural networks, but has been deemed uncompetitive by either post-training pruning or sparse training.

Original: In this paper, we highlight a perhaps counter-intuitive ﬁnding: random pruning  can be quite powerful for the sparse training of modern neural networks .
Revised: In this paper, we focus on sparse training and highly a perhaps counterintuitive ﬁnding, that random pruning at initialization (PaI) can be quite powerful for the sparse training of modern neural networks.

Original: Without any delicate pruning criteria or carefully pursued sparsity structures, we empirically demonstrate that  training a randomly prunedsparse network from scratch can match the performance of its dense equivalent.
Revised: Without any delicate pruning criteria or carefully pursued sparsity structures, we empirically demonstrate that sparsely training a randomly pruned network from scratch can match the performance of its dense equivalent.

Original: There are two key factors that contribute to this revival: (i) the network sizes matter : as the original dense networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios; (ii) appropriate layer-wise sparsity ratios can be pre-chosen for random pruning, which shows to be another important performance booster.
Revised: There are two key factors that contribute to this revival: (i) the network sizes matter : as the original dense networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios; (ii) appropriate layer-wise sparsity ratios can be pre-chosen for sparse training, which shows to be another important performance booster.

Original: Simple as it looks, a randomly pruned subnetwork of Wide ResNet-50 can be  trained to outperforming a dense  ResNet-50, on ImageNet.
Revised: Simple as it looks, a randomly pruned subnetwork of Wide ResNet-50 can be sparsely trained to outperforming a dense Wide ResNet-50, on ImageNet.

Original: We also observed such randomly pruned networks outperform dense counterparts in other favorable aspects, such as out-of-distribution detection, uncertainty estimation, and adversarial robustness.
Revised: We also observed such randomly pruned networks outperform dense counterparts in other favorable aspects, such as out-ofdistribution detection, uncertainty estimation, and adversarial robustness.

Original: Overall, our results strongly suggest there is larger-than-expected room for sparse training at scale, and the beneﬁts of sparsity might be more universal beyond carefully designed pruning.
Revised: Overall, our results strongly suggest there is larger-than-expected room for sparse training at scale, and the beneﬁts of sparsity might be more universal beyond carefully designed pruning.

Original: We have included our codes as supplementary ﬁles.
Revised: We have included our codes as supplementary ﬁles.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Most recent breakthroughs in deep learning are fairly achieved with the increased complexity of over-parameterized networks (Brown et al., 2020; Raffel et al., 2019; Dosovitskiy et al., 2020; Fedus et al., 2021; Jumper et al., 2021; Berner et al., 2019).
Revised: Most recent breakthroughs in deep learning are fairly achieved with the increased complexity of over-parameterized networks (Brown et al., 2020; Raffel et al., 2019; Dosovitskiy et al., 2020; Fedus et al., 2021; Jumper et al., 2021; Berner et al., 2019).

Original: It is well-known that large models train better (Neyshabur et al., 2018; Novak et al., 2018; Allen-Zhu et al., 2019), generalize better (Hendrycks & Dietterich, 2019; Xie & Yuille, 2019; Zhao et al., 2017), and transfer better (Chen et al., 2020; 2021).
Revised: It is well-known that large models train better (Neyshabur et al., 2018; Novak et al., 2018; Allen-Zhu et al., 2019), generalize better (Hendrycks & Dietterich, 2019; Xie & Yuille, 2019; Zhao et al., 2017), and transfer better (Chen et al., 2020; 2021).

Original: However, the upsurge in the investment of large models exacerbates the gap between research and real-life applications since many practical applications demand compact and efﬁcient networks.
Revised: However, the upsurge in the investment of large models exacerbates the gap between research and real-life applications since many practical applications demand compact and efﬁcient networks.

Original: Neural network pruning, since proposed by (Mozer & Smolensky, 1989;
Revised: Neural network pruning, since proposed by (Mozer & Smolensky, 1989;

Original: Janowsky, 1989), has evolved as the most common technique in literature to reduce the computational and memory requirements of neural networks.
Revised: Janowsky, 1989), has evolved as the most common technique in literature to reduce the computational and memory requirements of neural networks.

Original: Over the past few years, numerous pruning criteria have been proposed, including magnitude (Mozer & Smolensky, 1989; Han et al., 2015; Frankle & Carbin, 2018; Mocanu et al., 2018), Hessian (LeCun et al., 1990; Hassibi & Stork, 1993), mutual information (Dai et al., 2018),
Revised: Over the past few years, numerous pruning criteria have been proposed, including magnitude (Mozer & Smolensky, 1989; Han et al., 2015; Frankle & Carbin, 2018; Mocanu et al., 2018), Hessian (LeCun et al., 1990; Hassibi & Stork, 1993), mutual information (Dai et al., 2018),

Original: Taylor expansion (Molchanov et al., 2016), movement (Sanh et al., 2020), connection sensitivity (Lee et al., 2018), etc.
Revised: Taylor expansion (Molchanov et al., 2016), movement (Sanh et al., 2020), connection sensitivity (Lee et al., 2019), etc.

Original: Motivated for different scenarios, pruning can occur after training (Han et al., 2015; Frankle & Carbin, 2018;
Revised: Motivated for different scenarios, pruning can occur after training (Han et al., 2015; Frankle & Carbin, 2018;

Original: Molchanov et al., 2016; Lee et al., 2021), during training (Zhu &
Revised: Molchanov et al., 2016; Lee et al., 2021), during training (Zhu &

Original: Gupta, 2017; Gale et al., 2019; Louizos et al., 2018), and even before training (Mocanu et al., 2018;
Revised: Gupta, 2017; Gale et al., 2019; Louizos et al., 2018), and even before training (Mocanu et al., 2018;

Original: Lee et al., 2018; Gale et al., 2019).
Revised: Lee et al., 2019; Gale et al., 2019).

Original: The last regime can be further categorized into “static sparse training” (Mocanu et al., 2016;
Revised: The last regime can be further categorized into “static sparse training” (Mocanu et al., 2016;

Original: Gale et al., 2019; Lee et al., 2018; Wang et al., 2020) and “dynamic sparse training” (Mocanu et al., 2018; Bellec et al., 2018;
Revised: Gale et al., 2019; Lee et al., 2019; Wang et al., 2020) and “dynamic sparse training” (Mocanu et al., 2018; Bellec et al., 2018;

Original: Evci et al., 2020a; Liu et al., 2021).
Revised: Evci et al., 2020a; Liu et al., 2021).

Revised: While random pruning is a universal method that can happen at any stage of training, training a randomly pruned network from scratch is arguably the most appealing way, owing to its “end-to-end” saving potential for the entire training process besides the inference.

Revised: Due to this reason, we focus on random pruning at initialization (or sparse training) in this paper.

Original: When new pruning approaches bloom, random pruning naturally becomes their performance’s empirical “lower bound” since the connections are randomly chosen without any good reasoning
Revised: When new pruning approaches bloom, random pruning naturally becomes their performance’s empirical “lower bound” since the connections are randomly chosen without any good reasoning.

Original: Likely due to the same reason, the results of random pruning  (as “easy to beat” baselines to support fancier new pruning methods) reported in the literature are unfortunately vague, often inconsistent, and sometimes casual.
Revised: Likely due to the same reason, the results of random pruning at initialization (as “easy to beat” baselines to support fancier new pruning methods) reported in the literature are unfortunately vague, often inconsistent, and sometimes casual.

Original: For instance, it is found in Liu et al.
Revised: For instance, it is found in Liu et al.

Original: (2020b) that randomly pruned sparse networks can  match the full accuracy of dense networks with only 20% parameters, whereas around 80% parameters are required to do so in Frankle et al.(2020b).
Revised: (2020b) that randomly pruned sparse networks can be trained from scratch to match the full accuracy of dense networks with only 20% parameters, whereas around 80% parameters are required to do so in Frankle et al.

Original: The differences may arise from architecture choices, training recipes, distribution hyperparameters/layer-wise ratios, and so on.
Revised: The differences may arise from architecture choices, training recipes, distribution hyperparameters/layer-wise ratios, and so on.

Original: On the other hand, while random pruning is a universal method that can happen at any stage of training, training a randomly pruned network from scratch is arguably the most appealing way, due to its “end-to-end” saving potential for the entire training process besides the inference.

Original: However, to the best of our knowledge, no study has systematically examined random pruning for sparse training from scratch, and no prior art has ever found it can yield close to competitive results compared to more well-versed pruning approaches such as (Frankle & Carbin, 2018; Lee et al., 2018; Wang et al., 2020; Tanaka et al., 2020).

Original: In this paper, we attempt to show an afﬁrmative answer to this question, by conducting a comprehensive study on training a randomly pruned subnetwork from scratch, in the hope of providing a better understanding on this underrated baseline.

Original: In most pruning literature (Gale et al., 2019; Lee et al., 2018; Frankle et al., 2020b; Tanaka et al., 2020), random pruning usually refers to randomly removing the same proportion of parameters per layer, ending up with uniform layer-wise sparsities.
Revised: In most pruning literature (Gale et al., 2019; Lee et al., 2019; Frankle et al., 2021; Tanaka et al., 2020), random pruning usually refers to randomly removing the same proportion of parameters per layer, ending up with uniform layer-wise sparsities.

Original: Nevertheless, researchers have explored other pre-deﬁned layer-wise sparsities, e.g., uniform + (Gale et al., 2019), Erd˝os-R´enyi random graph (ER) (Mocanu et al., 2018), and Erd˝os-R´enyi-Kernel (ERK)
Revised: Nevertheless, researchers have explored other pre-deﬁned layer-wise sparsities, e.g., uniform + (Gale et al., 2019), Erd˝os-R´enyi random graph (ER) (Mocanu et al., 2018), and Erd˝os-R´enyi-Kernel (ERK)

Original: (Evci et al., 2020a).
Revised: (Evci et al., 2020a).

Original: These layer-wise sparsities naturally ﬁt the category of random pruning, as they require no training to obtain the corresponding sparsity ratios (followed by sampling randomly).
Revised: These layerwise sparsities naturally ﬁt the category of random pruning, as they require no training to obtain the corresponding sparsity ratios (followed by sampling randomly).

Original: We assess random pruning  with these layer-wise sparsity ratios, in terms of various perspectives besides the predictive accuracy.
Revised: We assess random pruning at initialization with these layer-wise sparsity ratios, in terms of various perspectives besides the predictive accuracy.

Original: Our main ﬁndings during this course of study are summarized below:
Revised: Our main ﬁndings during this course of study are summarized below:

Original: We ﬁnd that the network size matters for the effectiveness of random pruning.
Revised: We ﬁnd that the network size matters for the effectiveness of random pruning at initialization.

Original: With small networks, random pruning  hardly matches the full accuracy even at mild sparsities (10%, 20%).
Revised: With small networks, random pruning at initialization hardly matches the full accuracy even at mild sparsities (10%, 20%).

Original: However, as the networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios.
Revised: However, as the networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios.

Original: • We further identify that appropriate layer-wise sparsity ratios can be an important booster for the performance of random pruning.
Revised: • We further identify that appropriate layer-wise sparsity ratios can be an important booster for the performance of random pruning at initialization.

Original: We investigate several options to pre-deﬁne layerwise sparsity ratios before any training; one of them is able to push the performance of a completely random sparse Wide ResNet-50 over the densely trained ResNet-50 on ImageNet.
Revised: We investigate several options to pre-deﬁne layer-wise sparsity ratios before any training; one of them is able to push the performance of a completely random sparse Wide ResNet-50 over the densely trained ResNet-50 on ImageNet.

Original: • We systematically assess the performance of random pruning  and are somehow surprised by its accuracy and robustness.
Revised: • We systematically assess the performance of random pruning at initialization and are somehow surprised by its accuracy and robustness.

Original: The accuracy achieved by ERK ratio can even surpass the ones learned by complex criteria, e
Revised: The accuracy achieved by ERK ratio can even surpass the ones learned by complex criteria, e

Original: g., SNIP, GraSP.
Revised: g., SNIP, GraSP.

Original: In addition, randomly pruned sparse networks are found to outperform dense networks in other favorable aspects, such as out-of-distribution (OoD) detection, adversarial robustness, and uncertainty estimation.
Revised: In addition, randomly pruned sparse networks are found to outperform dense networks in other favorable aspects, such as out-of-distribution (OoD) detection, adversarial robustness, and uncertainty estimation.

Original:  R ELATED W ORK 2. S TATIC S PARSE T RAINING
Revised:  R ELATED W ORK 2. S TATIC S PARSE T RAINING

Original: Static sparse training represents a class of methods that aim to train a sparse subnetwork with a ﬁxed sparse connectivity pattern during the course of training.
Revised: Static sparse training represents a class of methods that aim to train a sparse subnetwork with a ﬁxed sparse connectivity pattern during the course of training.

Original: We divide the static sparse training into random pruning and non-random pruning according to whether the connection is randomly selected.
Revised: We divide the static sparse training into random pruning and non-random pruning according to whether the connection is randomly selected.

Original: Random Pruning.
Revised: Random Pruning.

Original: Static sparse training with random pruning samples masks within each layer in a random fashion based on pre-deﬁned layer-wise sparsities.
Revised: Static sparse training with random pruning samples masks within each layer in a random fashion based on pre-deﬁned layer-wise sparsities.

Original: The most naive approach is pruning each layer uniformly with the same pruning ratio, i.e., uniform pruning (Mariet & Sra, 2015;
Revised: The most naive approach is pruning each layer uniformly with the same pruning ratio, i.e., uniform pruning (Mariet & Sra, 2015;

Original: He et al., 2017; Suau et al., 2019; Gale et al., 2019).
Revised: He et al., 2017; Suau et al., 2019; Gale et al., 2019).

Original: Mocanu et al.
Revised: Mocanu et al.

Original: (2016) proposed a non-uniform and scale-free topology, showing better performance than the dense counterpart when applied to restricted
Revised: (2016) proposed a non-uniform and scale-free topology, showing better performance than the dense counterpart when applied to restricted

Original: Boltzmann machines (RBMs).
Revised: Boltzmann machines (RBMs).

Original: Later, expander graphs were introduced to build sparse CNNs and showed comparable performance against the corresponding dense CNNs (Prabhu et al., 2018; Kepner & Robinett, 2019).
Revised: Later, expander graphs were introduced to build sparse CNNs and showed comparable performance against the corresponding dense CNNs (Prabhu et al., 2018; Kepner & Robinett, 2019).

Original: While not initially designed for static sparse training, ER (Mocanu et al., 2018) 
Revised: While not initially designed for static sparse training, ER (Mocanu et al., 2018) and ERK (Evci et al., 2020a) are two advanced layer-wise sparsities introduced from the ﬁeld of graph theory with strong results.

Original: Non-Random Pruning.
Revised: Non-Random Pruning.

Original: Instead of pre-choosing a sparsity ratio for each layer, many works utilize the proposed saliency criteria to learn the layer-wise sparsity ratios before training.
Revised: Instead of pre-choosing a sparsity ratio for each layer, many works utilize the proposed saliency criteria to learn the layer-wise sparsity ratios before training, also termed as pruning at initialization (PaI).

Original: Lee et al.
Revised: Lee et al.

Original:  ﬁrst introduced SNIP that chooses structurally important connections at initialization via the proposed connection sensitivity.
Revised: (2019) ﬁrst introduced SNIP that chooses structurally important connections at initialization via the proposed connection sensitivity.

Original: Following SNIP, many efﬁcient criteria have been proposed to improve the performance of non-random pruning at initialization, including but not limited to gradient ﬂow (Wang et al., 2020), synaptic strengths (Tanaka et al., 2020), neural tangent kernel (Liu & Zenke, 2020), and iterative SNIP (de Jorge et al., 2020; Verdenius et al., 2020). 
Revised: Following SNIP, many efﬁcient criteria have been proposed to improve the performance of non-random pruning at initialization, including but not limited to gradient ﬂow (Wang et al., 2020), synaptic strengths (Tanaka et al., 2020), neural tangent kernel (Liu & Zenke, 2020), and iterative SNIP (de Jorge et al., 2020; Verdenius et al., 2020). The most closely related works of our work are Su et al. (2020) and Frankle et al.

Revised: Su et al.

Original: 2. D YNAMIC S PARSE T RAINING
Revised: (2020) proposed several sanity-check and showed that SNIP, GraSP, and Lottery Tickets (LT)hardly exploit any information from the training data and very robust to the mask shufﬂing. They further show that random pruning at initialization with hand-designed layer-wise sparsity ratios outperforms or attains a similar performance compared with PaI methods. Frankle et al. (2021) further conﬁrmed that what PaI discovers are the layerwise proportions rather than the speciﬁc weights or values, whereas magnitude pruning after training learns both, reﬂecting a broader challenge inherent to pruning at initialization.

Revised: While prior works have observed that random pruning can be more competitive in certain cases, they did not give principled guidelines on when and how it can become that good; nor do they show it can match the performance of dense networks on ImageNet.

Revised: Standing on the shoulders of giants, our work summarized principles by more rigorous studies, and demonstrate the strongest result so far, that randomly pruned sparse Wide ResNet-50 can be sparsely trained to outperform a dense Wide ResNet-50, on ImageNet.

Revised: Moreover, compared with the ad-hoc sparsity ratios used in (Su et al., 2020), we show that ERK (Evci et al., 2020a) and our modiﬁed ERK+ are more general sparsity ratios that consistently demonstrate competitive performance without careful layer-wise sparsity design for every architecture.

Revised: Speciﬁcally, ERK+ ratio achieves similar accuracy with the dense Wide ResNet-50 on ImageNet while being data free, feedforward free, and dense initialization free.

Original: In contrast to static sparse training, dynamic sparse training stems from randomly initialized sparse subnetworks, and meanwhile dynamically explores new sparse connectivity during training.
Revised: In contrast to static sparse training, dynamic sparse training stems from randomly initialized sparse subnetworks, and meanwhile dynamically explores new sparse connectivity during training.

Original: Dynamic sparse training starts from Sparse Evolutionary Training (SET) (Mocanu et al., 2018; Liu et al., 2020a) which initializes the sparse connectivity with Erd˝os-R´enyi (Erd˝os & R´enyi, 1959) topology and periodically explores the sparse connectivity via a prune-and-grow scheme during the course of training.
Revised: Dynamic sparse training starts from Sparse Evolutionary Training (SET) (Mocanu et al., 2018; Liu et al., 2020a) which initializes the sparse connectivity with Erd˝os-R´enyi (Erd ˝ os & R ´ enyi, 1959) topology and periodically explores the sparse connectivity via a prune-and-grow scheme during the course of training.

Original: While there exist numerous pruning criteria in the literature, simple magnitude pruning typically performs well in the ﬁeld of dynamic sparse training.
Revised: While there exist numerous pruning criteria in the literature, simple magnitude pruning typically performs well in the ﬁeld of dynamic sparse training.

Original: On the other hand, the criteria used to grow weights back differs from method to method, including randomness (Mocanu et al., 2018;
Revised: On the other hand, the criteria used to grow weights back differs from method to method, including randomness (Mocanu et al., 2018;

Original: Mostafa & Wang, 2019), momentum (Dettmers & Zettlemoyer, 2019), gradient (Evci et al., 2020a;
Revised: Mostafa & Wang, 2019), momentum (Dettmers & Zettlemoyer, 2019), gradient (Evci et al., 2020a;

Original: Jayakumar et al., 2020; Liu et al., 2021).
Revised: Jayakumar et al., 2020; Liu et al., 2021).

Original: Besides the prune-and-grow scheme, layer-wise sparsities are vital to achieving high accuracy.
Revised: Besides the prune-and-grow scheme, layer-wise sparsities are vital to achieving high accuracy.

Original: (Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019) reallocates weights across layers during training based on reasonable heuristics, demonstrating performance improvement.
Revised: (Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019) reallocates weights across layers during training based on reasonable heuristics, demonstrating performance improvement.

Original: Evci et al.
Revised: Evci et al.

Original: (2020a) extended ER to CNNs and showed considerable performance gains to sparse CNN training with the Erd˝os-R´enyi-Kernel (ERK) ratio.
Revised: (2020a) extended ER to CNNs and showed considerable performance gains to sparse CNN training with the Erd ˝ os-R ´ enyi-Kernel (ERK) ratio.

Original: Even though dynamic sparse training achieves promising sparse training performance, it changes the sparse connectivity during training and is thus out of the scope of random pruning.
Revised: Even though dynamic sparse training achieves promising sparse training performance, it changes the sparse connectivity during training and is thus out of the scope of random pruning.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We conduct extensive experiments to systematically evaluate the performance of random pruning.
Revised: We conduct extensive experiments to systematically evaluate the performance of random pruning.

Original: The experimental settings are described below.
Revised: The experimental settings are described below.

Original: 3. L AYER - WISE S PARSITIES R ATIOS
Revised: 3. L AYER - WISE S PARSITIES R ATIOS

Original: Denote s l as the sparsity of layer l .
Revised: Denote s l as the sparsity of layer l .

Original: Random pruning, namely, removes weights or ﬁlters in each layer randomly to the target sparsity s l .
Revised: Random pruning, namely, removes weights or ﬁlters in each layer randomly to the target sparsity s l .

Original: Different from works that learn layer-wise sparsities and model weights together during training using iterative global pruning techniques (Frankle & Carbin, 2018),soft threshold (Kusupati et al., 2020), and dynamic reparameterization (Mostafa & Wang, 2019), etc., the layer-wise sparsities of random pruning is pre-deﬁned before training.
Revised: Different from works that learn layer-wise sparsities and model weights together during training using iterative global pruning techniques (Frankle & Carbin, 2018),

Original: Many pre-deﬁned layer-wise sparsity ratios in the literature are suited for random pruning, while they may not initially be designed for random pruning. We choose the following 6 layer-wise sparsity ratios to study.
Revised: [Figure]

Original: ERK.
Revised: ERK.

Original: Introduced by Mocanu et al.
Revised: Introduced by Mocanu et al.

Original: (2018), Erd˝os-R´enyi (ER) sparsiﬁes the Multilayer Perceptron (MLP) with a random topology in which larger layers are allocated with higher sparsity then smaller layers.
Revised: , Erd ˝ os-R ´ enyi (ER) sparsiﬁes the Multilayer Perceptron (MLP) with a random topology in which larger layers are allocated with higher sparsity then smaller layers.

Original: Evci et al.
Revised: Evci et al.

Original: (2020a) further proposed a convolutional variant (Erd˝os-R´enyi-Kernel (ERK)) that takes the convolutional kernel into consideration.
Revised: (2020a) further proposed a convolutional variant (Erd ˝ os-R ´ enyi-Kernel (ERK)) that takes the convolutional kernel into consideration.

Original: Speciﬁcally, the sparsity of the convolutional layer is scaled proportional to 1 − n l − 1 + n l + w l + h l n l − 1 × n l × w l × h l where n l refers to the number of neurons/channels in layer l ; w l and h l are the corresponding width and height.
Revised: Speciﬁcally, the sparsity of the convolutional layer is scaled proportional to 1 − n l − 1 + n l + w l + h l n l − 1 × n l × w l × h l where n l refers to the number of neurons/channels in layer l ; w l and h l are the corresponding width and height.

Original: ERK+.
Revised: ERK+.

Original: We modify ERK by forcing the last fully-connected layer as dense if it is not, while keeping the overall parameter count the same.
Revised: We modify ERK by forcing the last fully-connected layer as dense if it is not, while keeping the overall parameter count the same.

Original: Doing so improves the test accuracy of Wide ResNet-50 on
Revised: Doing so improves the test accuracy of Wide ResNet-50 on

Original: ImageNet 1 as shown in Section 4.2.
Revised: ImageNet 2 as shown in Section 4.2.

Original: Uniform.
Revised: Uniform.

Original: Each layer is pruned with the same pruning rate so that the pruned network ends up with a uniform sparsity distribution, e.g., Zhu & Gupta (2017).[Figure]
Revised: Each layer is pruned with the same pruning rate so that the pruned network ends up with a uniform sparsity distribution, e.g., Zhu & Gupta (2017).

Original: Uniform+.
Revised: Uniform+.

Original: Instead of using a completely uniform sparsity ratio, Gale et al.
Revised: Instead of using a completely uniform sparsity ratio, Gale et al.

Original: (2019) keep the ﬁrst convolutional layer dense and maintain at least 20% parameters in the last fully-connected layer.
Revised: (2019) keep the ﬁrst convolutional layer dense and maintain at least 20% parameters in the last fully-connected layer.

Original: SNIP ratio.
Revised: SNIP ratio.

Original: SNIP selects important weights based on the connection sensitivity score | g (cid:12) w | , where w and g is the network weight and gradient, respectively.
Revised: SNIP selects important weights based on the connection sensitivity score | g (cid:12) w | , where w and g is the network weight and gradient, respectively.

Original: The weights with the lowest scores in one iteration are pruned before training.
Revised: The weights with the lowest scores in one iteration are pruned before training.

Original: While not initially designed for random pruning, we adjust
Revised: While not initially designed for random pruning, we adjust

Original: SNIP for random pruning by only keeping its layer-wise sparsity ratios, while discarding its mask positions.
Revised: SNIP for random pruning by only keeping its layer-wise sparsity ratios, while discarding its mask positions.

Original: New mask positions (non-zero elements) are re-sampled through a uniform distribution ∼ Uniform(0 , 1) with a probability of 1 − s l .
Revised: New mask positions (non-zero elements) are re-sampled through a uniform distribution ∼ Uniform(0 , 1) with a probability of 1 − s l .

Original: Such layer-wise sparsities are obtained in a (slightly) data-driven way, yet very efﬁciently before training without any weight update.
Revised: Such layer-wise sparsities are obtained in a (slightly) data-driven way, yet very efﬁciently before training without any weight update.

Original: The SNIP ratio is then treated as another pre-deﬁned (i.e., before the training starts) sampling ratio for random pruning.
Revised: The SNIP ratio is then treated as another pre-deﬁned (i.e., before the training starts) sampling ratio for random pruning.

Original: GraSP ratio.
Revised: GraSP ratio.

Original: GraSP is an other method that pursues network pruning before training.
Revised: GraSP is an other method that pursues network pruning before training.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: GraSP removes weights that have the least effect on the gradient norm based on the score of − w (cid:12)
Revised: GraSP removes weights that have the least effect on the gradient norm based on the score of − w (cid:12)

Original: Hg , where H is the Hessian matrix and g is the gradient.
Revised: Hg , where H is the Hessian matrix and g is the gradient.

Original: Same with SNIP, we keep the layer-wise sparsity ratios of GraSP and re-sample its mask positions.
Revised: Same with SNIP, we keep the layer-wise sparsity ratios of GraSP and re-sample its mask positions [Figure]

Original: 3. A RCHITECTURES , D ATASETS , AND I MPLEMENTATIONS
Revised: 3. A RCHITECTURES , D ATASETS , AND I MPLEMENTATIONS

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Our main experiments are conducted with the CIFAR version of ResNet (He et al., 2016) with varying depths and widths on CIFAR-10/100 (Krizhevsky et al., 2009).
Revised: Our main experiments are conducted with the CIFAR version of ResNet (He et al., 2016) with varying depths and widths on CIFAR-10/100 (Krizhevsky et al., 2009), VGG with varying depths on

Original: To draw solid conclusions, we also evaluate random pruning on ImageNet (Deng et al., 2009) with the ImageNet version of ResNet and Wide ResNet-50 (Zagoruyko & Komodakis, 2016).
Revised: CIFAR-10/100, and the ImageNet version of ResNet and Wide ResNet-50 (Zagoruyko & Komodakis, 2016) on ImageNet (Deng et al., 2009).

Original: For ImageNet, we follow the common setting in sparse training (Dettmers & Zettlemoyer, 2019;
Revised: For ImageNet, we follow the common setting in sparse training (Dettmers & Zettlemoyer, 2019;

Original: Evci et al., 2020b; Liu et al., 2021) and train sparse models for 100 epochs.
Revised: Evci et al., 2020b; Liu et al., 2021) and train sparse models for 100 epochs.

Original: All models are trained with stochastic gradient descent (SGD) with momentum.
Revised: All models are trained with stochastic gradient descent (SGD) with momentum.

Original: We share the summary of the architectures, datasets, and hyperparameters in Table 1.
Revised: We share the summary of the architectures, datasets, and hyperparameters in Table 1.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we report the results of random pruning under various evaluation measurements.
Revised: In this section, we report the results of random pruning under various evaluation measurements.

Original: For each measurement, the trade-off between sparsity and the corresponding metric is reported.
Revised: For each measurement, the trade-off between sparsity and the corresponding metric is reported.

Original: Moreover,[Figure] we also alter the depth and width of the model to check how the performance alters as the model size alters.
Revised: Moreover, we also alter the depth and width of the model to check how the performance alters as the model size alters.

Original: The results of CIFAR-10/100 are averaged over 3 runs, whereas the experiments of ImageNet are only ran once due to the limited resources.
Revised: The results of CIFAR-10/100 are averaged over 3 runs, whereas the experiments of ImageNet are only ran once due to the limited resources.

Original: We report the results on CIFAR-10 and ImageNet  in the main body of this paper.
Revised: We report the results on CIFARand ImageNet of ResNet in the main body of this paper.

Original: The results on CIFAR-100  are shown in Appendix C. Unless otherwise stated, the results of CIFAR-100 are qualitatively similar with CIFAR-10.
Revised: The results on CIFAR-100 of ResNet and CIFAR-10/100 of VGG are shown in Appendix C and Appendix D, respectively. Unless otherwise stated, the results of CIFAR-100 are qualitatively similar with CIFAR-10.

Revised: We provide a scaled-up overview of the accuracy and parameter count with various ResNet on ImageNet in

Revised: Appendix E. We report the trade-off between test accuracy and FLOPs of various randomly pruned ResNets and the corresponding dense ResNets on ImagetNet in Appendix F.

Revised: We see that the random pruning gets larger efﬁciency (the difference between the test FLOPs of sparse models and dense models) with the increased model size on ImageNet.

Original: [Figure]
Revised: [Figure]

Original:  ON CIFAR- 
Revised: 4. P REDICTIVE A CCURACY ON CIFAR-

Original: We ﬁrst demonstrate the performance of random pruning on the most common metric – test accuracy.
Revised: We ﬁrst demonstrate the performance of random pruning on the most common metric – test accuracy.

Original: To avoid overlapping of multiple curves, we share the results of GraSP in Appendix B.
Revised: To avoid overlapping of multiple curves, we share the results of GraSP in Appendix B.

Original: The main observations are as follows: 1 Performance of random pruning improves with the size of the network.
Revised: The main observations are as follows: 1 Performance of random pruning improves with the size of the network.

Original: We vary the depth of ResNet and report the test accuracy achieved by various pruning methods in Figure 1.
Revised: We vary the depth of ResNet and report the test accuracy achieved by various pruning methods in Figure 1.

Original: When operating on small networks, e.g., ResNet-20 and ResNet-32, we can hardly ﬁnd matching subnetworks even at mild sparsities, i.e., 10%, 20%.
Revised: When operating on small networks, e.g., ResNet-20 and ResNet-32, we can hardly ﬁnd matching subnetworks even at mild sparsities, i.e., 10%, 20%.

Original: With larger networks, e.g., ResNet-56 and ResNet-110, random pruning can match the dense performance at 60% ∼ 70% sparsity.
Revised: With larger networks, e.g., ResNet-56 and ResNet-110, random pruning can match the dense performance at 60% ∼ 70% sparsity.

Original: Similar behavior can also be observed when we increase the width of ResNet-20.
Revised: Similar behavior can also be observed when we increase the width of ResNet-20.

Original: Training with random pruning can just match the full accuracy as the model width increases as shown in Figure 2.
Revised: Training with random pruning can just match the full accuracy as the model width increases as shown in Figure 2.

Original: This encouraging trend is also shared with deeper architectures i.e., ResNet-32, ResNet-44 in Figure 3.
Revised: This encouraging trend is also shared with deeper architectures i.e., ResNet-32, ResNet-44 in Figure 3.

Original: The performance gap between densely trained networks and randomly pruned networks shrinks quickly as the model gets wider.
Revised: The performance gap between densely trained networks and randomly pruned networks shrinks quickly as the model gets wider.

Original:  The performance difference between different pruning methods becomes indistinct as the model size increases.
Revised: 2 The performance difference between different pruning methods becomes indistinct as the model size increases.

Original: Even though uniform sparsities fail to match the accuracy achieved by nonuniform sparsities (ERK and SNIP) with small models, their test accuracy raises to a comparable level as non-uniform sparsities with large models, e.g., ResNet-110 and ResNet-20-56.
Revised: Even though uniform sparsities fail to match the accuracy achieved by nonuniform sparsities (ERK and SNIP) with small models, their test accuracy raises to a comparable level as non-uniform sparsities with large models, e.g., ResNet-110 and ResNet-20-56.

Original: 3 Random pruning with ERK even outperforms pruning with the well-versed methods (SNIP, GraSP).
Revised: 3 Random pruning with ERK even outperforms pruning with the well-versed methods (SNIP, GraSP).

Original: Without using any information, e.g., gradient and magnitude, training a randomly pruned subnetwork with ERK topology leads to expressive accuracy, even better than the ones trained with the delicately designed sparsity ratios, i.e., SNIP, and GraSP (shown in Appendix B).
Revised: Without using any information, e.g., gradient and magnitude, training a randomly pruned subnetwork with ERK topology leads to expressive accuracy, even better than the ones trained with the delicately designed sparsity ratios, i.e., SNIP, and GraSP (shown in Appendix B).

Original: 4. P REDICTIVE A CCURACY ON I MAGE N ET
Revised: 4. P REDICTIVE A CCURACY ON I MAGE N ET

Original: We have learned from Section 4.1 that larger networks result in stronger random subnetworks on
Revised: We have learned from Section 4.1 that larger networks result in stronger random subnetworks on

Original: CIFAR-10/100.
Revised: CIFAR-10/100.

Original: We are also interested in how far we can go with random pruning on large-scale[Figure] datasets, i
Revised: We are also interested in how far we can go with random pruning on large-scale datasets, i.g.,

Original: [Figure] g., ImageNet (Deng et al., 2009), a non-saturated dataset on which deep neural networks are less over-parameterized than on CIFAR-10/100.[Figure]
Revised: ImageNet (Deng et al., 2009), a non-saturated dataset on which deep neural networks are less over-parameterized than on CIFAR-10/100.

Original: Performance improvement of ERK+.
Revised: Performance improvement of ERK+.

Original: We ﬁrst demonstrate the performance improvement on
Revised: We ﬁrst demonstrate the performance improvement on

Original: ImageNet caused by ERK+.
Revised: ImageNet caused by ERK+.

Original: As mentioned earlier, ERK naturally allocates higher sparsities to the larger layers while allocating lower sparsities to the smaller ones.
Revised: As mentioned earlier, ERK naturally allocates higher sparsities to the larger layers while allocating lower sparsities to the smaller ones.

Original: As a consequence, the last fullyconnected layer is very likely to be dense for the datasets with only a few classes, e.g., CIFAR-10.
Revised: As a consequence, the last fullyconnected layer is very likely to be dense for the datasets with only a few classes, e.g., CIFAR-10.

Original: However, for the datasets with a larger number of classes, e.g., ImageNet, the fully-connected layer is usually sparse.
Revised: However, for the datasets with a larger number of classes, e.g., ImageNet, the fully-connected layer is usually sparse.

Original: We empirically ﬁnd that allocating more parameters to the last fully-connected layer while keeping the overall parameter count ﬁxed leads to higher accuracy for ERK on ImageNet.
Revised: We empirically ﬁnd that allocating more parameters to the last fully-connected layer while keeping the overall parameter count ﬁxed leads to higher accuracy for ERK on ImageNet.

Original: Test accuracy of ERK with Wide ResNet-50 at various sparsity of the last layer is reported in Figure 4-left.
Revised: Test accuracy of ERK with Wide ResNet-50 at various sparsity of the last layer is reported in Figure 4-left.

Original: Consequently, we keep the last fully-connected layer of ERK dense for ImageNet and term this modiﬁed variant as ERK+.
Revised: Consequently, we keep the last fully-connected layer of ERK dense for ImageNet and term this modiﬁed variant as ERK+.

Original: Compare random pruning with its dense equivalents.
Revised: Compare random pruning with its dense equivalents.

Original: To draw a more solid conclusion, we train large, randomly pruned Wide ResNet-50 on ImageNet and compare it to the dense equivalents with  the same number of parameters on ImageNet.
Revised: To draw a more solid conclusion, we train large, randomly pruned Wide ResNet-50 on ImageNet and compare it to the dense equivalents with [Figure] [Figure] the same number of parameters on ImageNet.

Original: As shown in Figure4-right, all randomly pruned networks outperform the dense ResNet-34 with the same number of parameters.
Revised: As shown in Figure4-right, all randomly pruned networks outperform the dense ResNet-34 with the same number of parameters.

Original: ERK+ consistently achieves higher accuracy than ERK, even closely approaching the strong baseline – dense ResNet-50.
Revised: ERK+ consistently achieves higher accuracy than ERK, even closely approaching the strong baseline – dense ResNet-50.

Original: More interestingly, the layer-wise sparsities discovered by SNIP boost the accuracy of sparse
Revised: More interestingly, the layer-wise sparsities discovered by SNIP boost the accuracy of sparse

Original: Wide
Revised: Wide

Original: ResNet-50 over dense ResNet-50, highlighting the importance of layer-wise sparsity ratios on sparse training .
Revised: ResNet-50 over dense ResNet-50, highlighting the importance of layer-wise sparsity ratios on sparse training .

Original: Given the fact that the performance gap between SNIP ratio and ERK on CIFAR-10 is somehow vague, our results highlight the necessity of evaluating any proposed pruning methods with large-scale models and datasets, e.g., ResNet-50 on ImageNet.
Revised: Given the fact that the performance gap between SNIP ratio and ERK on CIFAR-10 is somehow vague, our results highlight the necessity of evaluating any proposed pruning methods with large-scale models and datasets, e.g., ResNet-50 on ImageNet.

Original: 4. U NDERSTANDING R ANDOM P RUNING VIA G RADIENT F LOW
Revised: 4. U NDERSTANDING R ANDOM P RUNING VIA G RADIENT F LOW

Original: The performance gap between the ERK and SNIP ratio on ImageNet raises the question – what beneﬁts do the layer-wise sparsities of SNIP provide to sparse training?Since SNIP takes gradients [Figure] [Figure]
Revised: The performance gap between the ERK and SNIP ratio on ImageNet raises the question – what beneﬁts do the layer-wise sparsities of SNIP provide to sparse training?

Original: [Figure] [Figure] into account when determines the layer-wise sparsities, we turn to gradient ﬂow in the hope of ﬁnding some insights on this question.
Revised: Since SNIP takes gradients into account when determines the layer-wise sparsities, we turn to gradient ﬂow in the hope of ﬁnding some insights on this question.

Original: The form of gradient norm we choose is the effective gradient ﬂow (Tessera et al., 2021) which only calculates the gradient norm of active weights.
Revised: The form of gradient norm we choose is the effective gradient ﬂow (Tessera et al., 2021) which only calculates the gradient norm of active weights.

Original: We measure the gradient norm of sparse networks generated by SNIP and ERK 2 during the early training phase.
Revised: We measure the gradient norm of sparse networks generated by SNIP and ERK 3 during the early training phase.

Original: The results are depicted in Figure 5. 
Revised: The results are depicted in Figure 5. [Figure]

Original: Overall, we see that the SNIP ratio indeed brings beneﬁts to the gradient norm at the initial stage compared with ERK.
Revised: Overall, we see that the SNIP ratio indeed brings beneﬁts to the gradient norm at the initial stage compared with ERK.

Original: Considering only SNIP (green lines) and ERK (orange lines), the ones with higher gradient norm at the beginning always achieve higher ﬁnal accuracy on both CIFAR-10 and
Revised: Considering only SNIP (green lines) and ERK (orange lines) [Figure] the ones with higher gradient norm at the beginning always achieve higher ﬁnal accuracy on both CIFAR-10 and

Original: ImageNet.
Revised: ImageNet.

Original: This result is on par with prior work (Wang et al., 2020), which claims that increasing the gradient norm at the beginning likely leads to higher accuracy.
Revised: This result is on par with prior work (Wang et al., 2020) [Figure] which claims that increasing the gradient norm at the beginning likely leads to higher accuracy

Original: However, this observation does not hold for dense Wide ResNet-50 on ImageNet (blue lines).
Revised: However, this observation does not hold for dense Wide ResNet-50 on ImageNet (blue lines).

Original: Even though dense models have a lower gradient norm than SNIP, they consistently outperform SNIP in accuracy by a large margin.
Revised: Even though dense models have a lower gradient norm than SNIP, they consistently outperform SNIP in accuracy by a large margin.

Original: Instead of focusing on the initial phase, we note that the gradient norm early in the training (the ﬂat phase after gradient norm drops) could better understand the behaviour of networks trained under different scenarios.
Revised: Instead of focusing on the initial phase [Figure] we note that the gradient norm early in the training (the ﬂat phase after gradient norm drops) could better understand the behaviour of networks trained under different scenarios.

Original: We empirically ﬁnd that ﬁnal performance gaps between sparse networks and dense networks is highly correlated with gaps of gradient norm early in training.
Revised: We empirically ﬁnd that ﬁnal performance gaps between sparse networks and dense networks is highly correlated with gaps of gradient norm early in training.

Original: Training with small networks (e.g., ResNet-20) on CIFAR-10 results in large performance and gradient norm gaps between sparse networks and dense networks, whereas these two gaps simultaneously vanish when trained with large networks, e.g., ResNet-56 and ResNet-110.
Revised: Training with small networks (e.g., ResNet-20) on CIFAR-10 results in large performance and gradient norm gaps between sparse networks and dense networks, whereas these two gaps simultaneously vanish when trained with large networks, e.g., ResNet-56 and ResNet-110.

Original: Similar behavior can be observed in
Revised: Similar behavior can be observed in

Original: Wide ResNet-50 on ImageNet in terms of sparsity.
Revised: Wide ResNet-50 on ImageNet in terms of sparsity.

Original: Both the performance and gradient norm gaps decrease gradually as the sparsity level drops from 85% to 60%.
Revised: Both the performance and gradient norm gaps decrease gradually as the sparsity level drops from 85% to 60%.

Original: This actually makes sense, since large networks (either large model size or large number of parameters) are likely to still be overparameterized after pruning, so that all pruning methods (including random pruning) can preserve gradient ﬂow along with test accuracy equally well.
Revised: This actually makes sense, since large networks (either large model size or large number of parameters) are likely to still be overparameterized after pruning, so that all pruning methods (including random pruning) can preserve gradient ﬂow along with test accuracy equally well.

Original: Our ﬁndings suggest that only considering gradient ﬂow at initialization might not be sufﬁcient for sparse training.
Revised: Our ﬁndings suggest that only considering gradient ﬂow at initialization might not be sufﬁcient for sparse training.

Original: More efforts should be invested to study the properties that sparse network training misses after initialization, especially the early stage of training (Frankle et al., 2020c).
Revised: More efforts should be invested to study the properties that sparse network training misses after initialization, especially the early stage of training (Frankle et al., 2020b).

Original: Techniques that change the sparse pattern during training (Mocanu et al., 2018; Evci et al., 2020a) and weight rewinding (Frankle et al., 2020a; Renda et al., 2020) could serve as good starting points.
Revised: Techniques that change the sparse pattern during training (Mocanu et al., 2018; Evci et al., 2020a) and weight rewinding (Frankle et al., 2020a; Renda et al., 2020) could serve as good starting points.

Original: 4. B ROADER E VALUATION OF R ANDOM P RUNING
Revised: 4. B ROADER E VALUATION OF R ANDOM P RUNING

Original: In most pruning literature, test accuracy is the core quality that researchers consider.
Revised: In most pruning literature, test accuracy is the core quality that researchers consider.

Original: However, the evaluation of other perspectives is also important for academia and industry before replacing[Figure] [Figure] dense networks with sparse networks on a large scope.
Revised: However, the evaluation of other perspectives is also important for academia and industry before replacing dense networks with sparse networks on a large scope.

Original: Therefore, we thoroughly evaluate random pruning from a broader perspective.
Revised: Therefore, we thoroughly evaluate random pruning from a broader perspective.

Original: Speciﬁcally, we assess random pruning from perspectives of out-of-distribution (OoD) performance (Hendrycks et al., 2020), adversarial robustness (Goodfellow et al., 2014), and uncertainty estimation (Lakshminarayanan et al., 2016).
Revised: Speciﬁcally, we assess random pruning from perspectives of out-of-distribution (OoD) performance (Hendrycks et al., 2020), adversarial robustness (Goodfellow et al., 2014), and uncertainty estimation (Lakshminarayanan et al., 2016).

Original: Measurements.
Revised: Measurements.

Original: We brieﬂy introduce each measurement and the corresponding metric used here.
Revised: We brieﬂy introduce each measurement and the corresponding metric used here.

Original: For more details, please see Appendix A. Formally, let’s denote a (sparse) network f : X → Y trained on samples from distribution D .
Revised: For more details, please see Appendix A. Formally, let’s denote a (sparse) network f : X → Y trained on samples from distribution D .

Original: f ( x, θ ) and f ( x, θ (cid:12) m ) refers to the dense network and the  pruned network, respectively.
Revised: f ( x, θ ) and f ( x, θ (cid:12) m ) refers to the dense network and the [Figure] [Figure] pruned network, respectively.

Original: To evaluate whether random pruning brings beneﬁts to OoD, we follow the classic routines (Augustin et al., 2020; Meinke & Hein, 2019), adopting SVHN (Netzer et al., 2011), CIFAR-100, and CIFAR-10 with random Gaussian noise (Hein et al., 2019) as OoD datasets for models trained on CIFAR-10.
Revised: To evaluate whether random pruning brings beneﬁts to OoD, we follow the classic routines (Augustin et al., 2020; Meinke & Hein, 2019), adopting SVHN (Netzer et al., 2011), CIFAR-100, and CIFAR-10 with random Gaussian noise (Hein et al., 2019) as OoD datasets for models trained on CIFAR-10; ImageNet-O as the OoD dataset for models trained on ImageNet.

Original: For adversarial robustness, we use Fast Gradient Sign Method (FGSM)
Revised: For adversarial robustness, we use Fast Gradient Sign Method (FGSM)

Original: (Goodfellow et al., 2014) to generate adversarial samples with a perturbation strength of 8255 .
Revised: (Goodfellow et al., 2014) to generate adversarial samples with a perturbation strength of 8255 .

Original: We choose two widely-used metrics, expected calibration error (ECE) (Guo et al., 2017) and negative log-likelihood (NLL) (Friedman et al., 2001) for the uncertainty estimation.
Revised: We choose two widely-used metrics, expected calibration error (ECE) (Guo et al., 2017) and negative log-likelihood (NLL) (Friedman et al., 2001) for the uncertainty estimation.

Original: [Figure]
Revised: Due to the limited space, we report the results of CIFAR-10 in the main paper. The results on ImageNet are reported in Appendix G.

Original: In general, random pruning achieves quite strong results on uncertainty estimation, OoD robustness, and adversarial robustness without any fancy techniques.
Revised: In general, random pruning achieves quite strong results on uncertainty estimation, OoD robustness, and adversarial robustness without any fancy techniques.

Original: We summary main observations as below:
Revised: We summary main observations as below:

Original:  Randomly pruned networks enjoy better uncertainty estimation than their dense counterparts.
Revised: 1 Randomly pruned networks enjoy better uncertainty estimation than their dense counterparts.

Original: Figure 6 shows that randomly pruned ResNet-20 matches or even improves the uncertainty estimation of dense networks with a full range of sparsities.
Revised: Figure 6 shows that randomly pruned ResNet-20 matches or even improves the uncertainty estimation of dense networks with a full range of sparsities.

Original: ECE of random pruning grows with the model size, in line with the ﬁnding of dense networks in Guo et al.
Revised: ECE of random pruning grows with the model size, in line with the ﬁnding of dense networks in Guo et al.

Original: Still, random pruning is capable of sampling matching subnetworks with high sparsities (e.g., 80%) except for the largest model, ResNet-110.
Revised: Still, random pruning is capable of sampling matching subnetworks with high sparsities (e.g., 80%) except for the largest model, ResNet-110.

Original: Results of NLL are presented in Appendix D.2, where randomly pruned networks also match NLL of dense networks at extremely high sparsities.
Revised: Results of NLL are presented in Appendix H.2, where randomly pruned networks also match NLL of dense networks at extremely high sparsities.

Original: 2 Random pruning produces extremely sparse yet robust subnetworks on OoD. Figure 7 plots the results of networks trained on CIFAR-10, tested on CIFAR-100.
Revised: 2 Random pruning produces extremely sparse yet robust subnetworks on OoD. Figure 7 plots the results of networks trained on CIFAR-10, tested on CIFAR-100.

Original: The other two sets of experiments are reported in Appendix D.1.
Revised: The other two sets of experiments are reported in Appendix H.1.

Original: Large network sizes signiﬁcantly improve the OoD performance of random pruning.
Revised: Large network sizes signiﬁcantly improve the OoD performance of random pruning.

Original: As the model size increases, random pruning can match the OoD performance of dense networks with only 20% parameters.
Revised: As the model size increases, random pruning can match the OoD performance of dense networks with only 20% parameters.

Original: Again, SNIP ratio and ERK outperform uniform sparsities in this setting.
Revised: Again, SNIP ratio and ERK outperform uniform sparsities in this setting.

Original: 3 Random pruning improves the adversarial robustness of large models.
Revised: 3 Random pruning improves the adversarial robustness of large models.

Original: The adversarial robustness of large models (e.g., ResNet-56 and ResNet-110) improves signiﬁcantly with mild sparsities in Figure 8.
Revised: The adversarial robustness of large models (e.g., ResNet-56 and ResNet-110) improves signiﬁcantly with mild sparsities in Figure 8.

Original: One explanation here is that, while achieving high clean accuracy, these over-parameterized large models are highly overﬁtted on CIFAR-10, and thus suffer from poor performance on adversarial examples (shown by Tsipras et al.
Revised: One explanation here is that, while achieving high clean accuracy, these over-parameterized large models are highly overﬁtted on CIFAR-10, and thus suffer from poor performance on adversarial examples (shown by Tsipras et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2019) as well).
Revised: (2019) as well).

Original: Sparsity induced by random pruning serves as a cheap type of regularization which possibly mitigates this overﬁtting problem.
Revised: Sparsity induced by random pruning serves as a cheap type of regularization which possibly mitigates this overﬁtting problem.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we systematically revisit the underrated baseline of sparse training – random pruning.
Revised: In this work, we systematically revisit the underrated baseline of sparse training – random pruning.

Original: Our results highlight a counter-intuitive ﬁnding, that is, training a randomly pruned network from scratch without any delicate pruning criteria can be quite performant.
Revised: Our results highlight a counter-intuitive ﬁnding, that is, training a randomly pruned network from scratch without any delicate pruning criteria can be quite performant.

Original: With proper network sizes and layer-wise sparsity ratios, random pruning can match the performance of dense networks even at extreme sparsities.
Revised: With proper network sizes and layer-wise sparsity ratios, random pruning can match the performance of dense networks even at extreme sparsities.

Original: Impressively, a randomly pruned subnetwork of Wide ResNet-50 can be trained to outperforming a strong benchmark, dense ResNet-50 on ImageNet.
Revised: Impressively, a randomly pruned subnetwork of Wide ResNet-50 can be trained to outperforming a strong benchmark, dense ResNet-50 on ImageNet.

Original: Moreover, training with random pruning intrinsically brings signiﬁcant beneﬁts to other desirable aspects, such as out-of-distribution, uncertainty estimation and adversarial robustness.
Revised: Moreover, training with random pruning intrinsically brings signiﬁcant beneﬁts to other desirable aspects, such as out-of-distribution, uncertainty estimation and adversarial robustness.

Original: Our paper indicates that in addition to appealing performance, large models also enjoy strong robustness to pruning.
Revised: Our paper indicates that in addition to appealing performance, large models also enjoy strong robustness to pruning.

Original: Even if we pruning with complete randomness, large models can preserve their performance well.
Revised: Even if we pruning with complete randomness, large models can preserve their performance well.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: We have shared the architectures, datasets, and hyperparameters used in this paper in Section 3.2.
Revised: We have shared the architectures, datasets and hyperparameters used in this paper in Section 3.2.

Original: Besides, the different measurements and metrics used in this paper are shared in Appendix A. We have also submitted our code as supplementary ﬁles.
Revised: Besides, the different measurements and metrics used in this paper are shared in Appendix A. We have also submitted our code as supplementary ﬁles.

Original: A M EASUREMENTS AND M ETRICS
Revised: A M EASUREMENTS AND M ETRICS

Original: Formally, let’s denote a (sparse) network f : X → Y trained on samples from distribution D .
Revised: Formally, let’s denote a (sparse) network f : X → Y trained on samples from distribution D .

Original: f ( x, θ ) and f ( x, θ (cid:12) m ) refers to the dense network and the pruned network, respectively.
Revised: f ( x, θ ) and f ( x, θ (cid:12) m ) refers to the dense network and the pruned network, respectively.

Original: The test accuracy of the pruned subnetworks is typically reported as their accuracy on test queries drawn from D , i.e., P ( x,y )
Revised: The test accuracy of the pruned subnetworks is typically reported as their accuracy on test queries drawn from D , i.e., P ( x,y )

Original: ∼D ( f ( x, θ (cid:12) m )
Revised: ∼D ( f ( x, θ (cid:12) m )

Original: = y ) .
Revised: = y ) .

Original: In addition to the test accuracy, we also evaluate random pruning from the perspective of natural corruption robustness, out-of-distribution performance, and uncertainty estimation.
Revised: In addition to the test accuracy, we also evaluate random pruning from the perspective of natural corruption robustness, out-of-distribution performance, and uncertainty estimation.

Original: Adversarial Robustness Despite of the remarkable ability solving classiﬁcation problems, the predictions of deep neural networks can often be fooled by small adversarial perturbations (Szegedy et al., 2013; Papernot et al., 2016).
Revised: Adversarial Robustness Despite of the remarkable ability solving classiﬁcation problems, the predictions of deep neural networks can often be fooled by small adversarial perturbations (Szegedy et al., 2013; Papernot et al., 2016).

Original: Prior works have shown the possibility of ﬁnding the sweet point of sparsity and adversarial robustness (Guo et al., 2018;
Revised: Prior works have shown the possibility of ﬁnding the sweet point of sparsity and adversarial robustness (Guo et al., 2018;

Original: Ye et al., 2019).
Revised: Ye et al., 2019).

Original: As the arguably most naive method of inducing sparsity, we are also interested in if training a randomly pruned subnetwork can improve the adversarial robustness of deep networks.
Revised: As the arguably most naive method of inducing sparsity, we are also interested in if training a randomly pruned subnetwork can improve the adversarial robustness of deep networks.

Original: We follow the classical method proposed in Goodfellow et al.
Revised: We follow the classical method proposed in Goodfellow et al.

Original: and generate adversarial examples with Fast Gradient Sign
Revised: and generate adversarial examples with Fast Gradient Sign

Original: Method (FGSM).
Revised: Method (FGSM).

Original: Speciﬁcally, input data is perturbed with (cid:15) sign ( ∇ x L ( θ, x, y )) , where (cid:15) refers to the perturbation strength, which is chosen as 8255 in our paper.
Revised: Speciﬁcally, input data is perturbed with (cid:15) sign ( ∇ x L ( θ, x, y )) , where (cid:15) refers to the perturbation strength, which is chosen as 8255 in our paper.

Original: Out-of-distribution performance.
Revised: Out-of-distribution performance.

Original: The investigation of out-of-distribution (OoD) generalization is of importance for machine learning in both academic and industry ﬁelds.
Revised: The investigation of out-of-distribution (OoD) generalization is of importance for machine learning in both academic and industry ﬁelds.

Original: Since the i.i.d.
Revised: Since the i.i.d.

Original: assumption can hardly be satisﬁed, especially those high-risk scenarios such as healthcare, and military.
Revised: assumption can hardly be satisﬁed, especially those high-risk scenarios such as healthcare, and military.

Original: We evaluate whether random pruning brings beneﬁts to OoD. Following the classic routines (Augustin et al., 2020; Meinke & Hein, 2019), SVHN (Netzer et al., 2011), CIFAR-100, and CIFAR-10 with random Gaussian noise (Hein et al., 2019) are adopted for models trained on CIFAR-10; SVHN,
Revised: We evaluate whether random pruning brings beneﬁts to OoD. Following the classic routines (Augustin et al., 2020; Meinke & Hein, 2019), SVHN (Netzer et al., 2011), CIFAR-100, and CIFAR-10 with random Gaussian noise (Hein et al., 2019) are adopted for models trained on CIFAR-10; SVHN,

Original: CIFAR-10, and CIFAR-100 with random Gaussian noise are used for models trained on CIFAR-100.
Revised: CIFAR-10, and CIFAR-100 with random Gaussian noise are used for models trained on CIFAR-100.

Original: Uncertainty estimation.
Revised: Uncertainty estimation.

Original: In the security-critical scenarios, e.g., self-driving, the classiﬁers not only must be accurate but also should indicate when they are likely to be incorrect (Guo et al., 2017).
Revised: In the security-critical scenarios, e.g., self-driving, the classiﬁers not only must be accurate but also should indicate when they are likely to be incorrect (Guo et al., 2017).

Original: To test effects of the induced sparsity on uncertainty estimation, we choose two widely-used metrics, expected calibration error (ECE) (Guo et al., 2017) and negative log likelihood (NLL) (Friedman et al., 2001).
Revised: To test effects of the induced sparsity on uncertainty estimation, we choose two widely-used metrics, expected calibration error (ECE) (Guo et al., 2017) and negative log likelihood (NLL) (Friedman et al., 2001).

Original: B P REDICTIVE A CCURACY OF  ON CIFAR-10 I NCLUDING G RA SP
Revised: B P REDICTIVE A CCURACY OF R ES N ET ON CIFAR-10 I NCLUDING G RA SP

Original: To avoid multiple lines overlap with each other
Revised: To avoid multiple lines overlap with each other

Original: [Figure]
Revised: [Figure]

Original: we report the results of GraSP ratio on CIFARseparately in this Appendix.
Revised: we report the results of GraSP ratio on CIFARseparately in this Appendix.

Original: It is surprising to ﬁnd that random pruning with GraSP ratio (red lines) consistently have lower accuracy than ERK and SNIP, except for extremely high sparsites,
Revised: It is surprising to ﬁnd that random pruning with GraSP ratio (red lines) consistently have lower accuracy than ERK and SNIP, except for extremely high sparsites,

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e., 0.
Revised: e., 0.

Original: and 0.9.
Revised: and 0.9.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: c y
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: C P REDICTIVE A CCURACY  ON CIFAR- 
Revised: D P REDICTIVE A CCURACY OF VGG ON CIFAR-10/

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D B ROADER E VALUATION OF R ANDOM P RUNING ON CIFARD. O UT - OF - DISTRIBUTION PERFORMANCE (ROC-AUC) .
Revised: E S AME S CALE OF E XPERIMENTS ON I MAGE N ET

Revised: We provide a scaled-up overview of the accuracy and parameter count with various ResNet on

Original: [Figure]
Revised: ImageNet.

Revised: We have expanded the experiments on ImageNet with various architectures from ResNet18 to ResNet-101 and Wide ResNet-50.

Revised: Due to the limited time and resources, we choose the SNIP and ERK+ sparsity ratios for Wide ResNet-50 and only SNIP ratios for the rest of the architectures.

Original: [Figure]
Revised: The sparsity of randomly pruned subnetworks is set as [0.7, 0.5, 0.3] from left to right. We run all this set of experiments with A100 GPUs whose accuracy is slightly different from the original submission (ran with V100). The results are shared in Figure 15. Overall, we observe a very similar pattern as illustrated in the main paper.

Revised: On smaller models like ResNet-18 and ResNet-34, random pruning at initialization can not ﬁnd matching subnetworks.

Original: D
Revised: When the model size gets considerably larger (ResNet-101 and Wide ResNet-50), random pruning can match the dense models at 50% sparsity. [Figure] F T RADE - OFF BETWEEN A CCURACY AND FLOP S OF V ARIOUS R ES N ET MODELS ON I MAGE N ET

Revised: We provide a scaled-up overview of the trade-off between accuracy and test FLOPs with various

Revised: ResNet on ImageNet.

Revised: The results are shared in Figure 16.

Revised: We can see that the number of FLOPs required by the pruned Wide ResNet-50 is still larger than the one required by the dense ResNet-50.

Revised: Yet, the corresponding performance gain is also notable (2%).

Revised: More importantly, when the model size is large enough, the accuracy of the randomly pruned sparse networks quickly grows and matches the corresponding large dense network (not only the small-dense models).

Revised: Random pruning at initialization gets larger efﬁciency (the difference between the test FLOPs of sparse models and dense models) when the model size increases.

Revised: [Figure]

Revised: G B ROADER E VALUATION OF R ANDOM P RUNING ON I MAGE N ET G. A DVERSARIAL R OBUSTNESS ON I MAGE N ET

Original: [Figure]
Revised: [Figure]

Revised: G. O UT - OF -D ISTRIBUTION ON I MAGE N ET

Revised: [Figure]

Revised: G.

Original: U NCERTAINTY E STIMATION (NLL)
Revised: U NCERTAINTY E STIMATION (ECE) ON I MAGE N ET

Original: [Figure]
Revised: [Figure]

Revised: G. U NCERTAINTY E STIMATION (NLL) ON I MAGE N ET

Revised: [Figure]

Revised: H B ROADER E VALUATION OF R ANDOM P RUNING ON CIFARH. O UT - OF - DISTRIBUTION PERFORMANCE (ROC-AUC) .

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]