Original: Dynamic Pruning of a Neural Network via Gradient Signal-to-Noise RatioAnonymous
Revised: Dynamic Pruning of a Neural Network via Gradient Signal-to-Noise Ratio

Original: Abstract
Revised: Abstract

Original: While training highly overparameterized neural networks is common practice in deep learning, research into post-hoc weight-pruning suggests that more than 90% of parameters can be removed without loss in predictive performance.
Revised: While training highly overparameterized neural networks is common practice in deep learning, research into post-hoc weight-pruning suggests that more than 90% of parameters can be removed without loss in predictive performance.

Original: To save resources, zero-shot and one-shot pruning attempt to ﬁnd such a sparse representation at initialization or at an early stage of training.
Revised: To save resources, zero-shot and one-shot pruning attempt to ﬁnd such a sparse representation at initialization or at an early stage of training.

Original: Though eﬃcient, there is no justiﬁcation, why the sparsity structure should not change during training.
Revised: Though eﬃcient, there is no justiﬁcation, why the sparsity structure should not change during training.

Original: Dynamic sparsity pruning undoes this limitation and allows to adapt the structure of the sparse neural network during training.
Revised: Dynamic sparsity pruning undoes this limitation and allows to adapt the structure of the sparse neural network during training.

Original: In this work we propose to use the gradient noise to make pruning decisions.
Revised: In this work we propose to use the gradient noise to make pruning decisions.

Original: The procedure enables us to automatically adjust the sparsity during training without imposing a hand-designed sparsity schedule, while at the same time being able to recover from previous pruning decisions by unpruning connections as necessary.
Revised: The procedure enables us to automatically adjust the sparsity during training without imposing a hand-designed sparsity schedule, while at the same time being able to recover from previous pruning decisions by unpruning connections as necessary.

Original: We evaluate our method on image and tabular datasets and demonstrate that we reach similar performance as the dense model from which the sparse network is extracted, while exposing less hyperparameters than other dynamic sparsity methods.
Revised: We evaluate our method on image and tabular datasets and demonstrate that we reach similar performance as the dense model from which the sparse network is extracted, while exposing less hyperparameters than other dynamic sparsity methods.

Original: 1. Introduction
Revised: 1. Introduction

Original: Deep neural networks have been applied successfully to several machine learning tasks spanning, for instance, computer vision and natural language processing.
Revised: Deep neural networks have been applied successfully to several machine learning tasks spanning, for instance, computer vision and natural language processing.

Original: However, neural networks tend to reach the best performance if they are highly overparameterized (Nakkiran et al., 2020), resulting in high inference and training cost.
Revised: However, neural networks tend to reach the best performance if they are highly overparameterized (Nakkiran et al., 2020), resulting in high inference and training cost.

Original: Post-hoc pruning and retraining can often prune parameters in a network to 1/10 or 1/100 of the total number of parameters of the unpruned network and result in a better predictive performance (Han et al., 2015), raising questions of how one may prune parameters already during training.
Revised: Post-hoc pruning and retraining can often prune parameters in a network to 1/10 or 1/100 of the total number of parameters of the unpruned network and result in a better predictive performance (Han et al., 2015), raising questions of how one may prune parameters already during training.

Original: To that end, several approaches have emerged, that can be diﬀerentiated roughly by when the network is being pruned: at or close to initialization, during training  and post-hoc after training (Han et al., 2015).
Revised: To that end, several approaches have emerged, that can be distinguished roughly by when the network is being pruned: at or close to initialization (Frankle and Carbin, 2018; Wang et al., 2019, SNIP, GASP respectively), during training (Lin et al., 2019, DPF) and post-hoc after training (Han et al., 2015).

Original: A fourth category would be iterative magnitude pruning to discover lottery tickets (Frankle and Carbin, 2018).
Revised: A fourth category would be iterative magnitude pruning to discover lottery tickets (Frankle and Carbin, 2018).

Original: Pruning at initialization via gradient-based weight pruning such as SNIP(Lee et al., 2018) or GRASP(Wang et al., 2019) ﬁnd sparse networks which one can train eﬃciently from scratch.
Revised: Pruning at initialization via gradient-based weight pruning such as SNIP or GRASP ﬁnd sparse networks which one can train eﬃciently from scratch.

Original: However, they only prune once  and do not adaptduring training of the sparse network.
Revised: However, they only prune once early on and do not adapt the sparse network during training.

Original: An extensive comparison of diﬀerent pruning at initialization method was done by Frankle et al.
Revised: An extensive comparison of diﬀerent pruning at initialization method was done by Frankle et al.

Original: (2021) who discuss beneﬁts and shortcomings of each method.
Revised: (2021) who discuss beneﬁts and shortcomings of each method.

Original: They found that magnitude pruning after training outperformed GRASP and SNIP and that methods for pruning at initialization appear to actually suﬀer from the initialization and work better when used after training for a few iterations.
Revised: They found that magnitude pruning after training outperformed GRASP and SNIP and that methods for pruning at initialization appear to actually suﬀer from the initialization and work better when used after training for a few iterations.

Original: Finally, they found that the layer ratios found via SNIP/GRASP are more informative than the individual weights per layer found.
Revised: Finally, they found that the layer ratios found via SNIP/GRASP are more informative than the individual weights per layer found.

Original: This work questions the utility of the pruning at initialization setting.
Revised: This work questions the utility of the pruning at initialization setting.

Original: In this paper, we propose a method that extends Dynamic Pruning with Feedback (DPF)
Revised: In this paper, we propose a method that extends Dynamic Pruning with Feedback (DPF)

Original: (Lin et al., 2019).
Revised: (Lin et al., 2019).

Original: DPF prunes during training and ﬁnds the sparse network and weights simultaneously, due to which this setting is sometimes coined ‘dynamic sparsity’ (Mocanu et al., 2018; Mostafa and
Revised: DPF prunes during training and ﬁnds the sparse network and weights simultaneously, due to which this setting is sometimes coined ‘dynamic sparsity’ (Mocanu et al., 2018; Mostafa and

Original: Wang, 2019; Lin et al., 2019).
Revised: Wang, 2019; Lin et al., 2019).

Original: A dynamic sparsity/pruning algorithm caters to the dynamics of the optimization process rather than observing and pruning at one snapshot in time.
Revised: A dynamic sparsity/pruning algorithm caters to the dynamics of the optimization process rather than observing and pruning at one snapshot in time.

Original: Our extension provides a novel pruning criterion which collects insights from the optimization of the network to decide when to consider a weight for pruning and changes based on the dataset and the model architecture used.
Revised: Our extension provides a novel pruning criterion which collects statistics from the optimization of the network to decide when to consider a weight for pruning and changes based on the dataset and the model architecture used.

Original: Concretely our contributions are the following:
Revised: Concretely our contributions are the following:

Original: • We propose to use the signal-to-noise ratio  of the mini-batch gradient to measure when a weight can be considered for pruning.
Revised: • We propose to use the signal-to-noise ratio (SNR) of the mini-batch gradient to measure when a weight can be considered for pruning.

Original: This automatically provides a custom sparsity schedule that depends on the  dataset and network architecture used.
Revised: This automatically provides a custom sparsity schedule that depends on the present dataset and network architecture.

Original: • Using our novel sparsity criterion, we allow the number of non-zero entries in the sparsity masks to decrease and increase during training.
Revised: • Using our novel sparsity criterion, we allow the number of non-zero entries in the sparsity masks to decrease and increase during training.

Original: • We demonstrate the eﬀectiveness of our approach on image and tabular datasets where we obtain results on par with dense learning and DPF, while gaining additional insights into the learning procedure through the sparsity schedule.
Revised: • We demonstrate the eﬀectiveness of our approach on image and tabular datasets where we obtain results on par with dense learning and DPF, while gaining additional insights into the learning procedure through the sparsity schedule.

Original: In contrast to DPF, our algorithm needs no manual sparsity schedule which is a diﬃcult parameter to set in practice and depends on the network architecture and training dataset.
Revised: In contrast to DPF, our algorithm needs no manual sparsity schedule which is a diﬃcult parameter to set in practice and depends on the network architecture and training dataset.

Original: 2. Related Work
Revised: 2. Related Work

Original: Pruning before training/at initialization.
Revised: Pruning before training/at initialization.

Original: Several methods exist to prune at initialization.
Revised: Several methods exist to prune at initialization.

Original: One was proposed with Lottery Ticket Hypothesis (LTH) (Frankle and Carbin, 2018; Frankle et al., 2019a).
Revised: One was proposed with Lottery Ticket Hypothesis (LTH) (Frankle and Carbin, 2018; Frankle et al., 2019a).

Original: They showed that any suﬃciently overparameterized network contains sparse subnetworks which train at least as quickly and generalize as well as the original unpruned network.
Revised: They showed that any suﬃciently overparameterized network contains sparse subnetworks which train at least as quickly and generalize as well as the original unpruned network.

Original: However, they are expensive to ﬁnd.
Revised: However, they are expensive to ﬁnd.

Original: Alternatively, the gradients at initialization can be used to score connections in a network and identify dominant sparse networks, e.g., via SNIP (Lee et al., 2018) or GRASP (Wang et al., 2019).
Revised: Alternatively, the gradients at initialization can be used to score connections in a network and identify dominant sparse networks, e.g., via SNIP (Lee et al., 2018) or GRASP (Wang et al., 2019).

Original: In those settings the sparsity mask remains ﬁxed during training.
Revised: In those settings the sparsity mask remains ﬁxed during training.

Original: Pruning during training.
Revised: Pruning during training.

Original: Mocanu et al.
Revised: Mocanu et al.

Original: (2018) proposed sparse evolutionary training (SET) which prunes and regrows connections in a network at the end of each training epoch, following a predeﬁned sparsity schedule.
Revised: (2018) proposed sparse evolutionary training (SET) which prunes and regrows connections in a network at the end of each training epoch, following a predeﬁned sparsity schedule.

Original: This setting was extended by Mostafa and Wang (2019) and Dettmers and Zettlemoyer (2019) by regrowing via loss gradient and by momentum magnitude respectively, allowing to remove the hand-designed sparsity.
Revised: This setting was extended by Mostafa and Wang (2019) and Dettmers and Zettlemoyer (2019) by regrowing via loss gradient and by momentum magnitude respectively, allowing to remove the hand-designed sparsity.

Original: Our work is closely related to Dynamic Pruning with Feedback (DPF) (Lin et al., 2019), which is explained in more detail in Section 4.
Revised: Our work is closely related to Dynamic Pruning with Feedback (DPF) (Lin et al., 2019), which is explained in more detail in Section 4.

Original: Sparsity can also be learned via gradients during training as demonstrated by Louizos et al.
Revised: Sparsity can also be learned via gradients during training as demonstrated by Louizos et al.

Original: (2018) who uses L regularization, or more recently by Kusupati et al.
Revised: (2018) who uses L regularization, or more recently by Kusupati et al.

Original: (2020) who learn sparsity thresholds.
Revised: (2020) who learn sparsity thresholds.

Original: Pruning after training.
Revised: Pruning after training.

Original: This setting is the most commonly used in practice and has seen research interest for at least three decades.
Revised: This setting is the most commonly used in practice and has seen research interest for at least three decades.

Original: Prominent early examples were Optimal Brain Damage (Le Cun et al., 1989) and Optimal Brain Surgeon (Hassibi and Stork, 1992) which used approximations to the Hessian to prune neural networks.
Revised: Prominent early examples were Optimal Brain Damage (Le Cun et al., 1989) and Optimal Brain Surgeon (Hassibi and Stork, 1992) which used approximations to the Hessian to prune neural networks.

Original: More recent approaches in this setting include Molchanov et al.
Revised: More recent approaches in this setting include Molchanov et al.

Original: (2019) and Theis et al.
Revised: (2019) and Theis et al.

Original: (2018) who prune based on the ﬁrst and second order taylor expansions of the pruning problem respectively.
Revised: (2018) who prune based on the ﬁrst and second order taylor expansions of the pruning problem respectively.

Original: Neural architecture search.
Revised: Neural architecture search.

Original: The search for a sparse neural network may also be seen as a form of neural architecture search (NAS), which can be carried out in an existing overparameterizednetwork via dense to sparse pruning, similar to one-shot NAS (Liu et al., 2018), but may also include regrowing new connections (Mocanu et al., 2018; Dettmers and Zettlemoyer, 2019) like in evolutionary NAS (Real et al., 2019; Elsken et al., 2018). Sparse networks have also seen usage in continual learning (Cheung et al., 2019; Wortsman et al., 2020) where sparse subnetworks of a large neural network are used to avoid catastrophic forgetting. 3. Background: Signal-To-Noise Ratio of a Mini-Batch Gradient
Revised: The search for a sparse neural network may also be seen as a form of neural architecture search (NAS), which can be carried out in an existing overparameterized

Original: Here, we introduce the signal-to-noise ratio (SNR) of a mini-batch gradient; Section 4 shows how it will be used in the dynamic pruning algorithm.

Original: The gradient SNR assesses the reliability of a mini-batch gradient with respect to the noise resulting from the individual datapoints in the minibatch at every training step.

Original: Let B = { ( x i , y i ) } |B| i =1 be a mini-batch with ( x i , y i ) ∼ p ( x , y ) i.i.d.

Original: Require: uncompressed model weights θ ∈ R d , pruned weights: ˆ θ , mask:
Revised: Require: uncompressed model weights θ ∈ R d , pruned weights: ˆ θ , mask:

Original: m prune ∈ { 0 , 1 } d ; SNR exp.
Revised: m prune ∈ { 0 , 1 } d ; SNR exp.

Revised: avg.

Original: .: γ , burn-in steps: bin steps , mask: m  { 0 , 1 } d ;  iterations: T .
Revised: : γ , burn-in steps: bin steps , mask: m ∈ { 0 , 1 } d ; training iterations: T .

Original: 1: for t ...,T do 2: if t  steps then (cid:46) trigger mask update, by default after bin steps = 1 epoch 3: compute SNR mask m snr ← m snr,t ( θ t )(cid:46) by criterion SNR > 1 per weight 4: compute MAG mask m mag ← m mag,t,sp ( m snr )
Revised: 1: for t = 0 ,...,T do 2: if t > bin steps then (cid:46) trigger mask update, by default after bin steps = 1 epoch 3: compute SNR mask m snr ← { snr( θ it ) > 1 | i in | θ t |} (cid:46) let sp snr be the sparsity of the resulting mask 4: compute MAG mask m mag ← {| θ it | > γ | i in | θ t |} (cid:46) γ cut-oﬀ weight mag.

Original: ( θ t ) (cid:46) by unstructured magnitude pruning 5: compute PRUNE mask m prune ← m mag ∨ m snr (cid:46) only prune, if m mag and m snr agree to prune 6: end if 7: ˆ θ t ← m prune (cid:12) θ t (cid:46) apply resulting mask 8: compute (mini-batch) gradient ∇  B  θ ) (cid:46) /backward pass  weights ˆ θ t 9: update m snr,t (cid:46) update SNR exp mov.
Revised: acc to sp snr 5: compute PRUNE mask m prune ← m mag ∧ m snr (cid:46) only prune, if m mag and m snr agree to prune 6: end if 7: ˆ θ t ← m prune (cid:12) θ t (cid:46) apply resulting mask 8: compute (mini-batch) gradient ∇ L B (ˆ θ ) (cid:46) forward/backward pass with pruned weights ˆ θ t 9: update m snr,t (cid:46) update SNR exp mov.

Original: avg.
Revised: avg.

Original: per weight 10: θ t +1 ← gradient update ∇ L B (ˆ θ ) to θ t (cid:46) via 
Revised: per weight 10: θ t +1 ← gradient update ∇ L B (ˆ θ ) to θ t (cid:46) via arbitrary optimizer (e.g. SGD momentum) 11: end for

Revised: Ensure: θ T and ˆ θ T network via dense to sparse pruning, similar to one-shot NAS (Liu et al., 2018), but may also include regrowing new connections (Mocanu et al., 2018; Dettmers and Zettlemoyer, 2019) like in evolutionary NAS

Original: ) 11: end for
Revised: (Real et al., 2019; Elsken et al., 2018). Sparse networks have also seen usage in continual learning (Cheung et al., 2019; Wortsman et al., 2020) where sparse subnetworks of a large neural network are used to avoid catastrophic forgetting. 3. Background: Signal-To-Noise Ratio of a Mini-Batch Gradient This section introduces the signal-to-noise ratio (SNR) of a mini-batch gradient; Section 4 shows how it will be used in the dynamic pruning algorithm.

Revised: The gradient SNR assesses the reliability of a mini-batch gradient with respect to the noise resulting from the individual datapoints in the mini-batch at every training step.

Revised: Let B = { ( x i , y i ) } |B| i =1 be a mini-batch with ( x i , y i ) ∼ p ( x , y ) i.i.d.

Original: Ensure: θ and θ T draws and let ∇L ( θ ) be the true but unknown gradient of the risk L ( θ ) with respect to the weights θ of the neural network f θ ( x ).
Revised: draws and let ∇L ( θ ) be the true but unknown gradient of the risk L ( θ ) with respect to the weights θ of the neural network f θ ( x ).

Original: Furthermore, let Σ B ( θ ) :
Revised: Furthermore, let Σ B ( θ ) :

Original: =
Revised: =

Original: Cov[ ∇ (cid:96) i ( θ )] / |B| be the covariance of the mini-batch gradient ∇ L B ( θ ) =
Revised: Cov[ ∇ (cid:96) i ( θ )] / |B| be the covariance of the mini-batch gradient ∇ L B ( θ ) =

Original: (cid:80) i ∇ (cid:96) i ( θ )) / |B| with gradients for individual datapoints denoted by ∇ (cid:96) i ( θ ) :
Revised: (cid:80) i ∇ (cid:96) i ( θ )) / |B| with gradients for individual datapoints denoted by ∇ (cid:96) i ( θ ) :

Original: = ∇ (cid:96) ( f θ ( x i ) , y i ).
Revised: = ∇ (cid:96) ( f θ ( x i ) , y i ).

Original: Due to the i.i.d assumption on the training data one can motivate the  distribution ∇ L B ( θ ) ∼ N ( ∇L ( θ ) , Σ B ( θ )) via the central-limit theorem.
Revised: Due to the i.i.d assumption on the training data one can motivate the normal distribution ∇ L B ( θ ) ∼ N ( ∇L ( θ ) , Σ B ( θ )) via the central-limit theorem.

Original: Mahsereci et al.
Revised: Mahsereci et al.

Original: (2017) use the distribution ∇ L B ∼ N to construct the following criterion: If the statistic z ( θ ) :
Revised: (2017) use this distribution ∇ L B ∼ N to construct the following criterion: If the statistic z ( θ ) :

Original: = log p ( ∇ L B ( θ ) |∇L ( θ ) = 0) coincides with its expectation E x
Revised: = log p ( ∇ L B ( θ ) |∇L ( θ ) = 0) coincides with its expectation E x

Original: [ z ( θ )], then the mini-batch gradient is not informative anymore as it can be fully explained by sample noise and a vanishing, underlying true gradient ∇L ( θ ) = 0.
Revised: [ z ( θ )], then the mini-batch gradient is not informative anymore as it can be fully explained by sample noise and a vanishing, underlying true gradient ∇L ( θ ) = 0.

Original: More formally, per weight θ k , and with the simpliﬁcation Σ B ( θ ) =
Revised: More formally, per weight θ k , and with the simpliﬁcation Σ B ( θ ) =

Original: diag(Var[ ∇ (cid:96) i ( θ )]) / |B| , it happens when snr ( θ k ) ≤ 1, where snr ( θ k ) :
Revised: diag(Var[ ∇ (cid:96) i ( θ )]) / |B| , it happens when snr ( θ k ) ≤ 1, where snr ( θ k ) :

Original: = ∇ L 2 B ,k
Revised: = ∇ L 2 B ,k

Original: ( θ ) / Σ B ,kk ( θ ) is the signal-to-noise ratio (SNR).
Revised: ( θ ) / Σ B ,kk ( θ ) is the signal-to-noise ratio (SNR).

Original: In practice, the gradient variances Σ B ,kk ( θ ) required for the SNR can be eﬃciently estimated during backpropagation from the mini-batch itself (Dangel et al., 2020).
Revised: In practice, the gradient variances Σ B ,kk ( θ ) required for the SNR can be eﬃciently estimated during backpropagation from the mini-batch itself (Dangel et al., 2020).

Original: 4. Method
Revised: 4. Method

Original: Our work builds upon Dynamic Pruning with Feedback (DPF)
Revised: Our work builds upon Dynamic Pruning with Feedback (DPF)

Original: (Lin et al., 2019) which performs weight magnitude pruning according to a predeﬁned pruning schedule during training.
Revised: (Lin et al., 2019) which performs weight magnitude pruning according to a predeﬁned pruning schedule during training.

Original: Crucially, the method allows to recover from sub-optimal pruning decisions early on during training by updating the parameters of the underlying unpruned model alongside the pruned model and periodically recomputing the weight magnitude mask based on the unpruned weights, allowing weights to reenter the pruned network.
Revised: Crucially, the method allows to recover from sub-optimal pruning decisions early on during training by updating the parameters of the underlying unpruned model alongside the pruned model and periodically recomputing the weight magnitude mask based on the unpruned weights, allowing weights to reenter the pruned network.

Original: Due to the simplicity of this method it is a good baseline for the dynamic pruning setting, which the authors demonstrated to work well in practice.
Revised: Due to the simplicity of this method it is a good baseline for the dynamic pruning setting, which the authors demonstrated to work well in practice.

Original: However, two drawbacks of DPF should be noted.
Revised: However, two drawbacks of DPF should be noted.

Original: First, the sparsity schedule has to be manually set prior to training and is the same regardless of the dataset and model.
Revised: First, the sparsity schedule has to be manually set prior to training and is the same regardless of the dataset and model.

Original: Second, we argue that  dynamic pruning algorithms should consider the gradients during training to make pruning decisions, as these are more meaningful at the beginning of the optimization than only the weight magnitude due to the random weight initialization.
Revised: Second, we argue that [Table] dynamic pruning algorithms should consider the gradients during training to make pruning decisions, as these are more meaningful at the beginning of the optimization than only the weight magnitude due to the random weight initialization.

Original: In addition to these drawbacks the number of steps between mask updates (16 SGD steps) was found using hyperparameter optimization, leading to defaults which may not transfer to other settings and defeating the purpose of pruning.
Revised: In addition to these drawbacks the number of steps between mask updates (16 SGD steps) was found using hyperparameter optimization, leading to defaults which may not transfer to other settings and defeating the purpose of pruning

Original:  We propose to use the mini-batch gradient SNR to determine when a weight is no longer producing an informative gradient and only then to consider it for weight magnitude pruning.
Revised: [Table] We propose to use the mini-batch gradient SNR to determine when a weight is no longer producing an informative gradient and only then to consider it for weight magnitude pruning.

Original: Like DPF, we compute a global magnitude based mask (MAG) based on the unpruned weights.
Revised: Like DPF, we compute a global magnitude based mask (MAG) based on the unpruned weights.

Original: The decision to prune is then taken if the SNR mask and MAG mask agree to prune a weight.
Revised: The decision to prune is then taken if the SNR mask and MAG mask agree to prune a weight.

Original: This operation performs a logical AND and only prunes weights that are both converged as well as small.
Revised: This operation performs a logical AND and only prunes weights that are both converged as well as small.

Original: As the
Revised: As the

Original: SNR is computed per neuron and gradually decreases below 1 during training of the weight, our method does not need a manual sparsity schedule.
Revised: SNR is computed per neuron and gradually decreases below 1 during training of the weight, our method does not need a manual sparsity schedule.

Original: In particular, we set the sparsity of the MAG[Table] mask according to the sparsity arising automatically from the SNR mask, which allows us to prune less in the beginning and more at the end of training.
Revised: In particular, we set the sparsity of the MAG mask according to the sparsity arising automatically from the SNR mask, which allows us to prune less in the beginning and more at the end of training.

Original: At any stage of the pruning process, the algorithm has access to both gradient as well as weight magnitude information and is thus able to adapt to the dynamics of the optimization process (Algorithm 1)
Revised: At any stage of the pruning process, the algorithm has access to both gradient as well as weight magnitude information and is thus able to adapt to the dynamics of the optimization process (Algorithm 1).

Original: [Table]

Original: To avoid overpruning towards the end of training when the SNR of most weights transitions from > 1 to < 1, we put the pruning on hold based on the training loss, which deactivates pruning when the training loss increases.
Revised: To avoid overpruning towards the end of training when the SNR of most weights transitions from > 1 to < 1, we put the pruning on hold based on the training loss, which deactivates pruning when the training loss increases.

Original: Pruning is resumed, when the best so far seen training loss is reached again.
Revised: Pruning is resumed, when the best so far seen training loss is reached again.

Original: Although this heuristic seems to work reasonably well for the datasets we tried, and eﬀectively ties the total sparsity to the loss, learning a global sparsity parameter during dynamic pruning is so far unsolved, and we leave ﬁnding a more elegant solution to overpruning for future work.
Revised: Although this heuristic seems to work reasonably well for the datasets we tried, and eﬀectively ties the total sparsity to the loss, learning a global sparsity parameter during dynamic pruning is so far unsolved, and we leave ﬁnding a more elegant solution to overpruning for future work.

Original: Note, that as a limitation of our method we require more memory compared to DPF, since we do not only need to save the pruned and unpruned model like DPF, but in addition also the gradient SNR for each weight.
Revised: As a limitation of our method we require more memory compared to DPF, since we do not only need to save the pruned and unpruned model like DPF, but in addition also the gradient SNR for each weight.

Original: Further, our computational requirements also increase by a multiplicative factor of approximatly 1.25 (Dangel et al., 2020) since we compute the SNR per gradient computation.
Revised: Further, our computational requirements also increase by a multiplicative factor of approximatly 1.25 (Dangel et al., 2020) since we compute the SNR per gradient computation.

Original: 5. Experiments
Revised: 5. Experiments

Original: In our experiments, we directly compare to DPF (Lin et al., 2019) and normal dense training.
Revised: In our experiments, we directly compare to DPF (Lin et al., 2019) and normal dense training.

Original: In addition, we compare with GRASP-ABS, a variant of GRASP (Evci et al., 2020) proposed by Frankle et al.
Revised: In addition, we compare with GRASP-ABS, a variant of GRASP (Evci et al., 2020) proposed by Frankle et al.

Original: (2021) which uses the absolute value of the score in order to make GRASP more reliable.
Revised: (2021) which uses the absolute value of the score in order to make GRASP more reliable.

Original: In section 5.1, we test our pruning method on MNIST (LeCun et al., 2010), FashionMNIST (Xiao et al., 2017) and Cifar-10 (Krizhevsky et al., 2009) using a simple Convolutional Neural Network and ResNet18 (He et al., 2016).
Revised: In section 5.1, we test our pruning method on MNIST (LeCun et al., 2010), FashionMNIST (Xiao et al., 2017) and Cifar-10 (Krizhevsky et al., 2009) using a simple Convolutional Neural Network and ResNet18 (He et al., 2016).

Original: In section 5.2 we perform a similar analysis on tabular data from
Revised: In section 5.2 we perform a similar analysis on tabular data from

Original: OpenML (Vanschoren et al., 2013) using multi-layer perceptrons.
Revised: OpenML (Vanschoren et al., 2013) using multi-layer perceptrons.

Original: Details in Appendix A.1.
Revised: Details in Appendix A.1.

Original: 5.1 Image data
Revised: 5.1 Image data

Revised: [Figure]

Original: The results on FashionMNIST and MNIST are shown on the left in Figure 1.
Revised: The results on FashionMNIST and MNIST are shown on the left in Figure 1.

Original: Our algorithm compares favorably with DPF and dense training.
Revised: Our algorithm compares favorably with DPF and dense training.

Original: Compared to DPF’s schedule, our inferred schedule tends to prune many irrelevant weights at the beginning of training and the SNR mask may momentarily reduce the sparsity during training, which is not possible for DPF’s mask.
Revised: Compared to DPF’s schedule, our inferred schedule tends to prune many irrelevant weights at the beginning of training and the SNR mask may momentarily reduce the sparsity during training, which is not possible for DPF’s mask.

Original: The initial pruning is followed by several consolidation epochs, in which the SNR mask will only slowly increase in sparsity.
Revised: The initial pruning is followed by several consolidation epochs, in which the SNR mask will only slowly increase in sparsity.

Original: This consolidation period is shorter for MNIST compared to FashionMNIST, pointing to the intuition, that MNIST is easier to solve than FashionMNIST, which in turn allows to remove parameters more quickly.
Revised: This consolidation period is shorter for MNIST compared to FashionMNIST, pointing to the intuition, that MNIST is easier to solve than FashionMNIST, which in turn allows to remove parameters more quickly.

Original: An ablation experiment using only the magnitude mask or only the SNR mask for pruning is shown in Figure 3 in the Appendix.
Revised: An ablation experiment using only the magnitude mask or only the SNR mask for pruning is shown in Figure 3 in the Appendix.

Original: Pruning based only on the SNR mask leads to eventual overpruning and pruning based only on the MAG mask with a ﬁxed sparsity prunes too many weights too early
Revised: Pruning based only on the SNR mask leads to eventual overpruning and pruning based only on the MAG mask with a ﬁxed sparsity prunes too many weights too early.

Original: [Figure] We also visualize the intersection over union of consecutive masks in the lower plots of each Figure measured in terms of their intersection over union (IOU) before and after a mask update.
Revised: We also visualize the intersection over union of consecutive masks in the lower plots of each Figure measured in terms of their intersection over union (IOU) before and after a mask update.

Original: We show 1 - IOU in order to use a log scale as the mask changes happen at diﬀerent magnitudes during training.
Revised: We show 1 - IOU in order to use a log scale as the mask changes happen at diﬀerent magnitudes during training.

Original: Similarly to (Frankle et al., 2019b), we found that the weight magnitude mask (MAG) changes less towards the end of training.
Revised: Similarly to (Frankle et al., 2019b), we found that the weight magnitude mask (MAG) changes less towards the end of training.

Original: In contrast, the SNR mask changes more towards the end of training, which is to be expected as more weights will converge and hence their SNR will be less than 1.
Revised: In contrast, the SNR mask changes more towards the end of training, which is to be expected as more weights will converge and hence their SNR will be less than 1.

Original: The results for pruning a ResNet-18 on CIFAR-10 are shown in Figure 2.
Revised: The results for pruning a ResNet-18 on CIFAR-10 are shown in Figure 2.

Original: Our method has the same ﬁnal performance as DPF, but training is less noisy and at a similar speed as the dense model.
Revised: Our method has the same ﬁnal performance as DPF, but training is less noisy and at a similar speed as the dense model.

Original: 5.2 Tabular data
Revised: 5.2 Tabular data

Original: The results obtained on Electricity and Covertype from OpenML Vanschoren et al.
Revised: The results obtained on Electricity and Covertype from OpenML Vanschoren et al.

Original: (2013) are shown in the right half of Figure 1.
Revised: (2013) are shown in the right half of Figure 1.

Original: DPF performs better than our method and dense training for Covertype and on par to our method and dense on Electricity.
Revised: DPF performs better than our method and dense training for Covertype and on par to our method and dense on Electricity.

Original: 5.3 A diﬀerent weight update strategy
Revised: 5.3 A diﬀerent weight update strategy

Original: In the course of our experiments, we also implemented a diﬀerent strategy for weight updates which diﬀers from DPF (Lin et al., 2019).
Revised: In the course of our experiments, we also implemented a diﬀerent strategy for weight updates which diﬀers from DPF (Lin et al., 2019).

Original: For context: In DPF a weight w i may become unpruned if one of the following conditions holds:
Revised: For context: In DPF a weight θ i may become unpruned if one of the following conditions holds:

Original: If a diﬀerent weight w j decreases in magnitude, below the current cutoﬀ threshold, and the weight w i is close to this threshold then the weight is unpruned.
Revised: If a diﬀerent weight θ j decreases in magnitude, below the current cutoﬀ threshold, and the weight θ i is close to this threshold then the weight is unpruned.

Original: In DPF gradient updates are also performed for pruned weights, hence a weight may become unpruned if the gradient update pushes its magnitude back into the current magnitude mask.
Revised: In DPF, gradient updates are also performed for pruned weights, hence a weight may become unpruned if the gradient update pushes its magnitude back into the current magnitude mask.

Original: In our experiments, we investigated our proposed method and DPF if we only use condition (1), which hence reduces the chance of unpruning a weight during training.
Revised: In our experiments, we investigated our proposed method and DPF if we only use condition (1), which hence reduces the chance of unpruning a weight during training.

Original: The results are shown in Figures 5 and 6 in the Appendix.
Revised: The results are shown in Figures 5 and 6 in the Appendix.

Original: DPF is noticeably less noisy in this setting, otherwise the ﬁnal performances for our method and DPF are almost identical, apart from CIFAR-10 where DPF works worse in this setting.
Revised: DPF is noticeably less noisy in this setting, otherwise the ﬁnal performances for our method and DPF are almost identical, apart from CIFAR-10 where DPF works worse in this setting.

Original: For CIFAR-10, our method also reaches higher ﬁnal sparsities than in the results shown in Section 5.1.
Revised: For CIFAR-10, our method also reaches higher ﬁnal sparsities than in the results shown in Section 5.1.

Original: Since the SNR criterion makes sure that only small  converged weights with gradients resembling white noise are being pruned, we conjecture that updating them still after being pruned mostly adds disruptive noise, hence not updating them (dropping condition 2.)
Revised: Since the SNR criterion ensures that only small and converged weights, with gradients resembling white noise, are being pruned, we conjecture that updating them still after being pruned mostly adds disruptive noise into the network, hence not updating them (dropping condition 2.)

Original: smoothes the curves without loosing performance.
Revised: smoothes the curves without loosing performance.

Original: DFP, however, may suﬀer from this change, since weight magnitude pruning alone may prune weights that did not reach their ﬁnal value yet and hence updating them  is more crucial.
Revised: DFP in contrast may suﬀer from this change, since weight magnitude pruning alone may prune weights that did not reach their ﬁnal value yet and hence updating them (at the expense of a noisy learning curve) is more crucial.

Original: 6. Conclusion & Future Work
Revised: 6. Conclusion & Future Work

Original: We proposed the signal-to-noise ratio of mini-batch gradients as a new pruning criterion, which measures when a weight is no longer producing an informative gradient.
Revised: We proposed the signal-to-noise ratio of mini-batch gradients as a new pruning criterion, which measures when a weight is no longer producing an informative gradient.

Original: We explain how to extend
Revised: We explain how to extend

Original: DPF (Lin et al., 2019) using the new criterion which allows the network to decrease its sparsity if necessary, which makes the pruning more ﬂexible.
Revised: DPF (Lin et al., 2019) using the new criterion which allows the network to decrease its sparsity if necessary, which makes the pruning more ﬂexible.

Original: In our experiments, we demonstrate that we can obtain results on par with normal dense training and DPF on image and tabular data, while exposing no freeparameters apart from the maximum sparsity parameter.
Revised: In our experiments, we demonstrate that we can obtain results on par with normal dense training and DPF on image and tabular data, while exposing no free parameter.

Original: We leave as future work the question on how to compute the SNR for more complex networks and how to prevent overpruning more elegantly than using early stopping on the pruning masks.
Revised: We leave as future work the question on how to compute the SNR for more complex networks and how to prevent overpruning more elegantly than using early stopping on the pruning masks.

Original: 7. Broader impact
Revised: 7. Broader impact

Original: We see opportunities of our work for positive impact, by contributing to potential energy savings in training neural networks and making them more memory eﬃcient, hence reducing their carbon footprint and ﬁnancial cost.
Revised: We see opportunities of our work for positive impact, by contributing to potential energy savings in training neural networks and making them more memory eﬃcient, hence reducing their carbon footprint and ﬁnancial cost.

Original: However, our work may also lead to negative side eﬀects.
Revised: However, our work may also lead to negative side eﬀects.

Original: For example, while our method produces sparse networks, the resulting classiﬁers may not necessarily be fair and could for example give dominating groups in a population even greater weight, than non-sparse networks.
Revised: For example, while our method produces sparse networks, the resulting classiﬁers may not necessarily be fair and could for example give dominating groups in a population even greater weight, than non-sparse networks.

Original: Appendix A. Appendix A.1 Experimental details A.1.1 Image data
Revised: Appendix A. Appendix A.1 Experimental details A.1.1 Image data

Original: All experiments for MNIST and FashionMNIST use a small CNN with 4 convolutional layers with relu activation function, each followed by max pooling and a ﬁnal linear layer.
Revised: All experiments for MNIST and FashionMNIST use a small CNN with 4 convolutional layers with relu activation function, each followed by max pooling, and a ﬁnal linear layer.

Original: The number of ﬁlters doubles between consecutive ﬁlters with an initial ﬁlter count of 16.
Revised: The number of ﬁlters doubles between consecutive ﬁlters with an initial ﬁlter count of 16.

Original: An initial learning rate of 1e-2 which decays according to a cosine schedule (Loshchilov and
Revised: An initial learning rate of 1e-2 which decays according to a cosine schedule (Loshchilov and

Original: Hutter, 2016) to 1e-4 and is used with the Adam optimizer (Kingma and Ba).
Revised: Hutter, 2016) to 1e-4 and is used with the Adam optimizer (Kingma and Ba).

Original: An additional L regularization of 1e-4 is used for all experiments.
Revised: An additional L regularization of 1e-4 is used for all experiments.

Original: A.2 Tabular data
Revised: A.2 Tabular data

Original: We use multi-layer perceptrons with 7 layers and a width of 512 to be suﬃciently overparameterized.
Revised: We use multi-layer perceptrons with 7 layers and a width of 512 to be suﬃciently overparameterized.

Original: The optimization was carried out with the same parameters as for image data, however with an initial learning rate of 1e-1.
Revised: The optimization was carried out with the same parameters as for image data, however with an initial learning rate of 1e-1.

Original: A.3 Ablation study
Revised: A.3 Ablation study

Original: A.4 Mask update strategies
Revised: A.4 Mask update strategies

Original: DPF (Lin et al., 2019) used an extra hyperparameter search to determine the number of SGD steps between mask updates to be 16.
Revised: DPF (Lin et al., 2019) used an extra hyperparameter search to determine the number of SGD steps between mask updates to be 16.

Original: However, depending on the dataset and network architecture the default may no longer be optimal.
Revised: However, depending on the dataset and network architecture the default may no longer be optimal.

Original: Therefore, we investigate the impact of the frequency of mask updates on the optimization.
Revised: Therefore, we investigate the impact of the frequency of mask updates on the optimization.

Original: We compare dense training to updating the mask after every and after 32 and 128 SGD steps.
Revised: We compare dense training to updating the mask after every and after 32 and 128 SGD steps.

Original: Note, that in these experiments we do not perform any early stopping on the pruning level.
Revised: Note, that in these experiments we do not perform any early stopping on the pruning level.

Original: We ﬁnd that the number of mask updates has only a small inﬂuence on the optimization overall and only slightly changes the maximum attained sparsity level.
Revised: We ﬁnd that the number of mask updates has only a small inﬂuence on the optimization overall and only slightly changes the maximum attained sparsity level.

Original: More mask updates tend to lead to higher levels of sparsity in particular for the openml datasets electricity and covertype for which we use MLPs.
Revised: More mask updates tend to lead to higher levels of sparsity in particular for the openml datasets electricity and covertype for which we use MLPs.

Original: For the ConvNets used for MNIST and FMNIST .
Revised: For the ConvNets used for MNIST and FMNIST .

Original: A.5 Ablation on weight update strategy
Revised: A.5 Ablation on weight update strategy

Original: In the course of our experiments, we also implemented a diﬀerent strategy  forweight updates which diﬀers from DPF (Lin et al., 2019).
Revised: The eﬀect of the alternative weight update strategy as outlined in Section 5.3 in the main paper is shown in Figure 5 for CIFAR-10, and in Figure 6 for

Original: For context: In DPF a weight w i may become unpruned if one of the following conditions holds: If a diﬀerent weight w j decreases in magnitude, below the current cutoﬀ threshold, and the weight w i is close to this threshold then the weight is unpruned. In DPF gradient updates are also performed for pruned weights, hence a weight may become unpruned if the gradient update pushes its magnitude back into the current magnitude mask In our experiments, we investigated our proposed method andDPF if we only use condition (1).
Revised: , FashionMNIST, COVERTYPE and

Original: The results are shown in Figures 5 and 6. DPF is noticeably less noisy in this setting, otherwise the ﬁnal performances for our method and DPF are almost identical, apart from CIFAR-10 where DPF works worse in this setting. For CIFAR-10, our method also reaches higher ﬁnal sparsities than in the results shown in the main paper.
Revised: ELECTRICITY.

Original: A.6 Eﬀect of the number of burn-in epochs and the memory width of the exp. mov. avg of the gradient SNR
Revised: A.6 Eﬀect of the number of burn-in epochs and the memory width of the exp. mov. avg of the gradient SNR

Original: [Figure]
Revised: The gradient SNR can be a noisy quantity. Hence, in the algorithm we smooth it with an exponential moving average with a ﬁxed factor γ ∈ [0 , 1] over optimization steps as proposed by Mahsereci et al.

Original: In this section, wewill investigate what inﬂuence the parameters of the exponential moving average of the gradient SNR have on theoverall pruning algorithm.
Revised: In this section, we investigate what inﬂuence the parameter γ has on the pruning algorithm, and motivate a non-sensitive default.

Original: As the gradient SNR is highly noisy ﬁltering is needed and we follow Mahsereci et al. and use an exp.
Revised: The smoothed SNR is computed as follows: snr smootht = γsnr smootht −+ (1 − γ ) snr t ( θ ).

Original: mov.
Revised: The smoothing factor γ has a simple interpretation.

Original: avg per weight. The two main parameters are the number of burn-in steps ( bin steps ) after which we start pruning and the memory width used for the calculation of the width of  the exp.
Revised: Roughly after m steps, the contribution of the SNR value snr t − m of previous iteration t − m will have decayed to about γ m percent relative to the current value snr t .

Original: mov. avg.
Revised: This number of iterations m is roughly the ‘memory’ of the smoother.

Original: The memorywidth ( m ) is tied tothe total number of optimization iterations/SGD steps ( N ) and calculated as follows: [Equation] m can beinterpreted as  the maximum ratio ofsteps we can shift the optimization iteration when the gradient SNR moves from >to < = 1.
Revised: We want the memory to be as large as possible (to reduce as much noise as possible) much small enough such that the smoothed SNR is ﬂexible, that is the memory should be small in comparison to the total length T of the optimization run.

Original: From this expression we obtain the gamma used in exp.
Revised: The formula is thus γ = (0 . 01) − m for a residual contribution of 1%.

Original: mov. avg. of the SNR.
Revised: The relative memory r m := mT can be set to a meaningful default.

Original: Note, that the smaller  m , the smaller gamma resulting in less smoothing.
Revised: The smaller r m , the smaller γ resulting in less smoothing.

Revised: A second parameter is the number of burn-in steps bin steps after which we start pruning.

Original: The results of the ablation are shown in Figure 7 for varying  m in [0.01, 0.1] and bin steps in [1, 10]. We ﬁnd that, the ﬁnal sparsity level is raised by a lower m , which makes sense as the SNR will only fall below one when gradients become close to zero andhence less smoothing results in the average to fall quicker below 1.
Revised: The results of the ablation are shown in Figure 7 for varying r m ∈ { 0 . 02 , 0 . 1 } which equals 1/50-th and 1/10-th of the total run respectively, and bin steps ∈ { 1 , 10 } . We ﬁnd that a large window of r m = 1 / 10 prunes too cautiously, and the smaller window r m = 1 / 50 works well across all tested datasets and networks.

Original:  bin steps has the eﬀect to mainly change the initial sparsity level.
Revised: The number of burning steps bin steps seems to have no eﬀect on the ﬁnal pruned network.

Original: As defaults for all experiments we chose  m = 0.01 and bin steps = 1 epoch.
Revised: Hence, as defaults for all experiments we chose r m = 0 . 02 = 1 / 50 and bin steps =epoch.

Original: A.7 Layer-wise pruning ratios of our method
Revised: A.7 Layer-wise pruning ratios of our method

Original: In Figures 8 and 9 we visualize the sparsity per layer at each training epoch for our proposed method.
Revised: In Figures 8 and 9 we visualize the sparsity per layer at each training epoch for our proposed method.

Original: We ﬁnd that the SNR mask tends to want to prune more layers in the layers closer to the input and less closer to the output, while the reverse is true for the MAG mask.
Revised: We ﬁnd that the SNR mask tends to want to prune more layers in the layers closer to the input and less closer to the output, while the reverse is true for the MAG mask.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]