Original: R IGGING THE L OTTERY : M AKING A LL T ICKETS W INNERS
Revised: R IGGING THE L OTTERY : M AKING A LL T ICKETS W INNERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Sparse neural networks have been shown to be computationally efﬁcient  networks with improved inference times.
Revised: Sparse neural networks have been shown to be more parameter and compute efﬁcient compared to dense networks and in some cases they are even successfully used to decrease wall clock inference times.

Original: There is a large body of work on training dense networks to yield sparse networks for inference (Molchanov et al., 2017; Zhu & 
Revised: There is a large body of work on training dense networks to yield sparse networks for inference (Molchanov et al., 2017; Zhu & Gupta, 2018;

Original: Gupta, 2018; Louizos et al., 2017; Li et al., 2016; Guo et al., 2016).
Revised: Louizos et al., 2017; Li et al., 2016; Guo et al., 2016).

Original: This limits the size of the largest trainable sparse model to that of the largest trainable dense model.
Revised: This limits the size of the largest trainable sparse model to that of the largest trainable dense model.

Original: In this paper we introduce a method to train sparse neural networks with a ﬁxed parameter count and a ﬁxed computational cost throughout training, without sacriﬁcing accuracy relative to existing dense-to-sparse training methods.
Revised: In this paper we introduce a method to train sparse neural networks with a ﬁxed parameter count and a ﬁxed computational cost throughout training, without sacriﬁcing accuracy relative to existing dense-to-sparse training methods.

Original: Our method updates the topology of the network during training by using parameter magnitudes and infrequent gradient calculations.
Revised: Our method updates the topology of the network during training by using parameter magnitudes and infrequent gradient calculations.

Original: We show that this approach requires less ﬂoating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques.
Revised: We show that this approach requires fewer ﬂoating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques.

Original: We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNetdataset.
Revised: We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNet-2012 dataset, WideResNets on the CIFAR-10 dataset and RNNs on the

Original:  Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static.
Revised: WikiText-103 dataset. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The parameter and FLOP efﬁciency of sparse neural networks is now well demonstrated on a variety of problems (Han et al., 2015; Srinivas et al., 2017).
Revised: The parameter and ﬂoating point operation (FLOP) efﬁciency of sparse neural networks is now well demonstrated on a variety of problems (Han et al., 2015; Srinivas et al., 2017).

Original: Some work has even shown inference time speedups are possible on Recurrent Neural Networks (RNNs)
Revised: Some work has even shown inference time speedups are possible on Recurrent Neural Networks (RNNs)

Original: (Kalchbrenner et al., 2018) and Convolutional Neural Networks (ConvNets) (Park et al., 2016).
Revised: (Kalchbrenner et al., 2018) and Convolutional Neural Networks (ConvNets) (Park et al., 2016).

Original: Currently, the most accurate sparse models are obtained with techniques that require, at a minimum, the cost of training a dense model in terms of memory and FLOPs (Zhu & Gupta, 2018;
Revised: Currently, the most accurate sparse models are obtained with techniques that require, at a minimum, the cost of training a dense model in terms of memory and FLOPs (Zhu & Gupta, 2018;

Original: Guo et al., 2016), and sometimes signiﬁcantly more (Molchanov et al., 2017).
Revised: Guo et al., 2016), and sometimes signiﬁcantly more (Molchanov et al., 2017).

Original: This paradigm has two main limitations:
Revised: This paradigm has two main limitations:

Original: The maximum size of sparse models is limited to the largest dense model that can be trained.
Revised: The maximum size of sparse models is limited to the largest dense model that can be trained.

Original: Even if sparse models are more parameter efﬁcient, we can’t use pruning to train models that are larger and more accurate than the largest possible dense models.
Revised: Even if sparse models are more parameter efﬁcient, we can’t use pruning to train models that are larger and more accurate than the largest possible dense models.

Original: It is inefﬁcient.
Revised: It is inefﬁcient.

Original: Large amounts of computation must be performed for parameters that are zero valued or that will be zero during inference.
Revised: Large amounts of computation must be performed for parameters that are zero valued or that will be zero during inference.

Original: Additionally, it remains unknown if the performance of the current best pruning algorithms are an upper bound on the quality of sparse models.
Revised: Additionally, it remains unknown if the performance of the current best pruning algorithms are an upper bound on the quality of sparse models.

Original: Gale et al.
Revised: Gale et al.

Original: (2019) found that three different  algorithms all achieve about the same sparsity / accuracy trade-off.
Revised: (2019) found that three different dense-tosparse training algorithms all achieve about the same sparsity / accuracy trade-off.

Original: However, this is far from conclusive proof that no better performance is possible.
Revised: However, this is far from conclusive proof that no better performance is possible.

Original: In this work we show the surprising result that dynamic sparse training, which includes the method we introduce below, can ﬁnd more accurate models than the current best approaches to pruning initially dense networks.
Revised: In this work we show the surprising result that dynamic sparse training, which includes the method we introduce below, can ﬁnd more accurate models than the current best approaches to pruning initially dense networks.

Original: Importantly, our method does not change the FLOPs required to execute the model during training, allowing one to decide on a speciﬁc inference cost prior to training.
Revised: Importantly, our method does not change the FLOPs required to execute the model during training, allowing one to decide on a speciﬁc inference cost prior to training.

Original: The Lottery Ticket Hypothesis (Frankle & Carbin, 2019) hypothesized that if we can ﬁnd a sparse neural network with iterative pruning, then we can train that sparse network from scratch, to the same level of accuracy, by starting from the original initial conditions .
Revised: The Lottery Ticket Hypothesis (Frankle & Carbin, 2019) hypothesized that if we can ﬁnd a sparse neural network with iterative pruning, then we can train that sparse network from scratch, to the same level of accuracy, by starting from the original initial conditions .

Original: In this paper we introduce a new method for training sparse models without the need of a “lucky” initialization; for this reason, we call our method “The Rigged Lottery” or RigL ∗ .
Revised: In this paper we introduce a new method for training sparse models without the need of a “lucky” initialization; for this reason, we call our method “The Rigged Lottery” or RigL ∗ .

Original: We show that this method is:
Revised: We show that this method is:

Original: • Memory efﬁcient : It requires memory only proportional to the size of the sparse model.
Revised: • Memory efﬁcient : It requires memory only proportional to the size of the sparse model.

Original: It never requires storing quantities that are the size of the dense model.
Revised: It never requires storing quantities that are the size of the dense model.

Original: This is in contrast to Dettmers & Zettlemoyer (2019) which requires storing the momentum for all parameters, even those that are zero valued.
Revised: This is in contrast to Dettmers & Zettlemoyer (2019) which requires storing the momentum for all parameters, even those that are zero valued.

Original: • Computationally efﬁcient : The amount of computation required to train the model is proportional to the number of nonzero parameters in the model.
Revised: • Computationally efﬁcient : The amount of computation required to train the model is proportional to the number of nonzero parameters in the model.

Original: • Accurate : The performance achieved by the method matches and sometimes exceeds the performance of pruning based approaches.
Revised: • Accurate : The performance achieved by the method matches and sometimes exceeds the performance of pruning based approaches.

Original: Our method works by infrequently using instantaneous gradient information to inform a re-wiring of the network.
Revised: Our method works by infrequently using instantaneous gradient information to inform a re-wiring of the network.

Original: We show that this allows the optimization to escape local minima where it would otherwise become trapped if the sparsity pattern were to remain static.
Revised: We show that this allows the optimization to escape local minima where it would otherwise become trapped if the sparsity pattern were to remain static.

Original: Crucially, as long as the full gradient information is needed less than every
Revised: Crucially, as long as the full gradient information is needed less than every

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: − sparsity iterations, then the overall work remains proportional to the model sparsity.
Revised: sparsity iterations, then the overall work remains proportional to the model sparsity.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Research on ﬁnding sparse neural networks dates back  at least three decades, where Mozer &
Revised: Research on ﬁnding sparse neural networks dates back decades; for example, at least to Thimm &

Original: Smolensky (1989) concluded that pruning weights based on magnitude was a simple and powerful technique.
Revised: Fiesler (1995) who concluded that pruning weights based on magnitude was a simple and powerful technique.

Original: Str¨om (1997) later introduced the idea of retraining the previously pruned network to increase accuracy.
Revised: Str¨om (1997) later introduced the idea of retraining the previously pruned network to increase accuracy.

Original: Han et al.
Revised: Han et al.

Original: (2016b) went further and introduced multiple rounds of magnitude pruning and retraining.
Revised: (2016b) went further and introduced multiple rounds of magnitude pruning and retraining.

Original: This is, however, relatively inefﬁcient, requiring ten rounds of retraining when removing 20% of the connections to reach a ﬁnal sparsity of 90% .
Revised: This is, however, relatively inefﬁcient, requiring ten rounds of retraining when removing 20% of the connections to reach a ﬁnal sparsity of 90% .

Original: To overcome this problem,
Revised: To overcome this problem,

Original: Narang et al.
Revised: Narang et al.

Original: (2017) introduced gradual pruning, where connections are slowly removed over the course of a single round of training.
Revised: (2017) introduced gradual pruning, where connections are slowly removed over the course of a single round of training.

Original: Zhu & Gupta (2018) reﬁned the technique to minimize the amount of hyper-parameter selection required.
Revised: Zhu & Gupta (2018) reﬁned the technique to minimize the amount of hyper-parameter selection required.

Original: A diversity of approaches not based on magnitude based pruning have also been proposed.
Revised: A diversity of approaches not based on magnitude based pruning have also been proposed.

Original: LeCun et al.
Revised: LeCun et al.

Original: and Hassibi & Stork (1993) are some early examples, but impractical for modern neural networks as they use information from the Hessian to prune a trained network.
Revised: and Hassibi & Stork (1993) are some early examples, but impractical for modern neural networks as they use information from the Hessian to prune a trained network.

Original: More recent work includes L 0 Regularization (Christos Louizos, 2018), Variational Dropout (Molchanov et al., 2017),
Revised: More recent work includes L 0 Regularization (Christos Louizos, 2018), Variational Dropout (Molchanov et al., 2017),

Original: Dynamic Network Surgery (Guo et al., 2016) and Sensitivity Driven Regularization (Tartaglione et al., 2018).
Revised: Dynamic Network Surgery (Guo et al., 2016) and Sensitivity Driven Regularization (Tartaglione et al., 2018).

Original: Gale et al.
Revised: Gale et al.

Original: (2019) examined magnitude pruning, L 0 Regularization and Variational
Revised: (2019) examined magnitude pruning, L 0 Regularization and Variational

Original: Dropout and concluded that they all achieve about the same accuracy versus sparsity tradeoffs on
Revised: Dropout and concluded that they all achieve about the same accuracy versus sparsity trade-off on

Original: ResNet-50 and Transformer architectures.
Revised: ResNet-50 and Transformer architectures.

Original: Training techniques that allow for sparsity throughout the entire training process were, to our knowledge, ﬁrst introduced in Deep Rewiring (DeepR) (Bellec et al., 2017).
Revised: Training techniques that allow for sparsity throughout the entire training process were, to our knowledge, ﬁrst introduced in Deep Rewiring (DeepR) (Bellec et al., 2017).

Original: In DeepR, the standard
Revised: In DeepR, the standard

Original: Stochastic Gradient Descent (SGD) optimizer is augmented with a random walk in parameter space.
Revised: Stochastic Gradient Descent (SGD) optimizer is augmented with a random walk in parameter space.

Original: Additionally, connections have a pre-deﬁned sign assigned at random; when the optimizer would normally ﬂip the sign, the weight is set to 0 instead and new weights are activated at random.
Revised: Additionally, connections have a pre-deﬁned sign assigned at random; when the optimizer would normally ﬂip the sign, the weight is set to 0 instead and new weights are activated at random.

Original: Sparse Evolutionary Training (SET) (Mocanu et al., 2018) proposed a simpler scheme where weights are pruned according to the standard magnitude criterion used in pruning and  added back at random.
Revised: Sparse Evolutionary Training (SET) (Mocanu et al., 2018) proposed a simpler scheme where weights are pruned according to the standard magnitude criterion used in pruning and are added back at random.

Original: The method is simple and achieves reasonable performance in practice.
Revised: The method is simple and achieves reasonable performance in practice.

Original: Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang, 2019) introduces the idea of allowing the parameter budget to shift between different layers of the model, allowing for non-uniform sparsity.
Revised: Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang, 2019) introduced the idea of allowing the parameter budget to shift between different layers of the model, allowing for non-uniform sparsity.

Original: This allows the model to put parameters where they are most effective.
Revised: This allows the model to distribute parameters where they are most effective.

Original: Unfortunately, the models under consideration are mostly ConvNets, so the result of this parameter reallocation (which is to decrease the sparsity of early layers and increase the sparsity of later layers) has the overall effect of increasing the FLOP count because the spatial size is largest at the beginning.
Revised: Unfortunately, the models under consideration are mostly convolutional networks, so the result of this parameter reallocation (which is to decrease the sparsity of early layers and increase the sparsity of later layers) has the overall effect of increasing the FLOP count because the spatial size is largest at the beginning.

Original: Sparse Networks from Scratch (SNFS) (Dettmers & Zettlemoyer, 2019) introduces the idea of using the momentum of each parameter as the criterion to be used for growing weights and demonstrates it leads to an improvement in test accuracy. Like DSR, they allow the sparsity of each layer to change and focus on a constant Pronounced 
Revised: Sparse Networks from Scratch (SNFS) (Dettmers & Zettlemoyer, 2019) introduces the idea of using the momentum of each parameter as the criterion to be used for growing weights and demonstrates it ∗ Pronounced ”wriggle”.

Original: [Table]
Revised: [Table]

Original:  parameter, not FLOP, budget.
Revised: leads to an improvement in test accuracy. Like DSR, they allow the sparsity of each layer to change and focus on a constant parameter, not FLOP, budget.

Original: Importantly, the method requires computing gradients and updating the momentum for every parameter in the model, even those that are zero, at every iteration.
Revised: Importantly, the method requires computing gradients and updating the momentum for every parameter in the model, even those that are zero, at every iteration.

Original: This can result in a signiﬁcant amount of overall computation.
Revised: This can result in a signiﬁcant amount of overall computation.

Original: Additionally, depending on the model and training setup, the required storage for the full momentum tensor could be prohibitive.
Revised: Additionally, depending on the model and training setup, the required storage for the full momentum tensor could be prohibitive.

Original: Single-Shot Network Pruning (SNIP)
Revised: Single-Shot Network Pruning (SNIP)

Original: (Lee et al., 2019) attempts to ﬁnd an initial mask by calculating the full gradients only once and pruning based on the gradient magnitudes, then proceeding to train with this static sparse network.
Revised: (Lee et al., 2019) attempts to ﬁnd an initial mask with one-shot pruning and uses the saliency score of parameters to decide which parameters to keep. After pruning training proceeds with this static sparse network.

Original: Properties of the different sparse training techniques are summarized in Table 1.
Revised: Properties of the different sparse training techniques are summarized in Table 1.

Original: There has also been a line of work investigating the Lottery Ticket Hypothesis (Frankle & Carbin, 2019).
Revised: There has also been a line of work investigating the Lottery Ticket Hypothesis (Frankle & Carbin, 2019).

Original: Frankle et al.
Revised: Frankle et al.

Original: (2019) showed that the formulation must be weakened to apply to larger networks such as ResNet-50 (He et al., 2015).
Revised: (2019) showed that the formulation must be weakened to apply to larger networks such as ResNet-50 (He et al., 2015).

Original: In large networks, instead of the original initialization, the values after thousands of optimization steps must be used for initialization.
Revised: In large networks, instead of the original initialization, the values after thousands of optimization steps must be used for initialization.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2019) showed that lottery tickets obtain non-random accuracies even before the training has started.
Revised: (2019) showed that lottery tickets obtain non-random accuracies even before the training has started.

Original: Though the possibility of training sparse neural networks with a ﬁxed sparsity mask using lottery tickets is intriguing, it remains unclear whether it is possible to generate such initializations –  bothof masks and parameters – de novo .
Revised: Though the possibility of training sparse neural networks with a ﬁxed sparsity mask using lottery tickets is intriguing, it remains unclear whether it is possible to generate such initializations – for both masks and parameters – de novo .

Original:  R IGGING T HE L OTTERY
Revised:  R IGGING T HE L OTTERY

Original: Our method, RigL , is illustrated in Figure 1.
Revised: Our method, RigL , is illustrated in Figure 1.

Original: At regularly spaced intervals our method removes a fraction of connections based on weight magnitudes and activates new ones using instantaneous gradient information.
Revised: At regularly spaced intervals our method removes a fraction of connections based on weight magnitudes and activates new ones using instantaneous gradient information.

Original: After updating the connectivity, training continues with the updated network until the next update.
Revised: After updating the connectivity, training continues with the updated network until the next update.

Original: The main parts of our algorithm, Sparsity Distribution , Update Schedule ,
Revised: The main parts of our algorithm, Sparsity Distribution , Update Schedule ,

Original: Drop Criterion , Grow Criterion , and the various options we considered for each, are explained below.
Revised: Drop Criterion , Grow Criterion , and the various options we considered for each, are explained below.

Original: The improved performance of RigL is due to two reasons: the use of a new method for activating connections that is efﬁcient and more effective than choosing at random, and the use of a natural extension to an existing method for distributing parameters statically among convolutional layers.
Revised: The improved performance of RigL is due to two reasons: the use of a new method for activating connections that is efﬁcient and more effective than choosing at random, and the use of a natural extension to an existing method for distributing parameters statically among convolutional layers.

Original: (0) Notation.
Revised: (0) Notation.

Original: Given a dataset D with individual inputs x i and targets y i , one can train a neural network to minimize the loss function (cid:80) i L ( f θ ( x i ) , y i ) , where f θ ( x ) is the neural network withparameters θ of length N . The vector θ can be decomposed into parameters θ l , of length N l , for
Revised: Given a dataset D with individual inputs x i and targets y i , one can train a neural network to minimize the loss function (cid:80) i L ( f θ ( x i ) , y i ) , where f θ ( x ) is the neural network with

Original: [Figure]
Revised: [Figure]

Original:  each layer l .
Revised: parameters θ of length N . The vector θ can be decomposed into parameters θ l , of length N l , for each layer l .

Original: A sparse network keeps only a fraction D ∈ (0 , 1) of all connections, resulting in a sparsity of S = 1 − D .
Revised: A sparse network keeps only a fraction D ∈ (0 , 1) of all connections, resulting in a sparsity of S = 1 − D .

Original: More precisely, denoting the sparsity of individual layers by s l , the total parameter count of the sparse neural network satisﬁes (cid:80) l (1 − s l ) N l = (1 − S ) ∗ N .
Revised: More precisely, denoting the sparsity of individual layers with s l , the total parameter count of the sparse neural network satisﬁes (cid:80) l (1 − s l ) N l = (1 − S ) ∗ N .

Original: (1) Sparsity Distribution.
Revised: (1) Sparsity Distribution.

Original: There are many ways of distributing the non-zero weights across the layers while satisfying the equality above.
Revised: There are many ways of distributing the non-zero weights across the layers while satisfying the equality above.

Original: We avoid re-allocating parameters between layers during the training process as it makes it difﬁcult to target a speciﬁc ﬁnal FLOP budget, which is important for many inference applications.
Revised: We avoid re-allocating parameters between layers during the training process as it makes it difﬁcult to target a speciﬁc ﬁnal FLOP budget, which is important for many inference applications.

Original: We consider the following three strategies:
Revised: We consider the following three strategies:

Original: Uniform: The sparsity s l of each individual layer is the same as the total sparsity S , except for the ﬁrst and last layers; following the strategy of Gale et al.
Revised: Uniform: The sparsity s l of each individual layer is the same as the total sparsity S .

Original: (2019), we ﬁx the sparsity of the last layer to 0.8 and keep the ﬁrst layeras a dense layer.
Revised: We keep the ﬁrst layer dense ( s 0 = 0 ), since it has negligible number of parameters.

Original: Erd˝os-R´enyi:
Revised: Erd˝os-R´enyi:

Original: As introduced in Mocanu et al.
Revised: As introduced in Mocanu et al.

Original: (2018), s l scales proportionally to the ratio n l − 1 + n l n l − 1 ∗ n l , where n l denotes number of neurons at layer l .
Revised: (2018), s l scales with 1 − n l − 1 + n l n l − 1 ∗ n l , where n l denotes number of neurons at layer l .

Original: This enables the number of connections in a sparse layer to scale with the  number of output  channels.
Revised: This enables the number of connections in a sparse layer to scale with the sum of the number of output and input channels.

Original: Erd˝os-R´enyi-Kernel (ERK): This method modiﬁes the original Erd˝os-R´enyi formulation by including the kernel dimensions in the scaling factors.
Revised: Erd˝os-R´enyi-Kernel (ERK): This method modiﬁes the original Erd˝os-R´enyi formulation by including the kernel dimensions in the scaling factors.

Original: In other words, the number of parameters of the sparse convolutional layers are scaled by n l − 1 + n l + w l + h l n l −1 ∗ n l ∗ w l ∗ h l, where w l and h l are the width and the height of the l ’th convolutional kernel.
Revised: In other words, the number of parameters of the sparse convolutional layers are scaled proportional to 1 − n l − 1 + n l + w l + h l n l − ∗ n l ∗ w l ∗ h l where w l and h l are the width and the height of the l ’th convolutional kernel.

Original: Sparsity of the fully connected layers scale as in the original Erd ˝ os-R ´ enyi formulation.
Revised: Sparsity of the fully connected layers scale as in the original Erd ˝ os-R ´ enyi formulation.

Original: Similar to Erd ˝ os-R ´ enyi, ERK allocates higher sparsities to the layers with more parameterswhile allocating lower sparsities
Revised: Similar to Erd ˝ osR ´ enyi, ERK allocates higher sparsities to the layers with more parameters

Original:  to the smaller ones.
Revised: while allocating lower sparsities to the smaller ones.

Original: In all methods, the bias and batch-norm parameters are kept dense.
Revised: In all methods, the bias and batch-norm parameters are kept dense.

Original: (2) Update Schedule.
Revised: (2) Update Schedule.

Original: The update schedule is deﬁned by the following parameters: (1) the number of iterations between sparse connectivity updates ( ∆ I ), (2) the iteration at which to stop updating the sparse connectivity ( I end ), (3) the initial fraction of connections updated ( α ) and (4) a function f decay , invoked every ∆ I iterations until I end , possibly decaying the fraction of updated connections over time.
Revised: The update schedule is deﬁned by the following parameters: (1) the number of iterations between sparse connectivity updates ( ∆ T ), (2) the iteration at which to stop updating the sparse connectivity ( T end ), (3) the initial fraction of connections updated ( α ) and (4) a function f decay , invoked every ∆ T iterations until T end , possibly decaying the fraction of updated connections over time.

Original: For the latter we choose to use cosine annealing, as we ﬁnd it slightly outperforms the other methods considered.
Revised: For the latter we choose to use cosine annealing, as we ﬁnd it slightly outperforms the other methods considered.

Original: [Equation]
Revised: [Equation]

Original: Alternatives to cosine annealing like a constant schedule and inverse power annealing are studied in the Appendix D.
Revised: Alternatives to cosine annealing like a constant schedule and inverse power annealing are studied in the Appendix F.

Original: (3) Drop criterion.
Revised: (3) Drop criterion.

Original: Over the course of training, we drop the lowest magnitude weights according to the update schedule.
Revised: Over the course of training, we drop the lowest magnitude weights according to the update schedule since they are expected to effect the training loss least.

Original: Speciﬁcally, we drop the connections given by
Revised: Speciﬁcally, we drop the connections given by

Original: [Equation]
Revised: [Equation]

Original: (4) Grow criterion.
Revised: (4) Grow criterion.

Original: The novelty of our method lies in how we grow new connections.
Revised: The novelty of our method lies in how we grow new connections.

Original: We grow the connections with highest magnitude gradients, TopKw/ ∈
Revised: We grow the connections with highest magnitude gradients, TopK

Original: θ lactive ( | grad ( θ l ) | , f decay ( t )(1 − s l ) N l ) , where θ lactive is the set of active connections after the drop step.
Revised: [Equation] where θ lactive is the set of active connections after the drop step.

Original: Newly activated connections are initialized to zero.
Revised: Newly activated connections are initialized to zero and therefore don’t effect the output of the network.

Revised: However they are expected to receive gradients with high magnitudes in the next iteration and therefore reduce the loss fastest.

Original: This procedure can be applied to each layer in sequence and the dense gradients can be discarded immediately after selecting the top connections.
Revised: This procedure can be applied to each layer in sequence and the dense gradients can be discarded immediately after selecting the top connections.

Original: If a layer is too large to materialize the full gradient with respect to the weights, then we can further reduce the memory requirements by performing an iterative calculation:
Revised: If a layer is too large to materialize the full gradient with respect to the weights, then we can further reduce the memory requirements by performing an iterative calculation:

Original: Initialize the set TK = {} . 2.
Revised: Initialize the set TK = {} . 2.

Original: Materialize a subset of size M of the full gradient, which we denote G i : i + M . 3.
Revised: Materialize a subset of size M of the full gradient, which we denote G i : i + M . 3.

Original: Update TK to contain the Top-K elements of G i : i + M concatenated with TK . 4.
Revised: Update TK to contain the Top-K elements of G i : i + M concatenated with TK . 4.

Original: Repeat steps 1 through 3 untilthe all of the gradients have been materialized.
Revised: Repeat steps 1 through 3 until all of the gradients have been materialized.

Original: The ﬁnal set TK contains the connections we wish to grow. 
Revised: The ﬁnal set TK contains the connections we wish to grow. [Figure]

Original:  As long as ∆ I > 11 − s the total work in calculating dense gradients is amortized and still proportional to  S .
Revised: Performance on ImageNet- [Figure] As long as ∆ T > 11 − s the total work in calculating dense gradients is amortized and still proportional to 1 − S .

Original: This is in contrast to the method of Dettmers & Zettlemoyer (2019), which requires calculating and storing the full gradients at each optimization step. [Figure] Imagenet [Figure]
Revised: This is in contrast to the method of Dettmers & Zettlemoyer (2019) which requires calculating and storing the full gradients at each optimization step

Original:  E MPIRICAL E VALUATION
Revised:  E MPIRICAL E VALUATION

Original: Our experiments focus on the ImageNet-2012 dataset and we use the TensorFlow Model Pruning library (Zhu & Gupta, 2018) to sparsify our networks.
Revised: Our experiments include image classiﬁcation using CNNs on the ImageNet-2012 (Russakovsky et al., 2015) and CIFAR-10 (Krizhevsky et al.) datasets and character based language modelling using RNNs with the WikiText-103 dataset (Merity et al., 2016). We use the TensorFlow Model Pruning library (Zhu & Gupta, 2018) for our pruning baselines.

Original: A Tensorﬂow (Abadi et al., 2015) implementation of our method along with three other baselines (SET, SNFS, SNIP) will be open sourced.
Revised: A Tensorﬂow (Abadi et al., 2015) implementation of our method along with three other baselines (SET, SNFS, SNIP) will be open sourced.

Revised: When we increase the training steps by a factor M , the anchor epochs of the learning rate schedule and the end iteration of the mask update schedule are also scaled by the same factor; we indicate this scaling with a subscript (e.g. RigL M × ).

Revised: 4. I MAGE N ET -2012 D ATASET

Original: In all experiments  we use SGD with momentum as our optimizer.
Revised: In all experiments in this section, we use SGD with momentum as our optimizer.

Original: We set the momentum  of the optimizer to 0.9, L 2 regularization coefﬁcient to 0.0001, and label smoothing  to 0.1.
Revised: We set the momentum coefﬁcient of the optimizer to 0.9, L 2 regularization coefﬁcient to 0.0001, and label smoothing (Szegedy et al., 2016) to 0.1.

Original: The learning rate schedule starts with a linear warm up reaching its maximum value of 1.6 at epoch 5 which is then dropped by a factor of 10 at epochs 30, 70 and 90.
Revised: The learning rate schedule starts with a linear warm up reaching its maximum value of 1.6 at epoch 5 which is then dropped by a factor of 10 at epochs 30, 70 and 90.

Original: We train our networks with a batch size of 4096 for 32000 steps which roughly corresponds to 100 epochs of training.
Revised: We train our networks with a batch size of 4096 for 32000 steps which roughly corresponds toepochs of training.

Original: When we increase the training steps by a factor x , the anchor epochs of the learning rate schedule are also scaled by the same factor; we indicate this scaling with a subscript (e.g. RigL 5 × ).

Original: Our training pipeline uses standard data augmentation, which includes random ﬂips and crops.
Revised: Our training pipeline uses standard data augmentation, which includes random ﬂips and crops.

Original: 4. R ES N ET - 
Revised: 4.1. R ES N ET -

Original: Figure 2-left summarizes the performance of various methodstraining on  an 80% sparse ResNet50.
Revised: Figure 2-left summarizes the performance of various methods on training an 80% sparse ResNet-50.

Original: We also train small dense networks with equivalent parameter count.
Revised: We also train small dense networks with equivalent parameter count.

Original: All sparse networks use the constant sparsity distribution and a cosine update schedule ( α = 0 . 1 , ∆ I = 50 ).
Revised: All sparse networks use the constant sparsity distribution and a cosine update schedule ( α = 0 . 3 , ∆ T = 100 ).

Original: Overall, we observe that the performance of all methods improves with training time; thus, for each method we run extended training with up to 5 × the training steps of the original.
Revised: Overall, we observe that the performance of all methods improves with training time; thus, for each method we run extended training with up to 5 × the training steps of the original.

Original: The calculation of the number of FLOPs required by each method are detailed in Appendix E.

Original: As noted by Gale et al.
Revised: As noted by Gale et al.

Original: (2019), Evci et al.
Revised: (2019), Evci et al.

Original: Frankle et al.
Revised: Frankle et al.

Original: and Mostafa & Wang (2019), training a  ﬁxed sparsitynetwork from scratch ( Static ) leads to inferior performance.
Revised: and Mostafa & Wang (2019), training a network with ﬁxed sparsity from scratch ( Static ) leads to inferior performance.

Original: Training a small dense network with the same number of parameters gets better results than Static , butit fails to match the performance of dynamic sparse models. 
Revised: Training a small dense network with the same number of parameters gets better results than Static , but fails to match the performance of dynamic sparse models. Similarly SET improves the performance over Small-Dense , however saturates around 75% accuracy indicating the limits of growing new connec- [Table]

Original: All dynamic sparse methods are able to achieve high accuracies, but RigL achieves the highest accuracy and does so while consistently requiring fewer FLOPs than the other methods.
Revised: tions randomly. Methods that use gradient information to grow new connections ( RigL and SNFS ) obtain higher accuracies, but RigL achieves the highest accuracy and does so while consistently requiring fewer FLOPs than the other methods.

Original: Given that different applications or scenarios might require a limit on the number of FLOPs for inference, we investigate the performance of our method at various sparsity levels.
Revised: Given that different applications or scenarios might require a limit on the number of FLOPs for inference [Table] we investigate the performance of our method at various sparsity levels.

Original: As mentioned previously, one strength of our method is that its resource requirements are constant throughout training and  can be chosen before training begins.
Revised: As mentioned previously [Table] one strength of our method is that its resource requirements are constant throughout training and we can choose the level of sparsity that ﬁts our training and/or inference constraints.

Original: In Figure 2-right we show the performance of our method at different sparsities and compare them with the pruning results of Gale et al.
Revised: In Figure 2-right we show the performance of our method at different sparsities and compare them with the pruning results of Gale et al.

Original: (2019), which uses 1.5x training steps, relative to the original 32k.
Revised: (2019), which uses 1.5x training steps, relative to the original 32k iterations.

Original: To make a fair comparison with regards to FLOPs, we scale the learning schedule of all other methods by 5x.
Revised: To make a fair comparison with regards to FLOPs, we scale the learning schedule of all other methods by 5x.

Original: Note that even after extending the training, it takes less 
Revised: Note that even after extending the training, it takes less FLOPs to train sparse networks using rigL (except for the 80% sparse RigL -ERK) compared to the pruning method.

Original: [Table] RigL , our method with constant sparsity distribution, follows pruningvery closely in all sparsity levels.
Revised: RigL , our method with constant sparsity distribution, exceeds the performance of magnitude based iterative pruning in all sparsity levels while requiring less FLOPs to train.

Original: Sparse networks that use Erd˝os-Renyi-Kernel (ERK) sparsity distribution, however, obtain a remarkable performance that exceeds the performance of pruning based models.
Revised: Sparse networks that use Erd˝os-Renyi-Kernel (ERK) sparsity distribution obtains even greater performance. For example ResNet-50 with 96.5% sparsity achieves a remarkable 72.75% Top-1 Accuracy, around 3.5% higher than the extended magnitude pruning results reported by Gale et al.

Original: As observed earlier,training either smaller dense models (with the same number of parameters) orstatic sparse models fails to perform at a comparable level.
Revised: As observed earlier, smaller dense models (with the same number of parameters) or sparse models with a static connectivity can not perform at a comparable level.

Original: A more ﬁne grained comparison of sparse training methods is presented in Table 2.
Revised: A more ﬁne grained comparison of sparse training methods is presented in Table 2.

Original: Methods using uniform sparsity distribution and whose FLOP/memory footprint scales directly with the sparsity of the model are placed in the ﬁrst sub-group of the table.
Revised: Methods using uniform sparsity distribution and whose FLOP/memory footprint scales directly with (1-S) are placed in the ﬁrst sub-group of the table.

Original: The second sub-group includes DSR and networks with ERK sparsity distribution which require a higher number of FLOPs for inference.
Revised: The second sub-group includes DSR and networks with ERK sparsity distribution which require a higher number of FLOPs for inference with same parameter count.

Original: The ﬁnal sub-group includes methods that require larger storage and the work of a training  dense model. 
Revised: The ﬁnal sub-group includes methods that require the space and the work proportional to training a dense model. [Figure]

Original: 4. M OBILE N ET
Revised: 4.1. M OBILE N ET

Original: MobileNet is a compact architecture that performs remarkably well in resource constrained settings.
Revised: MobileNet is a compact architecture that performs remarkably well in resource constrained settings.

Original: Due to its compact nature with separable convolutions it is known to benotoriously difﬁcult to sparsify (Zhu & Gupta, 2018).
Revised: Due to its compact nature with separable convolutions it is known to be difﬁcult to sparsify (Zhu & Gupta, 2018).

Original: In this section we apply our method to MobileNet-v1 (Howard et al., 2017) and MobileNet-v2 (Sandler et al., 2018).
Revised: In this section we apply our method to MobileNet-v1 (Howard et al., 2017) and MobileNet-v2 (Sandler et al., 2018).

Original: Due to its low parameter count we keep the ﬁrst layer dense, and evaluate using ERK and Uniform sparsity distribution to sparsify the remaining layers.
Revised: Due to its low parameter count we keep the ﬁrst layer dense, and use ERK and Uniform sparsity distributions to sparsify the remaining layers.

Original: The performance of sparse MobileNets trained with RigL as well as the baselines are shown in
Revised: The performance of sparse MobileNets trained with RigL as well as the baselines are shown in

Original: Figure 3.
Revised: Figure 3.

Original: We do extended training (5x of the original number of steps) for all runs in this section.
Revised: We do extended training (5x of the original number of steps) for all runs in this section.

Original: Although MobileNets are more sensitive to sparsity compared to the ResNet-50 architecture, RigL successfully trains sparse MobileNets at high sparsities and exceeds the performance of previously reported pruning results.[Figure]
Revised: Although MobileNets are more sensitive to sparsity compared to the ResNet-50 architecture, RigL successfully trains sparse MobileNets at high sparsities and exceeds the performance of previously reported pruning results.

Original: To demonstrate the advantages of sparse models we train widersparse MobileNets while keeping the FLOPs and total number of parameters the same as the dense baseline.
Revised: To demonstrate the advantages of sparse models, next, we train wider MobileNets while keeping the FLOPs and total number of parameters the same as the dense baseline using sparsity.

Original: A sparse MobileNet v1 with width multiplier 1.98 and constant  sparsity has the same FLOP and parameter count as the dense baseline.
Revised: A sparse MobileNet-v1 with width multiplier 1.98 and constant 75% sparsity has the same FLOPs and parameter count as the dense baseline.

Original: This yields a 3.8% absolute improvement in Top-1 Accuracy.
Revised: Training this network with RigL yields an impressive 4.3% absolute improvement in Top-1 Accuracy.

Original: 4. A NALYZING THE PERFORMANCE OF R IG L 
Revised: 4. C HARACTER L EVEL L ANGUAGE M ODELLING

Revised: Most prior work has only examined sparse training on vision networks [the exception is the earliest work - Deep Rewiring (Bellec et al., 2017) which trained an LSTM (Hochreiter & Schmidhuber, 1997) on the TIMIT (Garofolo et al., 1993) dataset].

Revised: To fully understand these techniques it is important to examine different architectures on different datasets.

Revised: Kalchbrenner et al.

Revised: (2018) found sparse GRUs (Cho et al., 2014) to be very effective at modeling speech, however the dataset they used is not available.

Revised: We choose a proxy task with similar characteristics (dataset size and vocabulary size are approximately the same) - character level language modeling on the publicly available

Revised: WikiText-103 (Merity et al., 2016) dataset.

Revised: Our network consists of a shared embedding with dimensionality 128, a vocabulary size of 256, a

Revised: GRU with a state size of 512, a readout from the GRU state consisting of two linear layers with 256 units and 128 units respectively.

Revised: We train the next step prediction task with the standard cross entropy loss, the Adam optimizer, a learning rate of 7 e − 4 , an L2 regularization coefﬁcient of 5 e − 4 , a sequence length of 512, a batch size of 32 and gradient absolute value clipping of values larger (in magnitude) than 10.

Revised: Baseline training length is 200,000 iterations.

Revised: When inducing sparsity with magnitude pruning (Zhu & Gupta, 2018), we perform pruning between iterations 50,000 and

Revised: [Figure]

Revised: Training Flops

Revised: [Figure]

Revised: Performance on WikiText-

Revised: [Figure]

Revised: Sparsity Sweep on CIFAR-

Revised: [Figure]

Revised: 150,000 with a pruning frequency of 1,000.

Revised: We initialize sparse networks with a uniform sparsity distribution and use a cosine update schedule with α = 0 .

Revised: 1 and ∆ T = 100 .

Revised: Unlike the previous experiments we keep updating the mask until the end of the training; we observed this performed slightly better than stopping at iteration 150,000.

Revised: In Figure 4-left we report the validation loss of various solutions at the end of the training.

Revised: For each method we perform extended runs to see how they scale with increasing training time.

Revised: As observed before, SET performs worst than the other dynamic training methods and its performance improves only slightly with increased training time.

Revised: On the other hand the performance of RigL and SNFS improves constantly with more training steps.

Revised: Both of these methods falls short of matching the pruning performance.

Revised: 4. W IDE R ES N ET -22-2 ON CIFAR-

Revised: In this section, we evaluate the performance of RigL on CIFAR-10 image classiﬁcation benchmark.

Revised: We train Wide Residual Network’s (Zagoruyko & Komodakis, 2016) with 22 layers using a width multiplier of 2 for 250 epochs (97656 steps).

Revised: Learning rate starts at 0.1 and scaled down by a factor of 5 every 30,000 iterations.

Revised: We use an L2 regularization coefﬁcient of 5e-4, a batch size of 128 and a momentum coefﬁcient of 0.9.

Revised: We keep the hyper-parameters speciﬁc to RigL same as the ImageNet experiments, except the ﬁnal iteration for mask updates; which is adjusted to 75000.

Revised: Results with different mask update intervals can be found in Appendix H.

Revised: Performance of RigL across different sparsity levels is presented in Figure 4-right.

Revised: Corresponding ﬁnal training losses of the trained networks can be found in Appendix H.

Revised: The dense baseline obtains 94.1% test accuracy.

Revised: Networks with half of the connections removed (50% sparsity) achieves roughly the same accuracy as the dense baseline.

Revised: Surprisingly, some of the networks at this sparsity level generalize better than the dense baseline demonstrating the regularization aspect of using sparsity.

Revised: With increased sparsity, we start to see a performance gap between the Static and Pruning solutions.

Revised: Training static RigL networks longer seems to have limited effect on the ﬁnal performance.

Revised: On the other hand, RigL , matches the performance of pruning using only a fraction of resources needed for training a dense network.

Original: In this section we study the effect of sparsity distributions , update schedules , and dynamic connections on the performance of our method.
Revised: In this section we study the effect of sparsity distributions , update schedules , and dynamic connections on the performance of our method.

Original: The results for SET and SNFS are similar and are discussed in Appendices A and C.
Revised: The results for SET and SNFS are similar and are discussed in Appendices B and E.

Original: Effect of Mask Initialization: Figure 4-left shows how the sparsity distribution affects the ﬁnal test accuracy of sparse ResNet-50s trained with RigL .
Revised: Effect of Mask Initialization: Figure 5-left shows how the sparsity distribution affects the ﬁnal test accuracy of sparse ResNet-50s trained with RigL .

Original: Erd ˝ os-R ´ enyi-Kernel (ERK) performs signiﬁcantly better than the other two initializations overall.
Revised: Erd˝os-R´enyi-Kernel (ERK) performs consistently better than the other two distributions. ERK automatically allocates more parameters to the [Figure]

Original: Recall that uniform distribution keeps the ﬁrst layer dense and has a ﬁxed sparsity for  the last dense layer.
Revised: Sparsity [Figure] layers with few parameters by decreasing their sparsities † . This reallocation seems to be crucial for preserving the capacity of the network at high sparsity levels where ERK outperforms other distributions by a greater margin.

Original: We scale the uniform sparsities so that the overall parameter count is the same across all distributions considered.

Original: Though performing better,  ERK distribution requires around 2x FLOPsto run compared to uniform.
Revised: Though it performs better, the ERK distribution requires approximately twice as many FLOPs compared to a uniform distribution.

Original: This highlights an interesting trade-off between accuracy and computational efﬁciency.
Revised: This highlights an interesting trade-off between accuracy and computational efﬁciency even though both models have the same number of parameters.

Original: Effect of Update Schedule and Frequency:
Revised: Effect of Update Schedule and Frequency:

Original: In Figure 4-right, we evaluate the performance of our method on update intervals ∆ I ∈ [50 , 100 , 500 , 1000] and initial drop fractions α ∈ [0 . 1 , 0 . 3 , 0 . 5] .Even when we update the sparse connectivity every 1000 iterations, Rigl performs above 72.5%. [Figure] [Figure]
Revised: In Figure 5-right, we evaluate the performance of our method on update intervals ∆ T ∈ [50 , 100 , 500 , 1000] and initial drop fractions α ∈ [0 . 1 , 0 . 3 , 0 . 5] .

Original: T r a i n i n g

Original: L o ss

Original: Pruning

Original: Static

Original: [Figure] Iteration 1e [Figure]
Revised: The best accuracies are obtained when the mask is updated every 100 iterations with an initial drop fraction of 0.3 or 0.5.

Original: T

Original: [Figure]

Original: i n i

Original: [Figure]

Original: g

Original: [Figure]

Original: ss RigL Static Accuracy 73.
Revised: Notably, even with frequent update intervals (e.g. every 1000 iterations), RigL performs above 73.5%.

Original: Effect of Dynamic connections: Frankle et al.
Revised: Effect of Dynamic connections: Frankle et al.

Original: and Mostafa & Wang (2019) observed that static sparse training converges to a solution with a higher loss than dynamic sparse training.
Revised: and Mostafa & Wang (2019) observed that static sparse training converges to a solution with a higher loss than dynamic sparse training.

Original: In
Revised: In

Original: Figure 5-left we examine the loss landscape lying between a solution found via static sparse training and a solution found via dynamic sparse training.
Revised: Figure 6-left we examine the loss landscape lying between a solution found via static sparse training and a solution found via pruning to understand whether former lies in a basin isolated from the latter.

Original: Performing a linear interpolation between the two reveals a high-loss barrier which may be insurmountable with a ﬁxed topology.
Revised: Performing a linear interpolation between the two reveals the expected result – high-loss barrier – demonstrating that the loss landscape is not trivially connected.

Revised: However, this is only one of inﬁnitely many paths between the two points optimization can be used to ﬁnd parametric curves that connects solutions (Garipov et al., 2018; Draxler et al., 2018) subject to constraints.

Original: Following Garipov et al.
Revised: For example Garipov et al.

Revised: (2018) showed different dense solutions lie in the same basin by ﬁnding 2nd order B´ezier curves with low energy between the two solutions.

Revised: Following their method, we attempt

Original: , we attempt to ﬁnd quadratic and cubic B ´ ezier curves between the two  solutions.
Revised: [Figure] Path Between Two Solutions [Figure] to ﬁnd quadratic and cubic B´ezier curves between the two sparse solutions.

Original: Surprisingly, even with a cubic curve, we fail to ﬁnd a path without a high-loss barrier.
Revised: Surprisingly, even with a cubic curve, we fail to ﬁnd a path without a high-loss barrier.

Original: These results suggest that static sparse training can get stuck at local minima that are isolated from improved solutions.
Revised: These results suggest that static sparse training can get stuck at local minima that are isolated from improved solutions.

Original: On the other hand, when we optimize the quadratic B ´ ezier curve across the full dense space we ﬁnd a near-monotonic path to the improved solution, suggesting that allowing new connections to grow lends dynamic sparse training greater ﬂexibility in navigating the loss landscape.
Revised: On the other hand, when we optimize the quadratic B´ezier curve across the full dense space we ﬁnd a near-monotonic path to the improved solution, suggesting that allowing new connections to grow lends dynamic sparse training greater ﬂexibility in navigating the loss landscape.

Original: In Figure 5-right we train RigL starting from the sub-optimal solution found by static sparse training, demonstrating that it is able to escape the local minimum, whereas re-training with static sparse training cannot. 
Revised: In Figure 6-right we train RigL starting from the sub-optimal solution found by static sparse training, demonstrating that it is able to escape the local minimum, whereas re-training with static sparse training cannot. RigL ﬁrst removes connections with the smallest magnitudes since removing these connections have been shown to have a minimal effect on the loss (Han et al., 2015; Evci, 2018).

Revised: Next, it activates connections with the high gradients, since these connections are expected to decrease the loss fastest.

Revised: We hypothesize in Appendix A that RigL escapes bad critical points by replacing saddle directions with high gradient dimensions.

Original:  D ISCUSSION & C ONCLUSION
Revised:  D ISCUSSION & C ONCLUSION

Original: In this work we introduced ‘Rigged Lottery’ or RigL , an algorithm for training sparse neural networks efﬁciently.
Revised: In this work we introduced ‘Rigged Lottery’ or RigL , an algorithm for training sparse neural networks efﬁciently.

Original: For a given computational budget RigL achieves higher accuracies than existing dense-to-sparse and sparse-to-sparse training algorithms.
Revised: For a given computational budget RigL achieves higher accuracies than existing dense-to-sparse and sparse-to-sparse training algorithms.

Original: RigL is useful in three different scenarios: (1) To improve the accuracy of sparse models intended for deployment; (2) To improve the accuracy of large sparse models which can only be trained for a limited number of iterations; and (3) Combined with sparse primitives to enable training of extremely large sparse models which otherwise would not be possible.
Revised: RigL is useful in three different scenarios: (1) To improve the accuracy of sparse models intended for deployment; (2) To improve the accuracy of large sparse models which can only be trained for a limited number of iterations; and (3) Combined with sparse primitives to enable training of extremely large sparse models which otherwise would not be possible.

Original: The third scenario is unexplored due to the lack of hardware and software support for sparsity.
Revised: The third scenario is unexplored due to the lack of hardware and software support for sparsity.

Original: Nonetheless, work continues to improve the performance of sparse networks on current hardware (Hong et al., 2019; Merrill & Garland, 2016), and new types of hardware accelerators will have better support for parameter sparsity (Wang et al., 2018; Mike Ashby, 2019; Liu et al., 2018;
Revised: Nonetheless, work continues to improve the performance of sparse networks on current hardware (Hong et al., 2019; Merrill & Garland, 2016), and new types of hardware accelerators will have better support for parameter sparsity (Wang et al., 2018; Mike Ashby, 2019; Liu et al., 2018;

Original: Han et al., 2016a).
Revised: Han et al., 2016a; Chen et al., 2019).

Original: RigL provides the tools to take advantage of, and motivation for, such advances.
Revised: RigL provides the tools to take advantage of, and motivation for, such advances.

Original: A E FFECT OFS PARSITY D ISTRIBUTION ON O THER M ETHODS
Revised: A E FFECT OF M ASK U PDATES ON THE E NERGY L ANDSCAPE

Revised: To update the connectivity of our sparse network we ﬁrst need to drop a fraction d of the existing connections for each layer independently to create a budget for growing new connections.

Revised: Following the recipe of magnitude based pruning(Han et al., 2015) we order parameters at layer i by magnitude | θ i | and drop the N ∗ (1 − S ) ∗ d parameters with lowest magnitude.

Revised: The effectiveness of this simple criteria can be explained through the ﬁrst order Taylor approximation of the loss L around the current set of parameters θ .

Revised: [Equation]

Revised: The main goal of dropping connections is to remove parameters with minimal impact on the neural network and therefore on its performance.

Revised: Since removing the connection θ i corresponds to setting it to zero, it incurs a change of ∆ θ = − θ i in that direction and a change of ∆ L i =

Revised: −∇ θ i L ( θ ) θ i

Revised: + R ( θ 2 i ) in the loss, where the ﬁrst term is usually deﬁned as the saliency of a connection.

Revised: Though using saliency to remove connections has been used as a criteria for removing connections (Molchanov et al., 2016), it has been shown to produce inferior results compared to magnitude based removal, especially when used to remove multiple connections at once (Evci, 2018).

Revised: In contrast, picking the lowest magnitude connections ensures a small remainder term in addition to a low saliency, limiting the damage we make when we drop connections.

Revised: Additionally, we note that connections with small magnitude can only remain small if the gradient is also small, meaning that the saliency is likely small when the parameter itself is small.

Revised: Therefore we argue that the connections removed by RigL are likely to be saddle directions of the energy landscape.

Revised: After the removal of insigniﬁcant connections, we enable new connections that have the highest expected gradients.

Revised: Since we initialize these new connections to zero, they are guaranteed to have high gradients in the proceeding iteration and therefore to reduce the loss quickly.

Revised: By deﬁnition a direction with high magnitude gradient is not a saddle direction.

Revised: Combining this observation with the previous ( RigL is likely to remove saddle directions) and the results in Section 4.4 we suggest that RigL improves the energy landscape of the optimization by replacing saddle directions with the ones with high gradient.

Revised: This helps the optimization procedure to escape bad critical points and ﬁnd solutions with higher quality.

Original: In Figure 6-left we show the effect of sparsity distribution choice on 4 different sparse training methods.
Revised: In Figure 7-left we show the effect of sparsity distribution choice on 4 different sparse training methods.

Original: The ordering among the three distribution is the same across different methods.
Revised: ERK distribution performs better than other distributions for each training method.

Original: B E FFECT OF M OMENTUM C OEFFICIENT FOR SNFS
Revised: C E FFECT OF M OMENTUM C OEFFICIENT FOR SNFS

Original: In Figure 6-right we show the effect of the momentum coefﬁcient on the performance of SNFS.
Revised: In Figure 7 right we show the effect of the momentum coefﬁcient on the performance of SNFS.

Original: Our results suggest that the accumulated values are not important since setting the coefﬁcient to 0 brings 
Revised: Our results shows that using a coefﬁcient of 0.99 brings the best performance.

Original: [Figure]
Revised: On the other hand using the most recent gradient only (coefﬁcient of 0) performs as good as using a coefﬁcient of 0.9. This result might be due to the large batch size we are using (4096), but it still motivates using RigL and instantaneous gradient information only when needed, instead of accumulating them. D E XISTENCE OF L OTTERY T ICKETS We perform the following experiment to see whether Lottery Tickets exist in our setting.

Original: Sparsity
Revised: We take the sparse network found by RigL and restart training using original initialization, both with RigL and with ﬁxed topology as in the original Lottery Ticket Hypothesis. Results in table 3 demonstrate that training with a ﬁxed topology is signiﬁcantly worse than training with RigL and that RigL does not beneﬁt from starting again with the ﬁnal topology and the original initialization - training for twice as long instead of rewiring is more effective. In short, there are no special tickets, with RigL all tickets seems to win.

Original: [Figure]
Revised: [Figure]

Original:  c y
Revised: r a c y

Original: Effect of Sparsity Distributions
Revised: Effect of Sparsity Distributions

Original: [Figure]
Revised: [Figure]

Original: Effect of Momentum Coefficientaccumulated values are not important.
Revised: Effect of Momentum Coefficient

Original: the best performance. This surprising result might be due to the large batch size we are using (4096), but it still motivates using RigL and instantaneous gradient information only when needed, instead of accumulating them.
Revised: [Table]

Original: C E FFECT OF U PDATE S CHEDULES ON O THER D YNAMIC S PARSE M ETHODS
Revised: E E FFECT OF U PDATE S CHEDULES ON O THER D YNAMIC S PARSE M ETHODS

Original: In Figure 7 we repeat the hyper-parameter sweep done for RigL in Figure 4-right, but for SET and
Revised: In Figure 8 we repeat the hyper-parameter sweep done for RigL in Figure 5-right, using SET and

Original: SNFS.
Revised: SNFS.

Original: A cosine schedule with ∆ I = 50 and α = 0 .
Revised: Cosine schedule with ∆ T = 50 and α = 0 .

Original: 1 seems to work best across all methods.
Revised: 1 seems to work best across all methods.

Original: For this reason, we used these values in the experiments presented in the main text.

Original: An interesting observation is that higher drop fractions ( α ) seem to work better with longer intervals ∆ I .
Revised: An interesting observation is that higher drop fractions ( α ) seem to work better with longer intervals ∆ T .

Original: For example, SET with an update interval of 1000 seems to work quite well with˚a starting drop fraction α = 0 .
Revised: For example, SET with ∆ T = 1000 seems to work best with α = 0 .

Original: D T RYING D IFFERENT U PDATE S CHEDULES
Revised: F A LTERNATIVE U PDATE S CHEDULES

Original: In Figure 8, we share the results ofusing two alternative annealing functions for the update fractions:
Revised: In Figure 9, we share the performance of two alternative annealing functions:

Original: Constant: f decay ( t ) =
Revised: Constant: f decay ( t ) =

Original: α . 2.
Revised: α . 2.

Original: Inverse Power: The fraction of weights updated decreases similarly to the schedule used in Zhu & Gupta(2018): f decay ( t ) =
Revised: Inverse Power: The fraction of weights updated decreases similarly to the schedule used in Zhu & Gupta ) for iterative

Revised: pruning: f decay ( t ) =

Original: α (1 − t I end ) k .
Revised: α (1 − t T end ) k .

Original: In our experiments we tried k =which is the linear decay and their default k = 3 .
Revised: In our experiments we tried k = 1 which is the linear decay and their default k = 3 .

Original: Constant seems to perform well with low initial drop fractions like α = 0 . 1 , but it starts to perform worse with increasing α .
Revised: Constant seems to perform well with low initial drop fractions like α = 0 . 1 , but it starts to perform worse with increasing α .

Original: Inverse Power for k=3 and k=1 ( Linear ) seems to perform similarly for low α values.
Revised: Inverse Power for k=3 and k=1 ( Linear ) seems to perform similarly for low α values.

Original: However the performance drops noticeably for k=3 when we increase the update interval.
Revised: However the performance drops noticeably for k=3 when we increase the update interval.

Original: As reported by Dettmers & Zettlemoyer (2019) linear (k=1) seems to provide similar results as the cosine schedule.
Revised: As reported by Dettmers & Zettlemoyer (2019) linear (k=1) seems to provide similar results as the cosine schedule.

Original: E C ALCULATING FLOP S OF MODELS AND METHODS
Revised: G C ALCULATING FLOP S OF MODELS AND METHODS

Original: In order to calculate FLOPs needed for a single forward pass of a sparse model, we count the total number of multiplications and additions layer by layer for a given layer sparsity s l .
Revised: In order to calculate FLOPs needed for a single forward pass of a sparse model, we count the total number of multiplications and additions layer by layer for a given layer sparsity s l .

Original: The total FLOPs is then obtained by summing up all of these multiply and adds.
Revised: The total FLOPs is then obtained by summing up all of these multiply and adds.

Original: Different sparsity distributions require different number of FLOPs to compute a single prediction.
Revised: Different sparsity distributions require different number of FLOPs to compute a single prediction.

Original: For example Erd˝os-Renyi-Kernel distributions usually cause earlier layers to be less sparse than the later layers (see Appendix G).
Revised: For example Erd˝os-Renyi-Kernel distributions usually cause earlier layers to be less sparse than the later layers (see Appendix I). The inputs of earlier layers have greater spatial dimensions, so [Figure]

Original: The inputs of earlier layers have greater spatial dimensions, so a convolutional kernel that works on such inputs will require more FLOPs to compute the output features compared to later layers.
Revised: Cosine- [Figure] a convolutional kernel that works on such inputs will require more FLOPs to compute the output features compared to later layers.

Original: Thus, having earlier layers which are less sparse results in a higher total number of FLOPs required by a model.[Figure]
Revised: Thus, having earlier layers which are less sparse results in a higher total number of FLOPs required by a model.

Original: Training a neural network consists of 2 main steps:
Revised: Training a neural network consists of 2 main steps:

Original: forward pass: Calculating the loss of the current set of parameters on a given batch of data.
Revised: forward pass: Calculating the loss of the current set of parameters on a given batch of data.

Original: During this process layer activations are calculated in sequence using the previous activations and the parameters of the layer.
Revised: During this process layer activations are calculated in sequence using the previous activations and the parameters of the layer.

Original: Activation of layers are stored in memory for the backward pass.
Revised: Activation of layers are stored in memory for the backward pass.

Original: backward pass:
Revised: backward pass:

Original: Using the loss value as the initial error signal, we back-propagate the error signal while calculating the gradient of parameters.
Revised: Using the loss value as the initial error signal, we back-propagate the error signal while calculating the gradient of parameters.

Original: During the backward pass each layer calculates 2 quantities: the gradient of the activations of the previous layer and the gradient of its parameters.
Revised: During the backward pass each layer calculates 2 quantities: the gradient of the activations of the previous layer and the gradient of its parameters.

Original: Therefore in our calculations we count backward passes as two times the computational expense of the forward pass.
Revised: Therefore in our calculations we count backward passes as two times the computational expense of the forward pass.

Original: We omit the FLOPs needed for batch normalization and cross entropy.
Revised: We omit the FLOPs needed for batch normalization and cross entropy.

Original: Dynamic sparse training methods require some extra FLOPs to update the connectivity of the neural network.
Revised: Dynamic sparse training methods require some extra FLOPs to update the connectivity of the neural network.

Original: We omit FLOPs needed for dropping the lowest magnitude connections in our calculations.
Revised: We omit FLOPs needed for dropping the lowest magnitude connections in our calculations.

Original: For a given dense architecture with FLOPs f D and a sparse version with FLOPs f S , the total FLOPs required to calculate the gradient on a single sample is computed as follows:
Revised: For a given dense architecture with FLOPs f D and a sparse version with FLOPs f S , the total FLOPs required to calculate the gradient on a single sample is computed as follows:

Original: • Static Sparse and Dense.
Revised: • Static Sparse and Dense.

Original: Scales with 3 ∗ f S and 3 ∗ f D FLOPs, respectively.
Revised: Scales with 3 ∗ f S and 3 ∗ f D FLOPs, respectively.

Original: • Snip.
Revised: • Snip.

Original: We omit the initial dense gradient calculation since it is negligible, which means Snip scales in the same way as Static methods: 3 ∗ f S FLOPs.
Revised: We omit the initial dense gradient calculation since it is negligible, which means Snip scales in the same way as Static methods: 3 ∗ f S FLOPs.

Original: • SET.
Revised: • SET.

Original: We omit the extra FLOPs needed for growing random connections, since this operation can be done on chip efﬁciently.
Revised: We omit the extra FLOPs needed for growing random connections, since this operation can be done on chip efﬁciently.

Original: Therefore, the total FLOPs for SET scales with 3 ∗ f S .
Revised: Therefore, the total FLOPs for SET scales with 3 ∗ f S .

Original: • SNFS.
Revised: • SNFS.

Original: Forward pass and back-propagating the error signal needs 2 ∗ f S FLOPs.
Revised: Forward pass and back-propagating the error signal needs 2 ∗ f S FLOPs.

Original: However, the dense gradient needs to be calculated at every iteration.
Revised: However, the dense gradient needs to be calculated at every iteration.

Original: Thus, the total number of FLOPs scales with 2 ∗ f S + f D .
Revised: Thus, the total number of FLOPs scales with 2 ∗ f S + f D .

Original: • RigL. Iterations with no connection updates need 3 ∗ f S FLOPs.
Revised: • RigL. Iterations with no connection updates need 3 ∗ f S FLOPs.

Original: However, at every ∆ I iteration we need to calculate the dense gradients.
Revised: However, at every ∆ T iteration we need to calculate the dense gradients.

Original: This results in the average FLOPs for RigL given by (3 ∗ f S ∗ ∆ I +2 ∗ f S + f D ) (∆ I +1) .
Revised: This results in the average FLOPs for RigL given by (3 ∗ f S ∗ ∆ T +2 ∗ f S + f D ) (∆ T +1) .

Original: F E FFECT OF M ASK U PDATES ON THE E NERGY L ANDSCAPE 
Revised: H A DDITIONAL P LOTS AND E XPERIMENTS FOR CIFAR-

Original: To update the connectivity of our sparse network
Revised: In Figure 10-left

Original: [Figure]
Revised: [Figure]

Original: we ﬁrst need to drop a fraction d of the existing connections for each layer independently to create a budget for growing new connections.
Revised: we plot the ﬁnal training loss of experiments presented in Section 4.3 to investigate the generalization properties of the algorithms considered. Poor performance of Static reﬂects itself in training loss clearly across all sparsity levels.

Original: Following the recipe of magnitude based pruning(Han et al., 2015)
Revised: RigL achieves similar ﬁnal loss as the pruning, despite having around half percent less accuracy. Training longer with RigL decreases the ﬁnal loss further and the test accuracies start matching pruning (see Figure 4-right) performance. These results show that RigL improves the optimization as promised, however generalizes slightly worse than pruning

Original: [Figure] 
Revised: [Figure] [Figure]

Original: we order parameters at layer i by magnitude | θ i | and drop the N i ∗ (1 − s ) ∗ d parameters with lowest magnitude. The effectiveness of this simple criteria can be explained through the ﬁrst order Taylor approximation of the loss L around the current set of parameters θ .
Revised: Sparsity Sweep

Original: [Equation]
Revised: [Figure]

Original: The main goal ofdropping connections is to remove parameters with minimal impact on the function and therefore on its performance. Since removing a connection corresponds to setting it to
Revised: Performance of on CIFAR

Original: [Figure] 
Revised: [Figure] [Figure]

Original: zero, removing the parameter θ i incurs a change of ∆ θ = − θ i in the parameters and a change of ∆ L i =
Revised: In Figure 10-right, we sweep mask update interval ∆ T and plot the ﬁnal test accuracies. We ﬁx initial drop fraction α to 0 .

Original: −∇ θ i L ( θ ) θ i + R ( θ 2 i ) in the loss, where the ﬁrst term is usually deﬁned as the saliency of a connection. Though using saliency to remove connections has been used as a criteria for removing connections (Molchanov et al., 2016), it has been shown to produce inferior results compared to magnitude based removal, especially when used to remove multiple connections at once (Evci, 2018). In contrast, picking the lowest magnitude connections ensures a small remainder term in addition to a low saliency, limiting the damage we make to the network while dropping connections.
Revised: 3 and evaluate two different sparsity distributions: Uniform and ERK .

Original: In other words, dropped connections are likely to be connections which don’t affect the output of the neural network. After removal of these connections, we add new connections with the highest expected gradients. Note that although we initialize the added connections to zero, they are guaranteed to have a high gradient in the ﬁrst iteration after the mask update. A direction of non-zero gradient means that the energy landscape on that dimension is not ﬂat.
Revised: Both curves follow a similar pattern as in Imagenet-2012 sweeps (see Figure 8) and best results are obtained when ∆ T = 100 .

Original: G S PARSITY OF I NDIVIDUAL L AYERS FOR S PARSE R ES N ET - 
Revised: I S PARSITY OF I NDIVIDUAL L AYERS FOR S PARSE R ES N ET -

Original: Sparsity of ResNet-50 layers deﬁned by the Erd ˝ os-R ´ enyi-Kernel sparsity distribution plotted in Figure 9.
Revised: Sparsity of ResNet-50 layers given by the Erd ˝ os-R ´ enyi-Kernel sparsity distribution plotted in Figure 11. J B UGS D ISCOVERED D URING E XPERIMENTS

Original: [Table]
Revised: Our initial implementations contained some subtle bugs, which while not affecting the general conclusion that RigL is more effective than other techniques, did result in lower accuracy for all sparse training techniques. We detail these issues here with the hope that others may learn from our mistakes. Random operations on multiple replicas. We use data parallelism to split a mini-batch among multiple replicas.

Revised: Each replica independently calculates the gradients using a different sub-mini-batch of data.

Revised: The gradients are aggregated using an ALL - REDUCE operation before the optimizer update.

Revised: Our implementation of SET, SNFS and RigL depended on each replica independently choosing to drop and grow the same connections.

Revised: However, due to the nature of random operations in Tensorﬂow, this did not happen.

Revised: Instead, different replicas diverged after the ﬁrst drop/grow step.

Revised: This was most pronounced in SET where each replica chose at random and much less so for SNFS and RigL where randomness is only needed to break ties.

Revised: If left unchecked this might be expected to be catastrophic, but due to the behavior of Estimators and/or TF-replicator, the values on the ﬁrst replica are broadcast to the others periodically (every approximately 1000 steps in our case).

Revised: We ﬁxed this bug by using stateless random operations.

Revised: As a result the performance of SET improved slightly (0.1-0.3 % higher on Table 2).

Revised: Synchronization between replicas.

Revised: RigL and SNFS depend on calculating dense gradients with respect to the masked parameters.

Revised: However, as explained above, in the multiple replica setting these gradients need to be aggregated.

Revised: Normally this aggregation is automatically done by the optimizer, but in our case, this does not happen (only the gradients with respect to the unmasked parameters are aggregated automatically).

Revised: This bug affected SNFS and

Revised: RigL , but not SET since SET does not rely on the gradients to grow connections.

Revised: Again, the synchronization of the parameters from the ﬁrst replica every approximately 1000 steps masked this bug.

Revised: We ﬁxed this bug by explicitly calling ALL - REDUCE on the gradients with respect to the masked parameters.

Revised: With this ﬁx, the performance of RigL and SNFS improved signiﬁcantly, particularly for default training lengths (around 0.5-1% improvement).

Original:  [Figure]
Revised: [Table] [Figure]

Revised: SNIP Experiments.

Revised: Our ﬁrst implementation of SNIP used the gradient magnitudes to decide which connections to keep causing its performance to be worse than static.

Revised: Upon our discussions with the authors of SNIP, we realized that the correct metric is the saliency (gradient times parameter magnitude).

Revised: With this correction SNIP performance improved dramatically to better than random (Static) even at Resnet-50/ImageNet scale.

Revised: It is surprising that picking connections with the highest gradient magnitudes can be so detrimental to training (it resulted in much worse than random performance).