Reproducibility Report
Rigging the Lottery: Making All Tickets Winners
Reproducibility Summary Scope of Reproducibility
For a ﬁxed parameter count and compute budget, the proposed algorithm ( RigL ) claims to directly train sparse networks that match or exceed the performance of existing dense-to-sparse training techniques (such as pruning).
RigL does so while requiring constant Floating Point Operations (FLOPs) throughout training.
The technique obtains state-of-the-art performance on a variety of tasks, including image classiﬁcation and character-level language-modelling.
Methodology
We implement RigL from scratch in Pytorch using boolean masks to simulate unstructured sparsity.
We rely on the description provided in the original paper, and referred to the authors’ code for only speciﬁc implementation detail such as handling overﬂow in ERK initialization.
We evaluate sparse training using RigL for WideResNet-22-2 on CIFARand ResNet-50 on CIFAR-100, requiring 2 hours and 6 hours respectively per training run on a GTX 1080 GPU.
Results
We reproduce RigL ’s performance on CIFAR-10 within 0.1% of the reported value.
On both CIFAR-10/100, the central claim holds—given a ﬁxed training budget, RigL surpasses existing dynamic-sparse training methods over a range of target sparsities.
By training longer, the performance can match or exceed iterative pruning, while consuming constant
FLOPs throughout training.
We also show that there is little beneﬁt in tuning RigL ’s hyper-parameters for every sparsity, initialization pair—the reference choice of hyperparameters is often close to optimal performance.
Going beyond the original paper, we ﬁnd that the optimal initialization scheme depends on the training constraint.
While the Erdos-Renyi-Kernel distribution outperforms Random distribution for a ﬁxed parameter count, for a ﬁxed FLOP count, the latter performs better.
Finally, redistributing layer-wise sparsity while training can bridge the performance gap between the two initialization schemes, but increases computational cost.
What was easy
The authors provide code for most of the experiments presented in the paper.
The code was easy to run and allowed us to verify the correctness of our re-implementation.
The paper also provided a thorough and clear description of the proposed algorithm without any obvious errors or confusing exposition.
What was difﬁcult
Tuning hyperparameters involved multiple random seeds and took longer than anticipated.
Verifying the correctness of a few baselines was tricky and required ensuring that the optimizer’s gradient (or momentum) buffers were sparse (or dense) as speciﬁed by the algorithm.
Compute limits restricted us from evaluating on larger datasets such as Imagenet.
Communication with original authors
We had responsive communication with the original authors, which helped clarify a few implementation and evaluation details, particularly regarding the FLOP counting procedure.
Introduction
Sparse neural networks are a promising alternative to conventional dense networks—having comparatively greater parameter efﬁciency and lesser ﬂoating-point operations (FLOPs) (Han et al.
[2016], Ashby et al.
[2017], Srinivas et al.
Unfortunately, present techniques to produce sparse networks of commensurate accuracy involve multiple cycles of training dense networks and subsequent pruning.
Consequently, such techniques offer no advantage over training dense networks, either computationally or memory-wise.
In the paper Evci et al.
[2020], the authors propose RigL , an algorithm for training sparse networks from scratch.
The proposed method outperforms both prior art in training sparse networks, as well as existing dense-to-sparse training algorithms.
By utilising dense gradients only during connectivity updates and avoiding any global sparsity redistribution,
RigL can maintain a ﬁxed computational cost and parameter count throughout training.
As a part of the ML Reproducibility Challenge, we replicate RigL from scratch and investigate if dynamic-sparse training confers signiﬁcant practical beneﬁts compared to existing sparsifying techniques.
Scope of reproducibility
In order to verify the central claims presented in the paper we focus on the following target questions:
• Does RigL outperform existing sparse-to-sparse training techniques—such as SET (Mocanu et al.
[2018]) and SNFS (Dettmers and Zettlemoyer [2020])—and match the accuracy of dense-to-sparse training methods such as iterative pruning (Zhu and Gupta [2018])?
• RigL requires two additional hyperparameters to tune.
We investigate the sensitivity of ﬁnal performance to these hyperparameters across a variety of target sparsities (Section 5.3).
• How does the choice of sparsity initialization affect the ﬁnal performance for a ﬁxed parameter count and a ﬁxed training budget (Section 6.1)?
• Does redistributing layer-wise sparsity during connection updates (Dettmers and Zettlemoyer [2020]) improve RigL ’s performance?
Can the ﬁnal layer-wise distribution serve as a good sparsity initialization scheme (Section 6.2)?
Methodology
The authors provide publicly accessible code 1 written in Tensorﬂow (Abadi et al.
To gain a better understanding of various implementation aspects, we opt to replicate RigL in Pytorch (Paszke et al.
Our implementation extends the open-source code 2 of Dettmers and Zettlemoyer [2020] which uses a boolean mask to simulate unstructured sparsity.
Our source code is publicly accessible on Github 3 with training plots available on WandB 4 (Biewald [2020]).
Mask Initialization For a network with L layers and total parameters N , we associate each layer with a random boolean mask of sparsity s l ,
l ∈
[ L ] .
The overall sparsity of the network is given by S = (cid:80) l s l N l N , where N l is the parameter count of layer l .
Sparsities s l are determined by the one of the following mask initialization strategies:
• Uniform: Each layer has the same sparsity, i.e., s l = S ∀ l .
Similar to the original authors, we keep the ﬁrst layer dense in this initialization.
• Erdos-Renyi (ER): Following Mocanu et al.
[2018], we set s l ∝ (cid:16) 1 − C in + C out C in × C out (cid:17) , where C in , C out are the in and out channels for a convolutional layer and input and output dimensions for a fully-connected layer.
• Erdos-Renyi-Kernel (ERK): Modiﬁes the sparsity rule of convolutional layers in ER initialization to include kernel height and width, i.e., s l ∝ (cid:16) 1 − C in + C out + w + h C in × C out × w × h (cid:17) , for a convolutional layer with C in × C out × w × h parameters.
We do not sparsify either bias or normalization layers, since these have a negligible effect on total parameter count. [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
−
[Table]
Mask Updates Every ∆ T training steps, certain connections are discarded, and an equal number are grown.
Unlike
SNFS (Dettmers and Zettlemoyer [2020]), there is no redistribution of layer-wise sparsity, resulting in constant FLOPs throughout training.
Pruning Strategy Similar to SET and SNFS, RigL prunes f fraction of smallest magnitude weights in each layer.
As detailed below, the fraction f is decayed across mask update steps, by cosine annealing:
[Equation]
[Equation]
where, α is the initial pruning rate and T end is the training step after which mask updates are ceased.
Growth Strategy RigL ’s novelty lies in how connections are grown: during every mask update, k connections having the largest absolute gradients among current inactive weights (previously zero + pruned) are activated.
Here, k is chosen to be the number of connections dropped in the prune step.
This requires access to dense gradients at each mask update step.
Since gradients are not accumulated (unlike SNFS), RigL does not require access to dense gradients at every step.
Following the paper, we initialize newly activated weights to zero.
Experimental Settings 4.1 Model descriptions
For experiments on CIFAR-10 (Alex Krizhevsky [2009]), we use a Wide Residual Network (Zagoruyko and Komodakis [2016]) with depth 22 and width multiplier 2, abbreviated as WRN-22-2.
For experiments on CIFAR- (Alex Krizhevsky [2009]), we use a modiﬁed variant of ResNet-50 (He et al.
[2016]), with the initial 7 × 7 convolution replaced by two 3 × 3 convolutions (architecture details provided in the supplementary material).
4.2 Datasets and Training descriptions
We conduct our experiments on the CIFAR-10 and CIFAR-100 image classiﬁcation datasets.
For CIFAR-10, we use a train/val/test split of 45k/5k/10k samples.
In comparison, the authors use no dedicated validation set, with 50k samples and 10k samples comprising the train set and test set, respectively.
This causes a slight performance discrepancy between our reproduction and the metrics reported by the authors (dense baseline has a test accuracy of 93.4% vs 94.1% reported).
However, our replication matches the paper’s performance when 50k samples are used for the train set (Table 4).
We use a validation split of 10k samples for CIFAR-100 as well.
On both datasets, we train models for 250 epochs each, optimized by SGD with momentum.
Our training pipeline uses standard data augmentation, which includes random ﬂips and crops.
When training on CIFAR-100, we additionally include a learning rate warmup for 2 epochs and label smoothening of 0.1 (Goyal et al.
We also initialize the last batch normalization layer (Ioffe and Szegedy [2015]) in each BottleNeck block to 0, following He et al.
4.3 Hyperparameters
RigL includes two additional hyperparameters ( α, ∆ T ) in comparison to regular dense network training.
In Sections 5.1 and 5.2, we set α = 0 .
3 , ∆ T = 100 , based on the original paper.
Optimizer speciﬁc hyperparameters—learning rate, learning rate schedule, and momentum—are also set according to the original paper.
In Section 5.3, we tune these
[Table]
hyperparameters with Optuna (Akiba et al.
We also examine whether indivdually tuning the learning rate for each sparsity value offers any signiﬁcant beneﬁt.
4.4 Baseline implementations
We compare RigL against various baselines in our experiments: SET (Mocanu et al.
[2018]), SNFS (Dettmers and
Zettlemoyer [2020]), and Magnitude-based Iterative-pruning (Zhu and Gupta [2018]).
We also compare against two weaker baselines, viz., Static Sparse training and Small-Dense networks.
The latter has the same structure as the dense model but uses fewer channels in convolutional layers to lower parameter count.
We implement iterative pruning with the pruning interval kept same as the masking interval for a fair comparison
[Table]
4.5 Computational requirements
We run our experiments on a SLURM cluster node—equipped with 4 NVIDIA GTX1080 GPUs and a 32 core Intel
CPU.
Each experiment on CIFAR-10 and CIFAR-100 consumes about 1.6 GB and 7 GB of VRAM respectively and is run for 3 random seeds to capture performance variance.
We require about 6 and 8 days of total compute time to produce all results, including hyper-parameter sweeps and extended experiments, on CIFAR-10 and CIFAR-100 respectively.
Results
Given a ﬁxed training FLOP budget, RigL surpasses existing dynamic sparse training methods over a range of target sparsities, on both CIFAR-10 and 100 (Sections 5.1, 5.2).
By training longer, RigL matches or marginally outperforms iterative pruning.
However, unlike pruning, its FLOP consumption is constant throughout.
This a prime reason for using sparse networks, and makes training larger networks feasible.
Finally, as evaluated on CIFAR-10, the original authors’ choice of hyper-parameters are close to optimal for multiple target sparsities and initialization schemes (Section 5.3).
5.1 WideResNet-22 on CIFAR-
Results on the CIFAR-10 dataset are provided in Table 2.
Tabulated metrics are averaged across 3 random seeds and reported with their standard deviation.
All sparse networks use random initialization, unless indicated otherwise.
[Figure]
While SET improves over the performance of static sparse networks and small-dense networks, methods utilizing gradient information (SNFS, RigL ) obtain better test accuracies.
SNFS can outperform RigL , but requires a much larger training budget, since it (a) requires dense gradients at each training step, (b) redistributes layer-wise sparsity during mask updates.
For all sparse methods, excluding SNFS, using ERK initialization improves performance, but with increased FLOP consumption.
We calculate theoretical FLOP requirements in a manner similar to Evci et al.
[2020] (exact details in the supplementary material).
Figure 1 contains test accuracies of select methods across two additional sparsity values: ( 0 . 5 , 0 . 95 ).
At lower sparsities (higher densities), RigL matches the performance of the dense baseline.
Performance further improves by training for longer durations.
Particularly, training RigL (ERK) twice as long at 90% sparsity exceeds the performance of iterative pruning while requiring similar theoretical FLOPs.
This validates the original authors’ claim that RigL (a sparse-to-sparse training method) outperforms pruning (a dense-to-sparse training method).
5.2 ResNet-50 on CIFAR
Table 3 &
, tabulated by performance and cost (below) , and plotted across densities (right) .
In each group below, RigL outperforms or matches existing sparse-to-sparse and dense-to-sparse methods.
Notably,
RigL 3 × at 90% sparsity and RigL 2 × at 80% sparsity surpass iterative pruning with similar FLOP consumption.
RigL 2 × (ERK) further improves performance but requires a larger training budget
[Table]
We see similar trends when training sparse variants of ResNet-50 on the CIFAR-100 dataset (Table 3, metrics reported as in Section 5.1).
We also include a comparison against sparse networks trained with the Lottery Ticket Hypothesis
(Frankle and Carbin [2018]) in Table 3—we obtain tickets with a commensurate performance for sparsities lower than 80%.
Finally, the choice of initialization scheme affects the performance and FLOP consumption by a greater extent than the method used itself, with the exception of SNFS (groups 1 and 2 in Table 3).
5.3 Hyperparameter Tuning
[Table]
( α, ∆ T ) vs Sparsities To understand the impact of the two additional hyperparameters included in RigL , we use a
Tree of Parzen Estimator (TPE sampler, Bergstra et al.
[2011]) via Optuna to tune ( α, ∆ T ) .
We do this for sparsities (1 − s ) ∈ { 0 . 1 , 0 . 2 , 0 . 5 } , and a ﬁxed learning rate of 0 . 1 .
Additionally, we set the sampling domain for α and ∆ T as [0 . 1 , 0 . 6] and { 50 , 100 , 150 , ..., 1000 } respectively.
We use 15 trials for each sparsity value, with our objective function as the validation accuracy averaged across 3 random seeds.
Table 4 shows the test accuracies of tuned hyperparameters.
While the reference hyperparameters (original authors, α = 0 . 3 , ∆ T = 100 ) differ from the obtained optimal hyperparameters, the difference in performance is marginal,
[Figure]
especially for ERK initialization.
This in agreement with the original paper, which ﬁnds α ∈ { 0 . 3 , 0 . 5 } , ∆ T = 100 to be suitable choices.
We include contour plots detailing the hyperparameter trial space in the supplementary material.
Learning Rate vs Sparsities We further examine if the ﬁnal performance improves by tuning the learning rate ( η ) individually for each sparsity-initialization pair.
We employ a grid search over η ∈ { 0 . 1 , 0 .
and ( α, ∆ T ) ∈ { (0 . 3 , 100) , (0 . 4 , 200) , (0 . 4 , 500) , (0 . 5 , 750) } .
As seen in Figure 3, η = 0 .
1 and η = 0 .
05 are close to optimal values for a wide range of sparsities and initializations.
Since these learning rates also correspond to good choices for the Dense baseline, one can employ similar values when training with RigL .
Results beyond Original Paper 6.1 Sparsity Distribution vs FLOP Consumption
[Figure]
While ERK initialization outperforms Random initialization consistently for a given target parameter count, it requires a higher FLOP budget.
Figure 4 compares the two initialization schemes across ﬁxed training FLOPs.
Theoretical FLOP requirement for Random initialization scales linearly with density (1 − s ) , and is signiﬁcantly lesser than ERK’s FLOP requirements.
Consequently, Random initialization outperforms ERK initialization for a given training budget.
6.2 Effect of Redistribution
and dataset in bold.
[Table]
One of the main differences of RigL over SNFS is the lack of layer-wise redistribution during training.
We examine if using a redistribution criterion can be beneﬁcial and bridge the performance gap between Random and ERK initialization
[Table]
Following Dettmers and Zettlemoyer [2020], during every mask update, we reallocate layer-wise density proportional to its average sparse gradient or momentum ( RigL -SG, RigL -SM).
Table 5 shows that redistribution signiﬁcantly improves RigL (Random), but not RigL (ERK).
We additionally plot the FLOP requirement against training steps and the ﬁnal sparsity distribution in Figure 5.
The layer-wise sparsity distribution largely becomes constant within a few epochs.
The ﬁnal distribution is similar, but more “extreme” than
ERK—wherever ERK exceeds/falls short of Random, redistribution does so by a greater extent.
By allocating higher densities to 1 × 1 convolutions ( convShortcut in Figure 5), redistribution signiﬁcantly increases the FLOP requirement—and hence, is not a preferred alternative to ERK.
Surprisingly, initializing RigL with the ﬁnal sparsity distribution in a manner similar to the Lottery Ticket Hypothesis results in subpar performance (group 3, Table 5).
[Figure]
Discussion
Evaluated on image classiﬁcation, the central claims of Evci et al.
[2020] hold true— RigL outperforms existing sparse-to-sparse training methods and can also surpass other dense-to-sparse training methods with extended training.
RigL is fairly robust to its choice of hyperparameters, as they can be set independent of sparsity or initialization.
We ﬁnd that the choice of initialization has a greater impact on the ﬁnal performance and compute requirement than the method itself.
Considering the performance boost obtained by redistribution, proposing distributions that attain maximum performance given a FLOP budget could be an interesting future direction.
For computational reasons, our scope is restricted to small datasets such as CIFAR-10/100.
RigL ’s applicability outside image classiﬁcation—in Computer Vision and beyond (machine translation etc.) is not covered here.
What was easy The authors’ code covered most of the experiments in their paper and helped us validate the correctness of our replicated codebase.
Additionally, the original paper is quite complete, straightforward to follow, and lacked any major errors.
What was difﬁcult Implementation details such as whether momentum buffers were accumulated sparsely or densely had a substantial impact on the performance of SNFS.
Finding the right (cid:15) for ERK initialization required handling of edge cases—when a layer’s capacity is exceeded.
Hyperparameter tuning ( α, ∆ T ) involved multiple seeds and was compute-intensive.
Communication with original authors We acknowledge and thank the original authors for their responsive communication, which helped clarify a great deal of implementation and evaluation speciﬁcs.
Particularly, FLOP counting for various methods while taking into account the changing sparsity distribution.
We also discussed experiments extending the original paper—as to whether the authors had carried out a similar study before.