Reproducibility Report
Rigging the Lottery: Making All Tickets Winners
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of Reproducibility
For a ﬁxed parameter count and compute budget, the proposed algorithm ( RigL ) claims to directly train sparse networksthat match or exceed the performance of existing dense-to-sparse training techniques (such as pruning). RigL does sowhile requiring constant Floating Point Operations (FLOPs) throughout training.
The technique obtains state-of-the-artperformance on a variety of tasks, including image classiﬁcation and character-level language-modelling.
Methodology
We implement RigL from scratch in Pytorch using boolean masks to simulate unstructured sparsity.
We rely on thedescription provided in the original paper, and referred to the authors’ code for only speciﬁc implementation detail suchas handling overﬂow in ERK initialization.
We evaluate sparse training using RigL for WideResNet-22-2 on CIFAR-10and ResNet-50 on CIFAR-100, requiring 2 hours and 6 hours respectively per training run on a GTX 1080 GPU.
Results
We reproduce RigL ’s performance on CIFAR-10 within 0.1% of the reported value.
On both CIFAR-10/100, the centralclaim holds—given a ﬁxed training budget, RigL surpasses existing dynamic-sparse training methods over a range oftarget sparsities.
By training longer, the performance can match or exceed iterative pruning, while consuming constant
FLOPs throughout training.
We also show that there is little beneﬁt in tuning RigL ’s hyper-parameters for every sparsity,initialization pair—the reference choice of hyperparameters is often close to optimal performance.
Going beyond the original paper, we ﬁnd that the optimal initialization scheme depends on the training constraint.
Whilethe Erdos-Renyi-Kernel distribution outperforms Random distribution for a ﬁxed parameter count, for a ﬁxed FLOPcount, the latter performs better.
Finally, redistributing layer-wise sparsity while training can bridge the performancegap between the two initialization schemes, but increases computational cost.
What was easy
The authors provide code for most of the experiments presented in the paper.
The code was easy to run and allowed usto verify the correctness of our re-implementation.
The paper also provided a thorough and clear description of theproposed algorithm without any obvious errors or confusing exposition.
was difﬁcult
Tuning hyperparameters involved multiple random seeds and took longer than anticipated.
Verifying the correctness ofa few baselines was tricky and required ensuring that the optimizer’s gradient (or momentum) buffers were sparse (ordense) as speciﬁed by the algorithm.
Compute limits restricted us from evaluating on larger datasets such as Imagenet.
Communication with original authors
We had responsive communication with the original authors, which helped clarify a few implementation and evaluationdetails, particularly regarding the FLOP counting procedure.
Submitted to ML
Introduction
Sparse neural networks are a promising alternative to conventional dense networks—having comparatively greaterparameter efﬁciency and lesser ﬂoating-point operations (FLOPs) (Han et al.
[2016], Ashby et al.
[2017], Srinivas et al.[2017]).
Unfortunately, present techniques to produce sparse networks of commensurate accuracy involve multiplecycles of training dense networks and subsequent pruning.
Consequently, such techniques offer no advantage overtraining dense networks, either computationally or memory-wise.
In the paper Evci et al.
[2020], the authors propose RigL , an algorithm for training sparse networks from scratch.
Theproposed method outperforms both prior art in training sparse networks, as well as existing dense-to-sparse trainingalgorithms.
By utilising dense gradients only during connectivity updates and avoiding any global sparsity redistribution,
RigL can maintain a ﬁxed computational cost and parameter count throughout training.
As a part of the ML Reproducibility Challenge, we replicate RigL from scratch and investigate if dynamic-sparsetraining confers signiﬁcant practical beneﬁts compared to existing sparsifying techniques.
Scope of reproducibility
In order to verify the central claims presented in the paper we focus on the following target questions:
• Does RigL outperform existing sparse-to-sparse training techniques—such as SET (Mocanu et al.
andSNFS (Dettmers and Zettlemoyer [2020])—and match the accuracy of dense-to-sparse training methods such
as iterative pruning (Zhu and Gupta [2018])?• RigL requires two additional hyperparameters to tune.
We investigate the sensitivity of ﬁnal performance tothese hyperparameters across a variety of target sparsities (Section 5.3).
• How does the choice of sparsity initialization affect the ﬁnal performance for a ﬁxed parameter count and aﬁxed training budget (Section 6.1)?
• Does redistributing layer-wise sparsity during connection updates (Dettmers and Zettlemoyer [2020]) improveRigL ’s performance?
Can the ﬁnal layer-wise
distribution serve as a good sparsity initialization scheme(Section 6.2)?
Methodology
The authors provide publicly accessible code 1 written in Tensorﬂow (Abadi et al.
To gain a better understandingof various implementation aspects, we opt to replicate RigL in Pytorch (Paszke et al.
Our implementationextends the open-source code 2 of Dettmers and Zettlemoyer [2020] which uses a boolean mask to simulate unstructuredsparsity.
Our source code will be made publicly accessible on Github and training plots available on WandB (Biewald[2020]) upon acceptance 3 .
Mask Initialization For a network with L layers and total parameters N , we associate each layer with a randomboolean mask of sparsity s l ,
l ∈
[ L ] .
The overall sparsity of the network is given by S = (cid:80) l s l N l N , where N l is theparameter count of layer l .
Sparsities s l are determined by the one of the following mask initialization strategies:
• Uniform: Each layer has the same sparsity, i.e., s l = S ∀ l .
Similar to the original authors, we keep the ﬁrst
layer dense in this initialization.
• Erdos-Renyi (ER): Following Mocanu et al.
[2018], we set s l ∝ (cid:16) 1 − C in + C out C in × C out (cid:17) , where C in , C out are the
in and out channels for a convolutional layer and input and output dimensions for a fully-connected layer.
• Erdos-Renyi-Kernel (ERK): Modiﬁes the sparsity rule of convolutional layers in ER initialization to includekernel height and width,
i
e., s l ∝ (cid:16) 1 − C in + C out + w + h C in × C out × w × h (cid:17) , for a convolutional layer with C in × C out × w × hparameters.
We do not sparsify either bias or normalization layers, since these have
[Table] [Table]
Mask Updates Every ∆ T training steps, certain connections are discarded, and an equal number are grown.
Unlike
SNFS (Dettmers and Zettlemoyer [2020]), there is no redistribution of layer-wise sparsity, resulting in constant FLOPsthroughout training.
Pruning Strategy Similar to SET and SNFS, RigL prunes f fraction of smallest magnitude weights in each layer.
Asdetailed below, the fraction f is decayed across mask update steps, by cosine annealing:
[Equation]
where, α is the initial pruning rate and T end is the training step after which mask updates are ceased.
Growth Strategy RigL ’s novelty lies in how connections are grown: during every mask update, k connections havingthe largest absolute gradients among current inactive weights (previously zero + pruned) are activated.
Here, k is chosento be the number of connections dropped in the prune step.
This requires access to dense gradients at each mask updatestep.
Since gradients are not accumulated (unlike SNFS), RigL does not require access to dense gradients at every step.
Following the paper, we initialize newly activated weights to zero.
Experimental Settings
4.1 Model descriptions
For experiments on CIFAR-10 (Alex Krizhevsky [2009]), we use a Wide Residual Network (Zagoruyko and Komodakis [2016]) with depth 22 and width multiplier 2, abbreviated as WRN-22-2.
For experiments on CIFAR-100(Alex Krizhevsky [2009]), we use a modiﬁed variant of ResNet-50 (He et al.
, with the initial 7 × 7 convolutionreplaced by two 3 × 3 convolutions (architecture details provided in the supplementary material).
4.2 Datasets and Training descriptions
We conduct our experiments on the CIFAR-10 and CIFAR-100 image classiﬁcation datasets.
For CIFAR-10, we use atrain/val/test split of 45k/5k/10k samples.
In comparison, the authors use no dedicated validation set, with 50k samplesand 10k samples comprising the train set and test set, respectively.
This causes a slight performance discrepancybetween our reproduction and the metrics reported by the authors (dense baseline has a test accuracy of 93.4% vs 94.1%reported).
However, our replication matches the paper’s performance when 50k samples are used for the train set (Table4).
We use a validation split of 10k samples for CIFAR-100 as well.
On both datasets, we train models for 250 epochs each, optimized by SGD with momentum.
Our training pipeline usesstandard data augmentation, which includes random ﬂips and crops.
When training on CIFAR-100, we additionallyinclude a learning rate warmup for 2 epochs and label smoothening of 0.1 (Goyal et al.
We also initialize thelast batch normalization layer (Ioffe and Szegedy [2015]) in each BottleNeck block to 0, following He et al.
4.3 Hyperparameters
RigL includes two additional hyperparameters ( α, ∆ T ) in comparison to regular dense network training.
In Sections5.1 and 5.2, we set α = 0 .
3 , ∆ T = 100 , based on the original paper.
Optimizer speciﬁc hyperparameters—learningrate, learning rate schedule, and momentum—are also set according to the original paper.
In Section 5.3, we tune these
[Table]
hyperparameters with Optuna (Akiba et al.
We also examine whether indivdually tuning the learning rate foreach sparsity value offers any signiﬁcant beneﬁt.
4.4 Baseline implementations
We compare RigL against various baselines in our experiments: SET (Mocanu et al.
[2018]), SNFS (Dettmers and
Zettlemoyer [2020]), and Magnitude-based Iterative-pruning (Zhu and Gupta [2018]).
We also compare against twoweaker baselines, viz., Static Sparse training and Small-Dense networks.
The latter has the same structure as the densemodel but uses fewer channels in convolutional layers to lower parameter count.
We implement iterative pruning withthe pruning interval kept same as the masking interval for a fair comparison.
4.5 Computational requirements
We run our experiments on a SLURM cluster node—equipped with 4 NVIDIA GTX1080 GPUs and a 32 core Intel
CPU.
Each experiment on CIFAR-10 and CIFAR-100 consumes about 1.6 GB and 7 GB of VRAM respectively and isrun for 3 random seeds to capture performance variance.
We require about 6 and 8 days of total compute time to produceall results, including hyper-parameter sweeps and extended experiments, on CIFAR-10 and CIFAR-100 respectively.
Results
Given a ﬁxed training FLOP budget, RigL surpasses existing dynamic sparse training methods over a range of targetsparsities, on both CIFAR-10 and 100 (Sections 5.1, 5.2).
By training longer, RigL matches or marginally outperformsiterative pruning.
However, unlike pruning, its FLOP consumption is constant throughout.
This a prime reason for usingsparse networks, and makes training larger networks feasible.
Finally, as evaluated on CIFAR-10, the original authors’choice of hyper-parameters are close to optimal for multiple target sparsities and initialization schemes (Section 5.3).
5.1 WideResNet-22 on CIFAR-10
Results on the CIFAR-10 dataset are provided in Table 2.
Tabulated metrics are averaged across 3 random seeds andreported with their standard deviation.
All sparse networks use random initialization, unless indicated otherwise.
[Figure]
While SET improves over the performance of static sparse networks and small-dense networks, methods utilizinggradient information (SNFS, RigL ) obtain better test accuracies.
SNFS can outperform RigL , but requires a much largertraining budget, since it (a) requires dense gradients at each training step, (b) redistributes layer-wise sparsity duringmask updates.
For all sparse methods, excluding SNFS, using ERK initialization improves performance, but withincreased FLOP consumption.
We calculate theoretical FLOP requirements in a manner similar to Evci et al.
[2020](exact details in the supplementary material).
Figure 1 contains test accuracies of select methods across two additional sparsity values: ( 0 . 5 , 0 . 95 ).
At lower sparsities(higher densities), RigL matches the performance of the dense baseline.
Performance further improves by trainingfor longer durations.
Particularly, training RigL (ERK) twice as long at 90% sparsity exceeds the performance ofiterative pruning while requiring similar theoretical FLOPs.
This validates the original authors’ claim that RigL (asparse-to-sparse training method) outperforms pruning (a dense-to-sparse training method).
5.2 ResNet-50 on CIFAR100
Table 3 & Figure
Benchmarking sparse ResNet
50s on CIFAR100, tabulated by performance and cost (below) , and plotted across densities (right) .
In each group below, RigL outperforms or matches existing sparse-to-sparse and dense-to-sparse methods.
Notably,
RigL 3 × at 90% sparsity and RigL 2 × at 80% sparsity surpass iterative pruning with similar FLOP consumption.
RigL 2 × (ERK) further improves performance but requires a larger training budget
[Table]
We see similar trends when training sparse variants of ResNet-50 on the CIFAR-100 dataset (Table 3, metrics reportedas in Section 5.1).
We also include a comparison against sparse networks trained with the Lottery Ticket Hypothesis
(Frankle and Carbin [2018]) in Table 3—we obtain tickets with a commensurate performance for sparsities lower than80%.
Finally, the choice of initialization scheme affects the performance and FLOP consumption by a greater extentthan the method used itself, with the exception of SNFS (groups 1 and 2 in Table 3).
5.3 Hyperparameter Tuning
[Table]
( α, ∆ T ) vs Sparsities To understand the impact of the two additional hyperparameters included in RigL , we use a
Tree of Parzen Estimator (TPE sampler, Bergstra et al.
[2011]) via Optuna to tune ( α, ∆ T ) .
We do this for sparsities(1 − s ) ∈ { 0 . 1 , 0 . 2 , 0 . 5 } , and a ﬁxed learning rate of 0 . 1 .
Additionally, we set the sampling domain for α and ∆ T as[0 .
1 , 0 . 6] and { 50 , 100 , 150 , ..., 1000 } respectively.
We use 15 trials for each sparsity value, with our objective functionas the validation accuracy averaged across 3 random seeds.
Table 4 shows the test accuracies of tuned hyperparameters.
While the reference hyperparameters (original authors,α = 0 . 3 , ∆ T = 100 ) differ from the obtained optimal hyperparameters, the difference in performance is marginal,
[Figure]
especially for ERK initialization.
This in agreement with the original paper, which ﬁnds α ∈ { 0 . 3 , 0 . 5 } , ∆ T = 100 tobe suitable choices.
We include contour plots detailing the hyperparameter trial space in the supplementary material.
Learning Rate vs Sparsities We further examine if the ﬁnal performance improves by tuning the learning rate( η ) individually for each sparsity-initialization pair.
We employ a grid search over η ∈ { 0 . 1 , 0 .
and( α, ∆ T ) ∈ { (0 . 3 , 100) , (0 . 4 , 200) , (0 . 4 , 500) , (0 . 5 , 750) } .
As seen in Figure 3, η = 0 .
1 and η = 0 .
05 are close tooptimal values for a wide range of sparsities and initializations.
Since these learning rates also correspond to goodchoices for the Dense baseline, one can employ similar values when training with RigL .
Results beyond Original Paper
[Figure]
6.1 Sparsity Distribution vs FLOP Consumption
[Figure]
While ERK initialization outperforms Random initialization consistently for a given target parameter count, it requires ahigher FLOP budget.
Figure 4 compares the two initialization schemes across ﬁxed training FLOPs.
Theoretical FLOPrequirement for Random initialization scales linearly with density (1 − s ) , and is signiﬁcantly lesser than ERK’s FLOPrequirements.
Consequently, Random initialization outperforms ERK initialization for a given training budget.
6.2 Effect of Redistribution
experiments, we mark the best metrics under each sparsity and dataset in bold
[Table]
One of the main differences of RigL over SNFS is the lack of layer-wise redistribution during training.
We examine ifusing a redistribution criterion can be beneﬁcial and bridge the performance gap between Random and ERK initialization
[Table]
Following Dettmers and Zettlemoyer [2020], during every mask update, we reallocate layer-wise density proportionalto its average sparse gradient or momentum ( RigL -SG, RigL -SM).
Table 5 shows that redistribution signiﬁcantly improves RigL (Random), but not RigL (ERK).
We additionally plotthe FLOP requirement against training steps and the ﬁnal sparsity distribution in Figure 5.
The layer-wise sparsitydistribution largely becomes constant within a few epochs.
The ﬁnal distribution is similar, but more “extreme” than
ERK—wherever ERK exceeds/falls short of Random, redistribution does so by a greater extent.
By allocating higher densities to 1 × 1 convolutions ( convShortcut in Figure 5), redistribution signiﬁcantly increasesthe FLOP requirement—and hence, is not a preferred alternative to ERK.
Surprisingly, initializing RigL with the ﬁnalsparsity distribution in a manner similar to the Lottery Ticket Hypothesis results in subpar performance (group 3, Table5).
[Figure]
Discussion
Evaluated on image classiﬁcation, the central claims of Evci et al.
[2020] hold true— RigL outperforms existingsparse-to-sparse training methods and can also surpass other dense-to-sparse training methods with extended training.
RigL is fairly robust to its choice of hyperparameters, as they can be set independent of sparsity or initialization.
We ﬁndthat the choice of initialization has a greater impact on the ﬁnal performance and compute requirement than the methoditself.
Considering the performance boost obtained by redistribution, proposing distributions that attain maximumperformance given a FLOP budget could be an interesting future direction.
For computational reasons, our scope is restricted to small datasets such as CIFAR-10/100.
RigL ’s applicability outsideimage classiﬁcation—in Computer Vision and beyond (machine translation etc.) is not covered here.
What was easy The authors’ code covered most of the experiments in their paper and helped us validate thecorrectness of our replicated codebase.
Additionally, the original paper is quite complete, straightforward to follow, andlacked any major errors.
What was difﬁcult Implementation details such as whether momentum buffers were accumulated sparsely or denselyhad a substantial impact on the performance of SNFS.
Finding the right (cid:15) for ERK initialization required handling ofedge cases—when a layer’s capacity is exceeded.
Hyperparameter tuning ( α, ∆ T ) involved multiple seeds and wascompute-intensive.
Communication with original authors We acknowledge and thank the original authors for their responsive communication, which helped clarify a great deal of implementation and evaluation speciﬁcs.
Particularly, FLOP counting forvarious methods while taking into account the changing sparsity distribution.
We also discussed experiments extendingthe original paper—as to whether the authors had carried out a similar study before.