Rare Gems: Finding Lottery Tickets at Initialization
Anonymous Author(s)
AffiliationAddress email
Abstract
Large neural networks can be pruned to a small fraction of their original size,with little loss in accuracy, by following a time-consuming “train, prune, re-train”approach.
Frankle & Carbin [8] conjecture that we can avoid this by training lotterytickets , i.e. , special sparse subnetworks found at initialization , that can be trainedto high accuracy.
However, a subsequent line of work [10, 34] presents concreteevidence that current algorithms for finding trainable networks at initialization, failsimple baseline comparisons, e.g. , against training random sparse subnetworks.
Finding lottery tickets that train to better accuracy compared to simple baselinesremains an open problem.
In this work, we resolve this open problem by proposing
G EM -M INER which finds lottery tickets at initialization that beat current baselines.
G EM -M INER finds lottery tickets trainable to accuracy competitive or better than
Iterative Magnitude Pruning (IMP), and does so up to 19 × faster.
Introduction
A large body of research since the 1980s empirically observed that large neural networks can becompressed or sparsified to a small fraction of their original size while maintaining their predictiveaccuracy [13–15, 18, 20, 25, 38].
Although several pruning methods have been proposed during thepast few decades, many of them follow the “ train, prune, re-train ” paradigm.
Although the abovemethods result in very sparse, accurate models, they typically require several rounds of re-training,which is computationally intensive.
Frankle & Carbin [8] suggest that this computational burden may be avoidable.
They conjecturethat given a randomly initialized network, one can find a sparse subnetwork that can be trained toaccuracy comparable to that of its fully trained dense counterpart .
This trainable subnetwork found atinitialization is referred to as a lottery ticket .
The study above introduced iterative magnitude pruning(IMP) as a means of finding these lottery tickets.
Their experimental findings laid the groundwork forwhat is now known as the Lottery Ticket Hypothesis (LTH).
Although Frankle & Carbin [8] establish that the LTH is true for tasks like image classification on
MNIST, they were not able to get satisfactory results for more complex datasets like CIFAR-10 and
ImageNet when using deeper networks, such as VGG and ResNets [9].
In fact, subsequent workbrought the effectiveness of IMP into question.
Su et al.
[34] showed that even randomly sampledsparse subnetworks at initialization can beat lottery tickets found by IMP as long as the layerwisesparsities are chosen carefully.
Gale et al.
[11] showed that methods like IMP which train ticketsfrom initialization cannot compete with the accuracy of a model trained with pruning as part of theoptimization process.
Frankle et al.
[9] explain the failures of IMP using the concept of linear mode connectivity which measures the stability of these subnetworks to SGD noise.
Extensive follow-up studies propose several heuristics for finding trainable sparse subnetworks at initialization [21, 35, 36].
However,subsequent work by Frankle et al.
[10], Su et al.
[34] show experimentally that all of these methods
[Figure]
fail simple sanity checks.
Most methods seem to merely identify good sparsities at each layer, butgiven those, random sparse subnetworks can be trained to similar or better accuracy.
Frankle et al.
[9] show that with a small modification, IMP can beat these sanity checks; the caveat isthat it no longer finds these subnetworks at initialization, but after a few epochs of warm-up training.
Since these subnetworks are found after initialization , IMP with warmup does not find lotterytickets .
As noted in the original work by Frankle & Carbin [8], the importance of finding trainable subnetworksat initialization is computational efficiency.
It is far preferable to train a sparse model from scratch,rather than having to deal with a large dense model, even if that is for a few epochs (which is what IMP with warmpup does).
To the best of our knowledge, the empirical validity of the Lottery Ticket
Hypothesis , i.e. , the hunt for subnetworks at initialization trainable to SOTA accuracy, remains anopen problem.
Our Contributions.
We resolve this open problem by developing G EM -M INER , an algorithmthat finds sparse subnetworks at initialization , trainable to accuracy comparable or better than IMPwith warm-up .
G EM -M INER does so by first discovering rare gems .
Rare gems are subnetworks atinitialization that attain accuracy far above random guessing, even before training.
Rare gems canthen be refined to achieve near state-of-the-art accuracy.
Simply put, rare gems are lottery tickets thatalso have high accuracy at initialization.
High accuracy at initialization is not a requirement for a network to be defined as a lottery ticket.
However, if our end goal is high accuracy after training, then having high accuracy at initializationlikely helps.
Rare gems found by G EM -M INER are the first lottery tickets to beat all baselines in [10, 34].
In Fig.
1we give a sketch of how our proposed algorithm G EM -M INER compares with IMP with warm start.
G EM -M INER finds these subnetworks in exactly the same number of epochs that it takes to trainthem, and is up to 19 × faster than IMP with warmup .
Related Work
Lottery ticket hypothesis.
Following the pioneering work of Frankle & Carbin [8], the search forlottery tickets has grown across several applications, such as language tasks, graph neural networksand federated learning [3, 4, 12, 22].
While the LTH itself has yet to be proven mathematically, theso-called strong LTH has been derived which shows that any target network can be approximated bypruning a randomly initialized network with minimal overparameterization [24, 26, 27].
Recently, ithas been shown that for such approximation results it suffices to prune a random binary network withslightly larger overparameterization [6, 33].
Pruning at initialization.
While network pruning has been studied since the 1980s, finding sparsesubnetworks at initialization is a more recently explored approach.
Lee et al.
[21] propose SNIP,which prunes based on a heuristic that approximates the importance of a connection.
Tanaka et al.
[Table]
[35] propose SynFlow which prunes the network to a target sparsity without ever looking at the data.
Wang et al.
[36] propose GraSP which computes the importance of a weight based on the Hessiangradient product.
The goal of these algorithms is to find a subnetwork that can be trained to highaccuracy.
Ramanujan et al.
[28] propose Edge-Popup (EP) which finds a subnetwork at initializationthat has high accuracy to begin with.
Unfortunately, they also note that these subnetworks are notconducive to further finetuning.
The above algorithms are all based on the idea that one can assign a “score” to each weight to measureits importance.
Once such a score is assigned, one simply keeps the top fraction of these scores basedon the desired target sparsity.
This may be done by sorting the scores layer-wise or globally acrossthe network.
Additionally, this can be done in one-shot (SNIP, GraSP) or iteratively (SynFlow).
Notethat IMP can also be fit into the above framework by defining the “score” to be the magnitude of theweights and then pruning globally across the network iteratively.
More recently, Alizadeh et al.
[1] propose ProsPr which utilizes the idea of meta-gradients throughthe first few steps of optimization to determine which weights to prune.
Their intuition is that thiswill lead to masks at initialization that are more amenable to training to high accuracy within a fewsteps.
While it finds high accuracy subnetworks, we show in Section 4.2 that it fails to pass the sanitychecks of Frankle et al. [10] and Su et al.
Sanity checks for lottery tickets.
A natural question that arises with pruning at initialization iswhether these algorithms are truly finding interesting and nontrivial subnetworks, or if their performance after finetuning can be matched by simply training equally sparse, yet random subnetworks.
Ma et al.
[23] propose more rigorous definitions of winning tickets and study IMP under severalsettings with careful tuning of hyperparameters.
Frankle et al.
[10] and Su et al.
[34] introduce severalsanity checks (i)
Random shuffling (ii)
Weight reinitialization (iii) Score inversion and (iv) Random
Tickets.
Even at their best performance, they show that SNIP, GraSP and SynFlow merely find a goodsparsity ratio in each layer and fail to surpass, in term of accuracy, fully trained randomly selectedsubnetworks, whose sparsity per layer is similarly tuned.
Frankle et al.
[10] show through extensiveexperiments that none of these methods show accuracy deterioration after random reshuffling.
Weexplain the sanity checks in detail in Section 4 and use them as baselines to test our own algorithm.
Pruning during/after training.
While the above algorithms prune at/near initialization, thereexists a rich literature on algorithms which prune during/after training.
Unlike IMP, algorithms inthis category do not rewind the weights.
They continue training and pruning iteratively.
Frankleet al.
[10] and Gale et al.
[11] show that pruning at initialization cannot hope to compete with thesealgorithms.
While they do not find lottery tickets, they do find high accuracy sparse networks.
Zhu &
Gupta [38] propose a gradual pruning schedule where the smallest fraction of weights are prunedat a predefined frequency.
They show that this results in models up to 95% sparsity with negligibleloss in performance on language as well as image processing tasks.
Gale et al.
[11] and Frankleet al. [10] also study this as a baseline under the name magnitude pruning after training .
Rendaet al.
[29] show that rewinding the learning rate as opposed to weights(like in IMP) leads to the bestperforming sparse networks.
However, it is important to remark that these are not Lottery Tickets,merely high accuracy sparse networks.
We contrast these different methods in Table 1 in terms of whether they prune at initialization, their finetunability, whether they pass sanity checks as well astheir computational costs.
Finally, we note that identifying a good pruning mask can be thought of as training a binary networkwhere the loss is computed over the element-wise product of the original network with the mask.
This has been explored in the quantization during training literature [5, 17, 31].
G EM -M INER : Discovering Rare Gems
Setting and notation.
Let S = { ( x i , y i ) } ni =1 be a given training dataset for a k -classificationproblem, where x i ∈ R d 0 denotes a feature vector and label y i ∈ { 1 , . .
, k } denotes its label.
Typically, we wish to train a neural network classifier f ( w ; x ) : R d 0 → { 1 , . .
. , k } , where w ∈ R ddenotes the set of weight parameters of this neural network.
The goal of a pruning algorithm isto extract a mask m = { 0 , 1 } d , so that the pruned network is denoted by f ( w ⊙ m ; x ) , where⊙ denotes the element-wise product.
We define the sparsity of this network to be the fraction ofnon-zero weights s =
∥ w ⊙ m ∥ 0 /d .
The loss of a classifier on a single sample ( x , y ) is denoted byℓ ( f ( w ⊙ m ; x ) , y ) , which captures a measure of discrepancy between prediction and reality.
In whatfollows, we will denote by w
0 ∈ R
d to be the set of random initial weights.
The type of randomnesswill be explicitly mentioned when necessary.
On the path to rare gems; first stop: Maximize pre-training accuracy.
A rare gem needs tosatisfy three conditions: (i) sparsity, (ii) non-trivial pre-training accuracy, and (iii) that it can befinetuned to achieve accuracy close to that of the fully trained dense network.
This is not an easy taskas we have two different objectives in terms of accuracy (pre-training and post-training), and it isunclear if a good subnetwork for one objective is also good for the other.
However, since pre-trainingaccuracy serves as a lower bound on the final performace, we focus on maximizing that first, andthen attempt to further improve it by finetuning.
[Figure]
Our algorithm is inspired by Edge-Popup(EP)
EP successfully finds subnetworkswith high pre-training accuracy
but it has twomajor limitations: (i) it does not work well inthe high sparsity regime ( e.g. , < 5% ), and (ii)most importantly, the subnetworks it finds arenot conducive to further finetuning.
In the following, we take G EM -M INER apartand describe the components that allow it tosurpass these issues.
G EM -M INER without sparsity control.
Much like EP, G EM -M INER employs a form ofbackpropagation, and works as follows.
Eachof the random weights [ w 0 ] i in the originalnetwork is associated with a normalized scorep i ∈ [0 , 1] .
These normalized scores becomeour optimization variables and are responsiblefor computing the supermask m , i.e. , thepruning pattern of the network at initialization.
For a given set of weights w and scores p , G EM -M INER sets the effective weights as w eff =
w ⊙ r ( p ) ,where r( · ) is an element-wise rounding function, and m = r ( p ) is the resulting supermask.
Therounding function can be changed, e.g. , r can perform randomized rounding, in which case p i wouldbe the probability of keeping weight w i in m .
In our case, we found that simple deterministicrounding, i.e. , r ( p i ) =
1 p i ≥ 0 .
5 works well.
At every iteration G EM -M INER samples a batch of training data and performs backpropagation onthe loss of the effective weights, with respect to the scores p , while projecting back to [0 , 1] whenneeded.
During the forward pass, due to the rounding function, the effective network used is indeeda subnetwork of the given network.
Here, since r ( p ) is a non-differentiable operation we use the
Straight Through Estimator (STE) [2] which backpropagates through the indicator function as thoughit were the identity function.
Note that this vanilla version of G EM -M INER is unable to exercise control over the final sparsity ofthe model.
For reasons that will become evident in below, we will call this version of our algorithm
G EM -M INER ( 0 ).
There is already a stark difference from EP: G EM -M INER ( 0 ) will automaticallyfind the optimal sparsity, while EP requires the target sparsity s as an input parameter.
However, at the same time, this also significantly limits the applicability of G EM -M INER ( 0 ) asone cannot obtain a highly sparse gem.
Shown as a dark blue curve in Fig.
2 is the sparsity of
G EM -M INER ( 0 ).
Here, we run G EM -M INER with a randomly initialized MobileNet-V2 network on
CIFAR-10.
Note that the sparsity stays around 50% throughout the run, which is consistent with theobservation by Ramanujan et al.
[28] that accuracy of subnetworks at initialization is maximized ataround 50% sparsity.
Algorithm 1:
Input: Dataset D = { ( x i , y i ) } , learning rate η , rounding function r ( · ) , number of epochs
E , freezing period T , target sparsity s ∈ [0 , 1]
Output: Mask m =
r ( p ) ⊙ q ∈ { 0 , 1 } d 1 c ← ln(1 /s )
E , q ← 1 d 2 w , p ← random vector in R d , random vector in [0 , 1] d 4 for j in 1 , 2 , . .
, E do for ( x i , y i ) ∈
D do w eff ← ( w ⊙ q ) ⊙ r ( p ) p ← p − η ∇ p ℓ ( f ( w eff ; x i ) , y i ) /*
STE */ p ← proj [0 , 1] d p if mod( j, T )
{+= 
 then I 1+} ← { i : q i = 1 } p sorted ← sort( p i ∈ I 1 ) p [Table]
Regularization and Iterative freezing.
G EM - [Table] M INER ( 0 ) is a good baseline algorithm for finding accurate subnetworks at initialization, but itcannot be used to find rare gems , which needto be sparse and trainable.
To overcome thislimitation, we apply a standard trick – we add aregularization term to encourage sparsity.
Thus,in addition to the task loss computed with theeffective weights, we also compute the L 2 or L 1norm of the score vector p and optimize overthe total regularized loss.
More formally, weminimize ℓ := ℓ task + λℓ reg , where λ is the hyperparameter and ℓ reg is either L 2 or L 1 normof the score vector p .
We call this variant G EM -M INER ( λ ), where λdenotes the regularization weight.
This namingconvention should explain why we called theinitial version G EM -M INER ( 0 ).
The experimental results in Fig.
2 show that thissimple modification indeed allows us to controlthe sparsity of the solution.
We chose to usethe L 2 regularizer, however preliminary experiments showed that L 1 performs almost identically.
By varying λ from λ = 0 to λ = 7 · 10 − 6and λ
= 1 . 5 · 10 − 5 , the final sparsity of the gem found by G EM -M INER ( λ ) becomes 2 .
5% and 1 . 4% ,respectively.
One drawback of this regularization approach is that it only indirectly controls the sparsity.
If wehave a target sparsity s , then there is no easy way of finding the appropriate value of λ such that theresulting subnetwork is s -sparse.
If we choose λ to be too large, then it will give us a gem that is waytoo sparse; too small a λ
and we will end up with a denser gem than what is needed.
As a simpleheuristic, we employ iterative freezing , which is widely used in several existing pruning algorithms,including IMP [8, 11, 38].
More specifically, we can design an exponential function s ( j ) =
e − cj forsome c > 0 , which will serve as the upper bound on the sparsity.
If the total number of epochs is Eand the target sparsity is s , we have s ( E ) =
e − cE = s .
Thus, we have c = ln (1 /s ) /E .
Once this sparsity upper bound is designed, the iterative freezing mechanism regularly checks thecurrent sparsity to see if the upper bound is violated or not.
If the sparsity bound is violated, it findsthe smallest scores, zeros them out, and freezes their values thereafter.
By doing so, we can guaranteethe final sparsity even when λ was not sufficiently large.
To see this freezing mechanism in action,refer the blue curve in Fig.
Here, the sparsity upper bounds (decreasing exponential functions) arevisualized as dotted lines.
Note that for the case of λ = 3 · 10 − 6 , the sparsity of the network doesnot decay as fast as desired, so it touches the sparsity upper bound around epoch 220.
The iterativefreezing scheme kicks in here, and the sparsity decay is controlled by the upper bound thereafter,achieving the specified target sparsity at the end.
The full pseudocode of G EM -M INER is provided in Algorithm 1.
There are two minor implementationdetails which differ from the explanation above: (i) we impose the iterative freezing every T epochs,not every epoch and (ii) iterative freezing is imposed even when the sparsity bound is not violated.
[Figure]
Experiments
In this section, we present the experimental results 1 for the performance of G EM -M INER acrossvarious tasks.
Tasks.
We evaluate our algorithm on (Task 1)
CIFAR-10 classification, on various networksincluding ResNet-20, MobileNet-V2, VGG-16, and WideResNet-28-2, (Task 2) TinyImageNetclassification on ResNet-18 and ResNet-50, and (Task 3)
Finetuning on the Caltech-101 [7] datasetusing a ResNet-50 pretrained on ImageNet.
The detailed description of the datasets, networks andhyperparameters can be found in Section A of the Appendix.
Proposed scheme.
We run G EM -M INER with an L 2 regularizer.
If a network reaches its bestaccuracy after E epochs of weight training, then we run G EM -M INER for E epochs to get a sparsesubnetwork, and then run weight training on the sparse subnetwork for another E epochs.
Comparisons.
We tested our method against five baselines: dense weight training and four pruningalgorithms: (i) IMP [9], (ii) Learning rate rewinding [29], denoted by Renda et al.
, (iii) Edge-Popup(EP)
[28], and (iv) Smart-Ratio (SR) which is the random pruning method proposed by Su et al.
We also ran the following sanity checks, proposed by Frankle et al.
[10]: (i) (Random shuffling):
Totest if the algorithm prunes specific connections, we randomly shuffle the mask at every layer.
(ii)(Weight reinitialization): To test if the final mask is specific to the weight initialization, we reinitializethe weights from the original distribution.
(iii) (Score inversion): Since most pruning algorithmsuse a heuristic/score function as a proxy to measure the importance of different weights, we invertthe scoring function to check whether it is a valid proxy.
More precisely, this test involves pruningthe weights which have the smallest scores rather than the largest.
In all of the above tests, if theaccuracy after finetuning the new subnetwork does not deteriorate significantly, then the algorithm ismerely identifying optimal layerwise sparsities.
4.1 Rare gems obtained by G EM -M INER
Task 1.
Fig.
3 shows the sparsity-accuracy tradeoff for various pruning methods trained on CIFAR10 using ResNet-20, MobileNet-V2, VGG-16 and WideResNet-28-2.
For each column (network),we compare IMP, IMP with learning rate rewinding (Renda et al.), G EM -M INER , EP, and SR in twoperformance metrics: the top row shows the accuracy of the subnetwork after weight training andbottom row shows the result of the sanity checks on G EM -M INER .
[Figure]
As shown in the top row of Fig.
3, G EM -M INER finds a lottery ticket at initialization.
It reachesaccuracy similar to IMP after weight training.
Moreover, for in the sparse regime (e.g., below 1.4%for ResNet-20 and MobileNet-V2), G EM -M INER outperforms IMP in terms of post-finetune accuracy.
The bottom row of Fig.
3 shows that G EM -M INER passes the sanity check methods.
For all networks,the performance in the sparse regime (1.4% sparsity or below) shows that the suggested G EM -M INERalgorithm enjoys 3–10% accuracy gap with the best performance among variants.
The results in thetop row show that G EM -M INER far outperforms the random network with smart ratio (SR).
[Figure]
Tasks 2–4.
Fig.
4 shows the sparsity-accuracy tradeofffor Tasks 2–4.
Similar to Fig.
3, the top row reports the accuracy after weight training, and the bottom row containsthe results of the sanity checks.
As shown in Fig.
4a and Fig.
4b, the results for Task 2show that (i)
G EM -M INER achieves accuracy comparableto IMP as well as Renda et al.
(IMP with learning raterewinding) even in the sparse regime, (ii) G
EM -M INER hasnon-trivial accuracy before finetuning (iii)
G EM -M INERpasses all the sanity checks, and (iv) G EM -M INER outperforms EP and SR.
These results show that G EM -M INERsuccessfully finds rare gems even in the sparse regime for
Task 2 .
Fig.
4c shows the result for Task 3 .
Unlike other tasks,
G EM -M INER does not reach the post-finetune accuracyof IMP, but G EM -M INER enjoys over an 8% accuracygap compared with EP and SR.
Moreover, the bottom rowshows that G EM -M INER has over 20% higher accuracythan the sanity checks below 5% sparsity showing thatthe subnetwork found by G EM -M
INER is unique in thissparse regime.
4.2 Comparison to ProsPr
Alizadeh et al.
[1] recently proposed a pruning at init method called ProsPr which utilizes metagradients through the first few steps of optimization to determine which weights to prune, therebyaccounting for the “trainability” of the resulting subnetwork.
In Table 2 we compare it against G EMM INER on ResNet-20, CIFAR-10 and also run the (i) Random shuffling and (ii) Weight reinitializationsanity checks from Frankle et al.
We were unable to get ProsPr using their publicly availablecodebase to generate subnetworks at sparsity below 5% and therefore chose that sparsity.
Notethat G EM -M
INER produces a subnetwork that is higher accuracy despite being more sparse.
Afterfinetuning for 150 epochs, our subnetwork reaches 83 .
4% accuracy while the subnetwork found by
[Table]
ProsPr only reaches 82 .
67% after training for 200 epochs.
More importantly, ProsPr does show after the or
Therefore, as Frankle et al.
[10] remark, it is likely that it is identifying good layerwise sparsity ratios,rather than a mask specific to the initialized weights.
4.3 Observations on G EM -M INER
Convergence of accuracy and sparsity.
Fig.
5 shows how the accuracy of G EM -M INER improvesas training progresses, for MobileNet-V2 on CIFAR-10 at sparsity 1 . 4% .
This shows that G EMM INER , reaches high accuracy even early in training, and can be finetuned to accuracy higher thanthat of IMP (which requires 19 × the runtime than our algorithm). [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Figure]
High pre-finetune accuracy.
A shown in
Fig. 6, G EM -M INER finds subnetworks at initialization that have a reasonably high accuracyeven before the weight training, e.g., above 90%accuracy for 1.4% sparsity in VGG-16, and 85%accuracy for 1.4% sparsity in MobileNet-V2.
Note that, in contrast, IMP and SR have accuracy scarcely better than random guessing atinitialization.
Clearly, G EM -M INER fulfills itsobjective in maximizing accuracy before finetuning and therefore finds rare gems – lotterytickets at initialization which already have highaccuracy.
Limitations of G EM -M INER .
We observedthat in the dense regime (50% sparsity, 20% sparsity), G EM -M INER sometimes performs worsethan IMP.
While we believe that this can beresolved by appropriately tuning the hyperparameters, we chose to focus our attention on thesparse regime.
We would also like to remark that
G EM -M
INER is fairly sensitive to the choice ofhyperparameters and for some models, we had to choose different hyperparameters for each sparsityto ensure optimal performance. Though this occurs rarely, we also find that an extremely aggressivechoice of λ can lead to layer-collapse where one or more layers gets pruned completely. This happenswhen all the scores p of that layer drop below 0 .
[Table]
Layer-wise sparsity.
We compare the layer-wise sparsity pattern of different algorithms for ResNet20 trained on CIFAR-10 in Fig.
Both G EM -M INER and IMP spend most of their sparsity budgeton the first and last layers.
By design, SR assigns 30% sparsity to the last layer and the budget decayssmoothly across the others.
EP maintains the target sparsity ratio at each layer and therefore is alwaysa horizontal line.
How does G EM -M INER resolve EP’s failings?
An open problem from Ramanujan et al.
[28] iswhy the subnetworks found by EP are not fine-tunable.
While G EM -M INER is significantly differentfrom EP, it is reasonable to ask which modification allowed it to find lottery tickets without forgoinghigh accuracy at initialization.
Table 3 shows how we can modify EP to improve the pre/post-finetuneperformance, for ResNet-20, CIFAR-10 at 0.59% sparsity.
Here, we compare EP, G EM -M INER , aswell as three EP variants that we construct.
(i) (Global EP) : is a modification where the bottom- kscores are pruned globally, not layer-wise.
This allows the algorithm to trade-off sparsity in onelayer for another.
(ii) (Gradual pruning) reduces the parameter k gradually as opposed to setting itto the target sparsity from the beginning.
(iii) (Regularization) : we add an L 2 term on the score pof the weights to encourage sparsity.
The results indicate that global pruning and gradual pruningsignificantly improve both the pre and post-finetune accuracies of EP.
Adding regularization does notimprove the performance significantly.
Finally, adding all three features to EP allows it to achieve63.72% accuracy, while G EM -M INER reaches 66.15% accuracy.
It is important to note that even withall three features, EP is inherently different from G EM -M INER in how it computes the supermaskbased on the scores.
But we conjecture that aggressive, layerwise pruning is the key reason for EP’sfailings.
[Table]
Applying G EM -M INER for longer periods.
Recall that G EM -M
INER uses 19 × fewer trainingepochs than iterative train-prune-retrain methods like IMP [9] and Learning rate rewinding (Rendaet al.
[29]), to find a subnetwork at 1 . 4% sparsity which can then be trained to high accuracy.
Here,we consider a long version of G EM -M INER to see if it can benefit if it is allowed to run for longer.
Table 4 shows the comparison of post-finetune accuracy for G EM -M INER , L ONG G EM -M INER , IMP and Renda et al.
[29] tested on ResNet-20, CIFAR-10, sparsity=1.4% setting.
Conventional G EM -M INER , applies iterative freezing every 5 epochs to arrive at the target sparsity in 150 epochs.
L ONG G EM -M INER instead prunes every 150 epochs and therefore reaches the target sparsity in3000 epochs.
We find that applying G EM -M INER for longer periods improves the post-finetune accuracy in thisregime by 1.5%.
This shows that given equal number of epochs, G EM -M INER , which prunes atinitialization, can close the gap to Learning rate rewinding [29] which is a prune-after-trainingmethod.
Conclusion
In this work, we resolve the open problem of pruning at initialization by proposing G EM -M INERthat finds rare gems – lottery tickets at initialization that have non-trivial accuracy even beforefinetuning and accuracy rivaling prune-after-train methods after finetuning.
Unlike other methods,subnetworks found by G EM -M INER pass all known sanity checks and baselines.
Moreover, we showthat G EM -M INER is competitive with IMP despite not using warmup and up to 19 × faster.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary. • Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Refer Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes]
Our
codebase can be found at https://anonymous.4open.science/r/pruning_is_enough-F0B
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Refer the appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Wherever it was computationally feasible, we ran 3 trials and report the plots with an error bar
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Refer the appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the original papers that introduced ResNets, VGGs etc.
(b) Did you mention the license of the assets?
[Yes] We cite the original papers that introduced CIFAR-10, TinyImagenet etc.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
[N/A]
[Table]