Sparse Winning Tickets are Data-Efficient Image Recognizers
Anonymous Author(s)
AffiliationAddress email
Abstract
Improving performance of deep networks in data limited regimes has warrantedmuch attention.
In this work, we show that “winning tickets” (small sub-networks)obtained via magnitude pruning based on the lottery ticket hypothesis [1], apartfrom being sparse are also effective recognizers in data limited regimes.
Basedon extensive experiments, we find that in low data regimes (datasets of 50-100examples per class), sparse winning tickets substantially outperform the originaldense networks.
This approach, when combined with augmentations or finetuning from a self-supervised backbone network, shows further improvements inperformance by as much as 16% (absolute) on low sample datasets and long-tailedclassification.
Further, sparse winning tickets are more robust to synthetic noise anddistribution shifts compared to their dense counterparts.
Our analysis of winningtickets on small datasets indicates that, though sparse, the networks retain densityin the initial layers and their representations are more generalizable.
Code will bemade available after acceptance.
Introduction
Deep convolutional networks [2] have achieved wide success on a variety of tasks, but they dodemand large amounts of data.
However, there exists various domains and tasks where training datalabeled or unlabeled - is limited.
Often, in such data limited regimes, transfer learning has emerged asa dominant approach [3] as it achieves superior performance compared to training from scratch withrandom initialization [4].
In some domains that are highly specialized training performant modelscan still be a challenge, e g. in scientific or medical images [5], or just images from a differentdistribution - differing vastly in size, color, or channels - than seen in typical image datasets.
Insuch cases, training large-capacity data-hungry networks might be less performant than trainingmodels of similar architecture and lower capacity [6].
Much of the literature on sparser and lowercapacity networks has focused more on efficient models and mobile deployment [7].
But it is alsoknown [8] that sparsity as a regularization could reduce overfitting.
Thus it is interesting to studysparse networks in the context of low data or data efficient regimes (Fig. 1).
One of the popular ways of obtaining a sparse network of similar architecture but lower capacity is viapruning [9].
1 formulated the lottery ticket hypothesis (LTH) and proposed iterative magnitude basedpruning (IMP).
They demonstrated that dense networks can be magnitude pruned to obtain highlysparse sub-networks (“winning tickets”) which can match or potentially outperform the original densenetwork.
The caveat being that the winning ticket needs to use the same random initialization asthe original dense network.
This approach also induces an inductive bias specific to the task to belearned, which leads to a better network when compared to a dense network of similar size trainedfrom scratch [10].
Further, such pruning was recently found to reduce sample complexity in theoryand practice.
11 also suggest that the number of samples required to achieve zero generalization erroris proportional to the number of the non-pruned weights in the hidden layer.
Thus, despite indicationsof sparse networks being suitable for low data regimes, there is limited work [12] studying them in the context of language, and no prior work which evaluates their effectiveness for image recognitiontasks in the low data setting.
In this work , we provide empirical evidence of the suitability of sparse networks identified using IMP for data-limited regimes.
We conduct extensive experiments to compare the relationshipbetween network capacity, datasize, and image classification accuracy.
We first show that sparsewinning tickets have substantially better performance on low (50-100 samples per class) data regimescompared to their original dense counterparts.
Next, we show that sparse winning tickets for dataefficient learning compliments and can be combined with existing methods such as augmentationsor fine-tuning from a self-supervised backbone to further boost performance by as much as 16%.
We then evaluate the generalization capacity of the networks through detailed robustness tests onsynthetic and domain-shifted datasets.
We also evaluate on special data-deficient cases like long-tailedclassification to compare performance when some classes have far few samples than others.
Finally,we try to understand properties of sparse networks that lead to its superior performance.
We find thatboth capacity and connectivity together are important for generalization.
Our key contributions canbe summarized as follows:
We find that the sparsity of the winning ticket increases with reduction in training data.
Demonstrate that winning tickets can easily combine with existing data-efficient training strategies such as augmentations to outperform similarly trained dense networks on low data regimes by as much as 16%.
Show that winning tickets, on low data regimes, are robust to synthetic noise and domain shifts, and show improved performance on long-tailed classification.
Our analysis indicates that on data-limited regimes sparse winning tickets continue to pre-
serve density in initial layers, and both capacity and connectivity jointly are important for generalization.
Related Works
Sample-Efficient Image Recognition.
Learning from few samples and less data has been a longstanding topic of interest in machine learning.
While transfer learning, particularly from ImageNetpretrained models [3] is the most popular approach, semi-supervised techniques [13] and selfsupervision [14] have been successful when there is an abundance of unlabeled data in the samedomain.
All of these typically involve pretraining the model on data from a similar domain, or thesame domain, often with a variety of augmentations and then tuning it on a few labeled examples.
These are complimentary to our approach, and we explore combining augmentations and transferringin our experiments.
Few-shot methods are also related and they usually look at tuning just the finalclassification layer [5, 15].
A different approach is meta-learning [16], which aims to identify anetwork from the support classes, seen during pre-training, and which can easily adapt to the targetset of classes in few steps.
Our work is different from typical few-shot methods in two ways: 1 ⃝ We do not limit ourselves to transfer learning and also focus on training from scratch directly from thesmall dataset; 2 ⃝ We cover data sets with approximately 50-100 samples per class which lies betweenmost few shot settings and typical deep learning settings with more data.
There are also works that directly use small datasets without external dependence on pre-training,often relying on training strategies to prevent overfitting.
17 proposes to use the cosine loss while18 uses the t-Distributed von Mises-Fisher (t-vMF) loss along with the standard cross entropy fortraining.
These regularising loss functions reduce intra-class variance.
Some other methods relyon extracting useful geometric priors to help avoid over-fitting: 19 use preset cosine filters and 20introduces full convolutions which exploits the absolute spatial location of objects in the imageand improves translation invariance to strengthen the visual inductive prior.
In this work, we coverdatasets containing roughly 50-100 samples per class and the above methods have shown strongperformance in this regime [21] and offer easy implementation.
Therefore, in this work we compareagainst these methods and study if they supplement pruning techniques.
Pruning.
Among pruning based methods, the iterative magnitude pruning (IMP) (described in
Sec. 3.1) proposed by [1] is most relevant to our work.
IMP is a process that helps identify sparse subnetworks (“winning tickets”) within a larger dense network that is capable of matching or potentiallyout-performing the dense network.
This has proved successful on a variety of tasks including imageclassification [22], natural language processing [23, 24], and generative modelling [25].
While IMPhas been popular for obtaining sparse networks, we are not aware of works that explore its relationshipto data-efficiency.
In this paper, we carry out detailed experiments to validate if sparsity particularly“winning tickets” obtained via IMP improves data-efficiency of deep neural networks.
Methodology and Experiments
Our approach primarily uses iterative magnitude pruning (IMP) and we study the performance of the“winning tickets” in low data regimes.
In this section, we describe IMP, and provide an overview of our experimental studies including the basic experimental setup, augmentation strategies we compareagainst, and the datasets used for different studies.
3.1 Preliminary: Iterative Magnitude Pruning (IMP)
1 introduces IMP as a way to identify sparse sub-networks within a dense network that match orexceed the performance of the full dense network.
They view these sub-networks as having wonthe initialization “lottery” and refer to them as “winning tickets”.
We use the IMP procedure withrewinding [26, 27] and describe it briefly below.
Let f ( x , θ ) denote the neural network, where θ denotes the parameters of the network and x denoteits input.
Then the sub-network can be characterized by a binary mask m and can be applied to theoriginal network via a Hadamard product ( ⊙ ) to obtain the sub-network f ( x , θ ⊙ m ) .
For a networkinitialized with θ 0 , the IMP process aims to find a final mask m of a specified sparsity ratio ( s ) asfollows:
initialize m to be all ones mask.
train f
( x , θ 0 ⊙ m ) for r steps to get θ r .
This will be used as the initialization for the later pruning iterations.
3. continue to train f ( x , θ r ⊙ m ) to convergence.
4. remove a portion of the weights (determined by the pruning rate p ) with the smallest magnitudes
from θ ⊙ m .
Update m by setting the removed weights to 0.
repeat steps 3, 4 until the specified sparsity ratio ( s ) of pruned vs. original weights is achieved.
Note that when r = 0 , the above algorithm reduces to IMP without rewinding.
Rewinding is foundto be essential for successful and stable identification of winning tickets in large networks [26, 27].
Since this is an iterative process, we obtain sub-networks of different sparsity at each stage.
In all ourexperiments, the sub-network that has the best performance, across all iterations, on the validation setis termed as the “sparse winning ticket”.
3.2 Overview of Experimental Studies
We provide a broad overview of our experiments before presenting the experimental setup and datasetsused.
(1) We first study how network capacity of models pruned via IMP affects classification performance
when training on datasets of different sizes.
Since augmentations are often employed during training
of models in low-data regimes, we also perform this study with models trained using differentaugmentation strategies.
We compare performance of dense models with just augmentations againstthose with both augmentation and pruning.(2)
Next to study if sparsity prevents over-fitting by avoiding memorization, we evaluate performanceof the pruned models on a (a) synthetically transformed dataset, and on a (b) domain-shifted dataset.(3)
Aside from augmentations, we also compare IMP against other training techniques employedin low-data regimes as discussed in Sec. 2, as well as fine-tuning from ImageNet and SimCLRbackbones.(4)
To verify the generalizability of the approach and measure performance of sparse winning ticketsin data-limited regimes, we perform studies on (a) 3 real datasets with few (50-80) images per class,and also (b) simulated datasets with imbalanced (long-tail of) classes.(5)
Finally, we try to understand why sparse winning tickets are able to perform well in datalimited regimes.
We do this by analyzing network capacity and connectivity, and also how pruningaffects the layers - which layers are getting pruned, the layer-wise representation similarity, and thegeneralizability of the representations.
3.3 Experimental Setup
We use the ResNet model (specifically ResNet-18 unless otherwise specified) in all our experiments.
Since we are working with smaller-sized images (often 32x32 or 64x64), we modify the initialconvolution layer to a 3x3 kernel, with padding 1 with no max-pooling.
For images of size 224 wedo not make any modifications.
We follow the IMP lottery ticket finding procedure with 16 pruningiterations where 20% of the weights are pruned after each iteration.
In each pruning iteration, themodel is trained for 200 epochs to minimise the cross entropy loss using the SGD optimizer withweight decay 0.0005.
The initial learning rate is set to 0.1 and then cosine decayed for 200 epochs.
For fine-tuning experiments we use a lower learning rate of 0.001.
During the first iteration, we usethe rewinding technique and set r = 2 epochs.
3.4 Augmentation Strategies
Augmentations are one of the default strategies applied when working with limited data.
In thiswork, we look at augmentations in two scenarios: (1) as a comparison to the performance of thewinning ticket i.e. comparing the dense model with augmentations to that of a pruned model withoutaugmentations, and (2) we combine augmentations with IMP to see the benefit of augmentation inaddition to pruning.
We carry out experiments with four different augmentation strategies each ofdifferent strength.
The strategy Basic consists of just RandomCrop and RandomFlip.
All the otherstrategies are composed on top of the Basic augmentation.
Contrast strategy additionally consists of
ColorJitter and RandomGray [14].
Auto uses Basic augmentations and also the best policies identifiedfor each dataset by AutoAugment [28].
Rand is based on RandAugment [29], and composes fourrandomly selected augmentations in addition to the Basic augmentations.
We can potentially orderthese 4 augmentation strategies by increasing strength as: Basic < Contrast ≈ Auto < Rand.
3.5 Datasets
CIFAR10 [30] is our primary dataset for all analysis experiments.
It consists of 60,000 colour images(train: 50,000, val: 10,000) of size 32x32 split into 10 classes.
For experimental comparisons to thedata-limited regime, we sub-sample it at various sizes while maintaining the original class-size ratios.
E.g., at 1% data size, the training set contains 50 images per class.
We aways evaluate on the fullvalidation set in all our experiments.
For the robustness and extended experiments, we use additionaldatasets described in the corresponding sections.
CIFAR10-C [31].
We use the CIFAR10-C (C for corrupted) dataset to evaluate the robustness ofthe models to (unseen) synthetic transforms .
It contains 19 different transforms each of 5 differentstrength levels.
These transforms can be broadly classified into - Noise (e.g. impulse noise), Blur(e.g.
motion blur),
Weather (e.g. snow), Digital (e.g. JPEG), and Extra (e.g. brightness).
CIFAR10.2 [32].
To evaluate the network’s performance on domain shifts , we use the CIFAR10.2dataset derived from TinyImages [33].
It contains the exact same classes as the original CIFAR10dataset but has both different and harder collection of images ideal for evaluating distributional shifts.
CLaMM [34], ISIC [35] and EuroSAT [36].
Aside from experiments on subsampled versionsof CIFAR, we use three diverse low-data regime datasets covering different domains with varyingnumber of classes and images per class.
The EuroSAT dataset [36] contains 64 × 64 sized satelliteimages from 10 classes with 50 images per class captured from Sentinel-2 satellite.
It consists ofmultiple channels.
We follow the procedure given in [36] to use the RGB channels only.
ISIC2018 dataset [35] contains 224 × 224 sized dermoscopic skin lesion images from 7 classes with 80samples per class.
The CLaMM dataset [34] contains 224 × 224 sized gray-scale latin scripts fromhand-written books from 12 classes with 50 images per class.
CIFAR100 [30] For studies on the performance of the sparse networks on imabalanced and longtailed classification tasks we derive a dataset based on CIFAR100.
We selectively sample the
CIFAR100 dataset such that the class-wise image count follows a long-tailed distribution (defined byan exponential curve given by: Nn × λ j/ ( n − 1) where N is total number of samples in the dataset, j isthe class index and n (=100) is the number of classes in the dataset.
The degree of “long-tailedness”is defined by the imbalance factor ( λ ), i.e a lower value indicates more number of classes have alower sample count.
The derived long-tailed class-wise distribution of CIFAR100 is visualized in the
Appendix C.
Results
[Figure]
We now present the results of our experimental studies.
[Figure]
[Figure]
We first verify our main claim and show the suitability of sparse winning tickets for data-limitedregimes.
Fig.
3a (and Fig. 1) presents the results of training a ResNet-18 network on varying CIFAR10subsets of sizes 100% , 50% , 20% , 10% , 2% , and 1% .
At the lower data sizes the sparse winningticket (denoted as WT ) shows considerably improved performance compared to the dense model.
We also investigate how pruning compares with augmentation strategies and if both of these canbe combined.
Fig.
3a shows the results of the best performing dense and sparse networks withaugmentation.
The best is chosen across all augmentations for each data size and are indicated by
Dense+Aug ∗ (or D+Aug ∗ ) for the dense model and WT+Aug ∗ for the winning ticket.
We can seethat across all data sizes, models pruned with augmentation outperforms unpruned ones and as thedata size decreases the gap between the two widens indicating that sparse networks are suitable fordata-limited regimes.
Fig.
2 presents detailed comparisons of the performance of the model on the different data subsets(each subplot) at different sparsity levels (subplot x-axis) for the different augmentation strategies(lines of different styles).
In almost all cases, the best performing model is sparse i.e density < 100% .
While in the higher data regimes (top row), there is a decrease in performance with increased sparsity, the trend is reversed in the low data regimes (bottom row).
More importantly, at the lower datasizes i.e 2% , 1% , it’s clear that the sparse winning tickets (indicated by stars in Fig. 2) significantlyoutperform the dense (100% density) models with augmentations.
Further, among the augmentationstrategies, Auto-augment seems to consistently yield the best performance, except in the 1% data sizewhere Rand-augment does much better.
We further discuss the relevance of applying augmentationsduring the pruning stage in Appendix.
{+A.4.
+}
Winning tickets are robust to distributional shifts.
[Figure]
As the data size gets smaller, the chance of over-fitting and memorization of the training samplesincreases.
Thus, to study robustness, we evaluate the performance of the identified sparse winningtickets on two types of distributional shifts : (a) synthetic transforms via the CIFAR10-C dataset [31]and (b) domain-shifts via the CIFAR10.2 dataset [32].
Note that we only perform evaluation (i.e.inference) on these datasets to study the robustness of the winning tickets to these transformations.
We evaluate the network’s performance on: (a) synthetic transforms on the CIFAR10-C dataset.
Row-1 in Fig. 4 and Fig.
3b visualizes theaverage accuracy of the models across all 19 unseen transforms.(b) domain shifted CIFAR10.2 dataset consisting of wider variety of harder images from the
CIFAR10 classes.
Row-2 in Fig.
4 and Fig.
3b presents the results.
In both cases, similar to performance on the CIFAR10 data (Fig. 2) on 10% and lower training datasizes, sparser winning tickets generalize better to these unseen transforms and distribution shifteddata, much more so with augmentations.
The difference in classification accuracy between the sparsewinning ticket and dense model is also more clearly noted from Figures.
3b and 3c where we look atthe just the best performing models (as determined on CIFAR10 val.
set) - dense model and winningtickets with and without augmentations.
These results clearly indicate that sparse winning tickets, [Table] [Table] when trained as typical with augmentations, are capable of avoiding memorization of training samplesat the smaller data sizes, and much more so than the dense models.
This reinforces its suitability fordata-limited regimes.
4.3 IMP complements existing data-efficient training.
Table.
1 summarizes results using IMP to supplement and complement existing data-efficient methods.
In particular, we apply IMP in combination with methods that have been shown to work well on smalldatasets: fine-tuning from ImageNet and SimCLR [14] backbones, self-regularizing cosine loss [17],use of full convolutions [20] (Full Conv.), as well as 2 emergent methods, t-VMF loss [18] andharmonic nets [19].
Networks are trained using the best identified augmentation strategy at each datasize i.e Auto-augment at 2% and Rand-augment at 1% data-sizes respectively.
With all techniques,
IMP further improves performance on the CIFAR10 val.
set on an average by 8% and 15% at 2%and 1% data sizes respectively.
IMP shows more significant gains with non fine-tuning methods, as(absolute) magnitude based pruning strategy does not work well with pre-trained initializations [37],though we do see substantial improvements even on fine-tuning.
4.4 Generalization to low sample datasets.
To evaluate generalization to datasets other than subsets of CIFAR10, we experiment on three diversedatasets.
The EuroSAT [36], ISIC [35], and CLaMM [34] datasets cover different domains - satelliteimagery, medical images, and books respectively with 7-12 classes but few images (50-80) per class.
Table.
2 compares performance of the dense model and IMP, using the Basic augmentation strategyand in combination with existing data efficient techniques for low sample data.
The winning ticketsoutperform the dense model on an average by 12%, 2.3%, and 5.5% on the CLaMM, ISIC and
EuroSAT datasets respectively.
4.5 Long-tailed classification.
Long-tailed classification is a specific data-limited setting where some classes of the dataset containvery limited samples compared to the rest.
The presence of the dominant class can lead to over-fitting.
Therefore, we evaluate the effectiveness of sparse networks in such a dataset derived from CIFAR100(see.
Sec. 3.5).
Table.
3 summarizes these results and we can see that the winning ticket has slightlybetter performance than the original dense network.
It is important to note here that the total datasetsize after imbalancing is between 40% and 20% of the actual data size (100-200 examples per class)and therefore, the performance improvements are smaller.
Nevertheless, the gains indicate that sparsenetworks are able to avoid over-fitting due to class-imbalance and not just uniformly reduced datasize.
[Table] [Table]
- ent sub-
networks (backbones) trained on CIFAR10 data to evaluate generalizability of representations.
Discussion
In this section, we focus on why winning tickets showcase superior performance in data-limitedregimes, and try to infer their properties leading to the performance gains.5.1 Network capacity and connectivity are important.
During pruning even as the network capacity is reduced, the connectivity between layers is alsochanged.
We study the original dense network, and the sparse winning ticket with 3 additionalnetworks: (1) A small dense network which is a dense network but has overall parameters reduced tomatch the winning ticket.
(2) A randomly pruned network where we take the original dense model,but randomly set some of the weights to zero to match the sparsity of the winning ticket.
(3) Asame-layer-sparse network which follows the exact same layer-wise densities as the winning ticketbut the connectivity between the layers are random i.e. the binary mask at each layer is randomlyinitialised but must match the total number of active parameters in that specific layer as the winningticket.
We train these networks on the different CIFAR10 subset sizes using the identified best augmentationstrategy and compare their performance in Fig.
The performance of the networks with lowercapacities (small dense, random pruned, same-layer-sparse and winning ticket) are all improvedcompared to the base dense network particularly in the lowest data size setting.
However, the winningticket considerably outperforms the others indicating that neither network capacity nor sparsity aloneis sufficient but together their combination in the winning ticket leads to improved performance ondata-limited regimes.
5.2 Which layers are getting pruned?
Fig.
7 presents the layer-wise density plots of the sparsest winning ticket at various training data sizes.
Here density is defined as the ratio of the number of active parameters to total number of parametersin that specific layer.
We use the sparsest model identified (i.e 16 th pruning iteration) using thebest augmentation strategy at that particular data size.
We can see a striking difference between thesparse models identified at higher dataset sizes when compared to the lower ones.
Particularly, the initial layers are more dense as the training data size reduces.
Retaining more weights in the initiallayers appears to allow the sparse models to keep the filters for detecting primitive features suchas edges, and corners perhaps helping it generalize better.
Another interesting thing to note is that,the identified sparse networks at lower data sizes have three peaks which correspond to the residualconnection layers of each block.
In the ResNet architecture, the residual connections enable strongergradients and hence the maximum magnitude weights are expected to reside on the residual branch,thereby increasing its density. To probe the presence of dense residual connections further, we plot the ratio of norms of the outputfrom the residual branch and the main branch. A higher value would indicate stronger propagationof information from the residual branch while a lower value would indicate the vice-versa. Fig.
6visualizes these ratios for the winning ticket and dense model with and without augmentations whentrained on the 1% data subset. The ratio of norms are lower for the winning ticket suggesting perhapsthat the model learns more generalizable representations.
5.3 Generalizability of the learned representations.
[Figure]
To empirically verify that the representations of the sparse winning ticket are more transferable wetake 4 models: the dense network train on 100% of the data, along with the dense, dense+aug., and
WT+aug. networks trained on 1% of CIFAR10.
In Table.
4 we present the results of freezing thebackbone and fine-tuning only the final linear classifier on the CIFAR100 dataset.
Surprisingly thewinning ticket not only outperforms the dense (1%), and dense+aug.
network, but it also outperformsthe dense model trained on 100% of CIFAR10.
This indicates that the sparse sub-networks identifiedat the least data size learns generalizable representations of the input image.
5.4 Layer-wise representation similarity
[Figure]
We use Central Kernel Alignment (CKA) [38] to study the similarity of internal representationstructure across different models in Fig. 8.
We visualize the intra-layer wise similarity across threemodels: dense network trained on 100% data, along with dense+Aug ∗ , and WT+Aug ∗ networkstrained on 1% CIFAR10.
We see that the sparse winning ticket exhibits a larger span of similar initiallayers unlike the others which exhibit more intra-block similarity.
This could possibly imply thatthe network is extracting globally generalizable features right from the initial layers.
In fact, the
WT+Aug ∗ network showcases higher similarity to the network trained on 100% data when comparedto the dense+Aug ∗ network.
The above observations go in hand to further support our empiricalanalysis regarding the generalizability of the learned representations.
Conclusion
In this work, we hypothesized that winning tickets identified via magnitude pruning would be moreeffective in low-data regimes than their dense counter parts.
Based on extensive experiments onsub-sampled versions of the CIFAR10 dataset, we find that with decreased training data, the winningticket gets sparser and when combined with augmentations considerably outperforms the originaldense network.
We also verify that the sparsity of the winning tickets helps it avoid memorizationand prevents over-fitting, by evaluating performance on distributionally shifted datasets.
We showthat IMP continues to show improvements in performance when combined with other data-efficienttraining strategies for low data regimes.
We further evaluate the generalizability of the approach ondiverse datasets and simulated imabalanced datasets which have 50-100 examples per class.
Finally,we analyze why sparse winning tickets show superior performance on low-data regimes.
Our findingsindicate that their performance can be attributed to combined properties of lower network capacityand connectivity, and help the network learn more generalizable representations.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
[Yes] Please findcode in Supplemental Material, Relevant datasets are cited in Sec. 3.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
e ) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Augmentation is important to find better Winning Tickets
We can see from 3a that at 1% data size, the performance gap between WT+aug., Dense+aug. issignificantly higher than their corresponding non-augmented versions.
To verify the importance ofaugmentation during lottery ticket finding, we select the WT identified via Base-augment strategy,reinitialize the network to θ r (refer Sec. 3.1) and train using the best augmentation strategy at eachdata size.
From Table.
5, we can see that the winning ticket identified via Base-augment cannotmatch the performance of the WT+aug* even when trained later with the same best augmentationstrategy (indicated by WT(Base)+Aug* ).
Therefore, augmentation plays a significant role duringticket finding.
[Table] [Table]
B Adversarial Robustness of Winning Tickets
To evaluate the network’s performance on adversarial attacks, we perform the FGSM [39] one stepattack with an epsilon value of 8 / 255 .
Similar to Sec.
4.2, Fig.
9 presents adversarial robustnessof models trained on CIFAR10 subsets across different augmentation strategies and sparsity levels.
Unlike the other cases (i.e synthetic and domain shifts), the sparse networks perform worse thantheir corresponding dense counterparts across all data sizes.
We hypothesize that since the IMPprocedure only retains the maximum magnitude weights, they would directly affect the gradients,hence increasing the effects of the adversarial attack.
To verify the same, we compute the averageratio of norms of the remaining weights between WT+aug. and Dense+aug across layers.
Table.
6clearly indicates that the winning tickets though sparse, have a higher magnitude of weights across alldata sizes (ratio > 1).
At the least data sizes (2%, 1%), the absolute value of the remaining weights isroughly 1.3 times its dense counterpart thereby increasing the magnitude of the adversarial attack fora given epsilon value.
C Long-tailed CIFAR100 Data Distribution
We visualize the long-tailed data distribution of the sampled CIFAR100 dataset for varying λ valuesin Fig.
The number of samples per class varies from 5-500 samples/class when λ = 0.01 upto50-500 samples/class when λ = 0.1.
D Generalization to complex datasets
Besides the experiments in Sec. 4.4, we train a ResNet-18 model on 2% (or 10 samples/class) and 1%(or 5 samples/class) of CIFAR100 to evaluate our proposal with more classes.
Additionally, we have
[Figure]
also included results after 10 pruning iterations on a subset of ImageNet [40] (5% or 50 samples/class).
Table.
7 summarize these results and show that sparse winning tickets always outperforms the densenetwork, and justifies our primary claim that sparsity improves data-efficient performance.