Sparse Winning Tickets are Data-Efficient Image Recognizers
Anonymous Author(s)
AffiliationAddress email
Abstract
Improving performance of deep networks in data limited regimes has warrantedmuch attention.
In this work, we empirically show that “winning tickets” (small subnetworks) obtained via magnitude pruning based on the lottery ticket hypothesis [1],apart from being sparse are also effective recognizers in data limited regimes.
Based on extensive experiments, we find that in low data regimes (datasets of50-100 examples per class), sparse winning tickets substantially outperform theoriginal dense networks.
This approach, when combined with augmentations orfine-tuning from a self-supervised backbone network, shows further improvementsin performance by as much as 16% (absolute) on low sample datasets and longtailed classification.
Further, sparse winning tickets are more robust to syntheticnoise and distribution shifts compared to their dense counterparts.
Our analysis ofwinning tickets on small datasets indicates that, though sparse, the networks retaindensity in the initial layers and their representations are more generalizable.
Codewill be made available after acceptance.
Introduction
Deep convolutional networks [2] have achieved wide success on a variety of tasks, but they dodemand large amounts of data.
However, there exists various domains and tasks where training datalabeled or unlabeled - is limited.
Often, in such data limited regimes, transfer learning has emerged asa dominant approach [3] as it achieves superior performance compared to training from scratch withrandom initialization [4].
In some domains that are highly specialized training performant modelscan still be a challenge, e g. in scientific or medical images [5] or just images from a differentdistribution - differing vastly in size, color, or channels - than seen in typical image datasets.
Insuch cases, training large-capacity data-hungry networks might be less performant than trainingmodels of similar architecture and lower capacity [6].
Much of the literature on sparser and lowercapacity networks has focused more on efficient models and mobile deployment [7].
But it is alsoknown [8] that sparsity as a regularization could reduce overfitting.
Thus it is interesting to studysparse networks in the context of low data or data efficient regimes (Fig. 1).
One of the popular ways of obtaining a sparse network of similar architecture but lower capacity is viapruning [9].
1 formulated the lottery ticket hypothesis (LTH) and proposed iterative magnitude basedpruning (IMP).
They demonstrated that dense networks can be magnitude pruned to obtain highlysparse sub-networks (“winning tickets”) which can match or potentially outperform the original densenetwork.
The caveat being that the winning ticket needs to use the same random initialization asthe original dense network.
This approach also induces an inductive bias specific to the task to belearned, which leads to a better network when compared to a dense network of similar size trainedfrom scratch [10].
Further, such pruning was recently found to reduce sample complexity in theoryand practice.
11 also suggest that the number of samples required to achieve zero generalization error proportional to the number of the non-pruned weights in the hidden layer.
Thus, despite indicationsof sparse networks being suitable for low data regimes, there is limited work [12] studying them in the context of language, and no prior work which evaluates their effectiveness for image recognitiontasks in the low data setting.
In this work , we provide empirical evidenceof the suitability of sparse networks identifiedusing IMP for data-limited regimes.
We conductextensive experiments to compare the relationship between network capacity, datasize, andimage classification accuracy.
We first showthat sparse winning tickets have substantiallybetter performance on low (50-100 samples perclass) data regimes compared to their originaldense counterparts.
Next, we show that sparsewinning tickets for data-efficient learning compliments and can be combined with existingmethods such as augmentations or fine-tuningfrom a self-supervised backbone to further boostperformance by as much as 16%.
We then evaluate the generalization capacity of the networksthrough detailed robustness tests on syntheticand domain-shifted datasets.
We also evaluateon special data-deficient cases like long-tailedclassification to compare performance whensome classes have far few samples than others.
Finally, we try to understand properties of sparsenetworks that lead to its superior performance.
We find that both capacity and connectivity together are important for generalization.
Our keycontributions can be summarized as follows:
We find that the sparsity of the winning ticket increases with reduction in training data.2.
Demonstrate that winning tickets can easily combine with existing data-efficient trainingstrategies such as augmentations to outperform similarly trained dense networks on low dataregimes by as much as 16%.3.
Show that winning tickets, on low data regimes, are robust to synthetic noise and domain shifts,and show improved performance on long-tailed classification.4.
Our analysis indicates that on data-limited regimes sparse winning tickets continue to pre-
serve density in initial layers, and both capacity and connectivity jointly are important forgeneralization.
Related Works
Sample-Efficient Image Recognition.
Learning from few samples and less data has been a longstanding topic of interest in machine learning.
While transfer learning, particularly from ImageNetpretrained models [3] is the most popular approach, semi-supervised techniques [13] and selfsupervision [14] have been successful when there is an abundance of unlabeled data in the samedomain.
All of these typically involve pretraining the model on data from a similar domain, or thesame domain, often with a variety of augmentations and then tuning it on a few labeled examples.
These are complimentary to our approach, and we explore combining augmentations and transferringin our experiments.
Few-shot methods are also related and they usually look at tuning just the finalclassification layer [5, 15].
A different approach is meta-learning [16], which aims to identify anetwork from the support classes, seen during pre-training, and which can easily adapt to the targetset of classes in few steps.
[17] mentions random pruning for few-shot learning but leaves it largelyfor future investigation.
Our work is different from typical few-shot methods in two ways: 1 ⃝ We donot limit ourselves to transfer learning and also focus on training from scratch directly from the smalldataset; 2 ⃝ We cover data sets with approximately 50-100 samples per class which lies between mostfew shot settings and typical deep learning settings with more data.
There are also works that directly use small datasets without external dependence on pre-training,often relying on training strategies to prevent overfitting.
18 proposes to use the cosine loss while19 uses the t-Distributed von Mises-Fisher (t-vMF) loss along with the standard cross entropy fortraining.
These regularising loss functions reduce intra-class variance.
Some other methods rely on extracting useful geometric priors to help avoid over-fitting: 20 use preset cosine filters and 21introduces full convolutions which exploits the absolute spatial location of objects in the imageand improves translation invariance to strengthen the visual inductive prior.
In this work, we coverdatasets containing roughly 50-100 samples per class and the above methods have shown strongperformance in this regime [22] and offer easy implementation.
Therefore, in this work we compareagainst these methods and study if they supplement pruning techniques.
Pruning.
Among pruning based methods, the iterative magnitude pruning (IMP) (described in
Sec. 3.1) proposed by [1] is most relevant to our work.
IMP is a process that helps identify sparse subnetworks (“winning tickets”) within a larger dense network that is capable of matching or potentiallyout-performing the dense network.
This has proved successful on a variety of tasks including imageclassification [23], natural language processing [24, 25], and generative modelling [26].
While IMPhas been popular for obtaining sparse networks, we are not aware of works that explore its relationshipto data-efficiency.
In this paper, we carry out detailed experiments to validate if sparsity particularly“winning tickets” obtained via IMP improves data-efficiency of deep neural networks.
Methodology and Experiments
Our approach primarily uses iterative magnitude pruning (IMP) and we study the performance of the“winning tickets” in low data regimes.
In this section, we describe IMP, and provide an overview ofour experimental studies including the basic experimental setup, augmentation strategies we compareagainst, and the datasets used for different studies.
3.1 Preliminary: Iterative Magnitude Pruning (IMP)
1 introduces IMP as a way to identify sparse sub-networks within a dense network that match orexceed the performance of the full dense network.
They view these sub-networks as having wonthe initialization “lottery” and refer to them as “winning tickets”.
We use the IMP procedure withrewinding [27, 28] and describe it briefly below.
Let f ( x , θ ) denote the neural network, where θ denotes the parameters of the network and x denoteits input.
Then the sub-network can be characterized by a binary mask m and can be applied to theoriginal network via a Hadamard product ( ⊙ ) to obtain the sub-network f ( x , θ ⊙ m ) .
For a networkinitialized with θ 0 , the IMP process aims to find a final mask m of a specified sparsity ratio ( s ) asfollows:
initialize m to be all ones mask.
train f
( x , θ 0 ⊙ m ) for r steps to get θ r .
This will be used as the initialization for the laterpruning iterations.
3. continue to train f ( x , θ r ⊙ m ) to convergence.
4. remove a portion of the weights (determined by the pruning rate p ) with the smallest magnitudes
from θ ⊙ m .
Update m by setting the removed weights to 0.
repeat steps 3, 4 until the specified sparsity ratio ( s ) of pruned vs. original weights is achieved.
Note that when r = 0 , the above algorithm reduces to IMP without rewinding.
Rewinding is foundto be essential for successful and stable identification of winning tickets in large networks [27, 28].
Since this is an iterative process, we obtain sub-networks of different sparsity at each stage.
In all ourexperiments, the sub-network that has the best performance, across all iterations, on the validation setis termed as the “sparse winning ticket”.
3.2 Overview of Experimental Studies
We provide a broad overview of our experiments before presenting the details and datasets used.
(1) We first study how network capacity of models pruned via IMP affects classification performance
when training on datasets of different sizes.
Since augmentations are often employed during training
of models in low-data regimes, we also perform this study with models trained using different augmentation strategies.
We compare performance of dense models with just augmentations against those with both augmentation and pruning.
(2) Next to study if sparsity prevents over-fitting by avoiding memorization, we evaluate performance of the pruned models on a (a) synthetically transformed dataset , and on a (b) domain-shifted dataset.(3)
Aside from augmentations, we also compare IMP against other training techniques employedin low-data regimes as discussed in Sec. 2, as well as fine-tuning from ImageNet and SimCLR backbones.
All our main experiments are on ResNet-18, and we also show these results hold on
ResNet-50, non-residual networks like VGG, and parameter-efficient MobileNet-v2.(4) To verify the generalizability of the approach and measure performance of sparse winning ticketsin data-limited regimes, we perform studies on (a) 3 real datasets with few (50-80) images per class,and also (b) simulated datasets with imbalanced (long-tail of) classes. We also present results on 1%and 2% data subsets of ImageNet and CIFAR100 in Appendix. D.(5)
Finally, we try to understand why sparse winning tickets are able to perform well in datalimited regimes.
We do this by analyzing network capacity and connectivity, and also how pruningaffects the layers - which layers are getting pruned, the layer-wise representation similarity, and thegeneralizability of the representations.
3.3 Experimental Setup
We use the ResNet model (specifically ResNet-18 unless otherwise specified) in all our experiments.
Since we are working with smaller-sized images (often 32x32 or 64x64), we modify the initialconvolution layer to a 3x3 kernel, with padding 1 with no max-pooling.
For images of size 224 wedo not make any modifications.
We follow the IMP lottery ticket finding procedure with 16 pruningiterations where 20% of the weights are pruned after each iteration.
In each pruning iteration, themodel is trained for 200 epochs to minimise the cross entropy loss using the SGD optimizer withweight decay 0.0005.
The initial learning rate is set to 0.1 and then cosine decayed for 200 epochs.
For fine-tuning experiments we use a lower learning rate of 0.001.
During the first iteration, we usethe rewinding technique and set r = 2 epochs.
3.4 Augmentation Strategies
Augmentations are one of the default strategies applied when working with limited data.
In thiswork, we look at augmentations in two scenarios: (1) as a comparison to the performance of thewinning ticket i.e. comparing the dense model with augmentations to that of a pruned model withoutaugmentations, and (2) we combine augmentations with IMP to see the benefit of augmentation inaddition to pruning.
We carry out experiments with four different augmentation strategies each ofdifferent strength.
The strategy Basic consists of just RandomCrop and RandomFlip.
All the otherstrategies are composed on top of the Basic augmentation.
Contrast strategy additionally consists of
ColorJitter and RandomGray [14].
Auto uses Basic augmentations and also the best policies identifiedfor each dataset by AutoAugment [29].
Rand is based on RandAugment [30], and composes fourrandomly selected augmentations in addition to the Basic augmentations.
We can potentially orderthese 4 augmentation strategies by increasing strength as: Basic < Contrast ≈ Auto < Rand.
3.5 Datasets
CIFAR10 [31] is our primary dataset for all analysis experiments.
It consists of 60,000 colour images(train: 50,000, val: 10,000) of size 32x32 split into 10 classes.
For experimental comparisons to thedata-limited regime, we sub-sample it at various sizes while maintaining the original class-size ratios.
E.g., at 1% data size, the training set contains 50 images per class.
We aways evaluate on the fullvalidation set in all our experiments.
For the robustness and extended experiments, we use additionaldatasets described in the corresponding sections.
CIFAR10-C [32].
We use the CIFAR10-C (C for corrupted) dataset to evaluate the robustness ofthe models to (unseen) synthetic transforms .
It contains 19 different transforms each of 5 differentstrength levels.
These transforms can be broadly classified into - Noise (e.g. impulse noise), Blur(e.g.
motion blur),
Weather (e.g. snow), Digital (e.g. JPEG), and Extra (e.g. brightness).
CIFAR10.2 [33].
To evaluate the network’s performance on domain shifts , we use the CIFAR10.2dataset derived from TinyImages [34].
It contains the exact same classes as the original CIFAR10dataset but has both different and harder collection of images ideal for evaluating distributional shifts.
CLaMM [35], ISIC [36] and EuroSAT [37].
Aside from experiments on subsampled versionsof CIFAR, we use three diverse low-data regime datasets covering different domains with varyingnumber of classes and images per class.
The EuroSAT dataset [37] contains 64 × 64 sized satelliteimages from 10 classes with 50 images per class captured from Sentinel-2 satellite.
It consists ofmultiple channels.
We follow the procedure given in [37] to use the RGB channels only.
ISIC2018 dataset [36] contains 224 × 224 sized dermoscopic skin lesion images from 7 classes with 80 samples per class.
The CLaMM dataset [35] contains 224 × 224 sized gray-scale latin scripts fromhand-written books from 12 classes with 50 images per class.
CIFAR100 [31] For studies on the performance of the sparse networks on imabalanced and longtailed classification tasks we derive a dataset based on CIFAR100.
We selectively sample the
CIFAR100 dataset such that the class-wise image count follows a long-tailed distribution (defined byan exponential curve given by: Nn × λ j/ ( n − 1) where N is total number of samples in the dataset, j isthe class index and n (=100) is the number of classes in the dataset.
The degree of “long-tailedness”is defined by the imbalance factor ( λ ), i.e a lower value indicates more number of classes have alower sample count.
The derived long-tailed class-wise distribution of CIFAR100 is visualized in the
Appendix C.
Results
[Figure]
We first verify our main claim and show the suitability of sparse winning tickets for data-limitedregimes.
Fig.
3a (and Fig. 1) presents the results of training a ResNet-18 network on varying CIFAR10subsets of sizes 100% , 50% , 20% , 10% , 2% , and 1% .
At the lower data sizes the sparse winningticket (denoted as WT ) shows considerably improved performance compared to the dense model.
We also investigate how pruning compares with augmentation strategies and if both of these canbe combined.
Fig.
3a shows the results of the best performing dense and sparse networks withaugmentation.
The best is chosen across all augmentations for each data size and are indicated by
Dense+Aug ∗ (or D+Aug ∗ ) for the dense model and WT+Aug ∗ for the winning ticket.
We can seethat across all data sizes, models pruned with augmentation outperforms unpruned ones and as thedata size decreases the gap between the two widens indicating that sparse networks are suitable fordata-limited regimes.
Fig.
2 presents detailed comparisons of the performance of the model on the different data subsets(each subplot) at different sparsity levels (subplot x-axis) for the different augmentation strategies(lines of different styles).
In almost all cases, the best performing model is sparse i.e density < 100% .
While in the higher data regimes (top row), there is a decrease in performance with increased sparsity,the trend is reversed in the low data regimes (bottom row).
More importantly, at the lower datasizes i.e 2% , 1% , it’s clear that the sparse winning tickets (indicated by stars in Fig. 2) significantlyoutperform the dense (100% density) models with augmentations.
Further, among the augmentationstrategies, Auto-augment seems to consistently yield the best performance, except in the 1% data sizewhere Rand-augment does much better.
We further discuss the relevance of applying augmentationsduring the pruning stage in Appendix.
[-A.4.
-]A.
4.2 Winning tickets are robust to distributional shifts.
232 As the data size gets smaller, the chance of over-fitting and memorization of the training samplesincreases.
Thus, to study robustness, we evaluate the performance of the identified sparse winning [Figure] tickets on two types of distributional shifts : (a) synthetic transforms via the CIFAR10-C dataset [32]and (b) domain-shifts via the CIFAR10.2 dataset [33].
Note that we only perform evaluation (i.e.inference) on these datasets to study the robustness of the winning tickets to these transformations.
We evaluate the network’s performance on:(a) synthetic transforms on the CIFAR10-C dataset.
Row-1 in Fig. 4 and Fig.
3b visualizes theaverage accuracy of the models across all 19 unseen transforms.(b) domain shifted CIFAR10.2 dataset consisting of wider variety of harder images from the
CIFAR10 classes.
Row-2 in Fig.
4 and Fig.
3b presents the results.
In both cases, similar to performance on the CIFAR10 data (Fig. 2) on 10% and lower training datasizes, sparser winning tickets generalize better to these unseen transforms and distribution shifteddata, much more so with augmentations.
The difference in classification accuracy between the sparsewinning ticket and dense model is also more clearly noted from Figures.
3b and 3c where we look atthe just the best performing models (as determined on CIFAR10 val.
set) - dense model and winningtickets with and without augmentations.
These results clearly indicate that sparse winning tickets,when trained as typical with augmentations, are capable of avoiding memorization of training samplesat the smaller data sizes, and much more so than the dense models.
This reinforces its suitability fordata-limited regimes.
4.3 IMP complements existing data-efficient training.
Table.
1 summarizes results using IMP to supplement and complement existing data-efficient methods.
In particular, we apply IMP in combination with methods that have been shown to work well on smalldatasets: fine-tuning from ImageNet and SimCLR [14] backbones, self-regularizing cosine loss [18],use of full convolutions [21] (Full Conv.), as well as 2 emergent methods, t-VMF loss [19] andharmonic nets [20].
Networks are trained using the best identified augmentation strategy at each data [Table] [Table] size i.e Auto-augment at 2% and Rand-augment at 1% data-sizes respectively.
With all techniques,
IMP further improves performance on the CIFAR10 val.
set on an average by 8% and 15% at 2%and 1% data sizes respectively.
IMP shows more significant gains with non fine-tuning methods, as(absolute) magnitude based pruning strategy does not work well with pre-trained initializations [38],though we do see substantial improvements even on fine-tuning. As seen in Table 1, our results on
ResNet-18 extend beyond residual networks to VGG, and MobileNet-v2 based architectures.
4.4 Generalization to low sample datasets.
To evaluate generalization to datasets other than subsets of CIFAR10, we experiment on three diversedatasets.
The EuroSAT [37], ISIC [36], and CLaMM [35] datasets cover different domains - satelliteimagery, medical images, and books respectively with 7-12 classes but few images (50-80) perclass.
Table.
2 compares performance of the dense model and IMP, using the Basic augmentationstrategy and in combination with existing data efficient techniques for low sample data.
The winningtickets outperform the dense model on an average by 12%, 2.3%, and 5.5% on the CLaMM, ISICand EuroSAT datasets respectively.
Appendix. D discusses results on 1% and 2% data subsets of CIFAR100 and ImageNet, and our results indicate the effectiveness of winning tickets to handlemuch larger number of unique classes even in the absence of sufficient data.
4.5 Long-tailed classification
Long-tailed classification is a specific data-limited setting where some classes of the dataset containvery limited samples compared to the rest.
The presence of the dominant class can lead to over-fitting.
Therefore, we evaluate the effectiveness of sparse networks in such a dataset derived from CIFAR100(see.
Sec. 3.5).
Table.
3 summarizes these results and we can see that the winning ticket has slightlybetter performance than the original dense network.
An important note is that the total dataset sizeafter imbalance is between 40% and 20% of the actual size (100-200 examples per class) therefore,the performance improvements are smaller.
Nevertheless, the gains indicate that sparse networks areable to avoid over-fitting due to class-imbalance and not just uniformly reduced data size.
[Table] [Table]
Discussion
Here we empirically investigate the factors that contribute to the winning ticket network’s performancegains in data-limited regimes. Network capacity and connectivity are important.
We first investigate if it is the winning network’s capacity or the connectivity between neurons (thatare retained after pruning) that contribute to the performance.
During pruning even as the network capacity is reduced, the connectivity between layers is also changed.
We study the original densenetwork, and the sparse winning ticket with 3 additional networks: (1) A small dense network whichis a dense network but has overall parameters reduced to match the winning ticket.
(2) A randomlypruned network where we take the original dense model, but randomly set some of the weights tozero to match the sparsity of the winning ticket.
(3) A same-layer-sparse network which followsthe exact same layer-wise densities as the winning ticket but the connectivity between the layers arerandom i.e. the binary mask at each layer is randomly initialised but must match the total number ofactive parameters in that specific layer as the winning ticket.
We train these networks on the different CIFAR10 subset sizes using the identified best augmentationstrategy and compare their performance in Fig.
The performance of the networks with lowercapacities (small dense, random pruned, same-layer-sparse and winning ticket) are all improvedcompared to the base dense network particularly in the lowest data size setting, indicating that asmaller network size improves data-efficient performance.
However, the winning tickets significantlyoutperform a network of similar capacity i.e. the "Small Dense", indicating that beyond capacityperhaps the network connections also play an important role. To probe the importance of connectivity,we compare the winning ticket (WT+Aug*) against randomly pruned (Random Prune + Aug*) andthe same-layer-sparse network and observe that it outperforms both, emphasizing the importance ofconnectivity. These lead us to conclude that both network capacity and connectivity play a vital rolein improving the data efficiency of sparse networks.
5.2 Which layers are getting pruned?
Fig.
7 presents the layer-wise density plots of the sparsest winning ticket at various training data sizes.
Here density is defined as the ratio of the number of active parameters to total number of parametersin that specific layer.
We use the sparsest model identified (i.e 16 th pruning iteration) using thebest augmentation strategy at that particular data size.
We can see a striking difference betweenthe sparse models identified at higher dataset sizes when compared to the lower ones.
Particularly,the initial layers are more dense as the training data size reduces.
Retaining more weights in theinitial layers appears to allow the sparse models to keep the filters for detecting primitive featuressuch as edges, and corners perhaps helping it generalize better.
Another interesting thing to noteis that, the identified sparse networks at lower data sizes have three peaks which correspond to the residual connection layers of each block.
As discussed in Sec.
4.2, the winning tickets exhibit higherrobustness to several corruptions especially in the least data sizes (2%, 1%). We hypothesize that theretention of denser residual layers enables minimal change to the output even upon input change (oridentity connections), hence enabling increased robustness [39].
5.3 Generalizability of the learned representations.
To empirically verify that the representations of the sparse winning ticket are more transferable wetake 4 models: the dense network train on 100% of the data, along with the dense, dense+aug., and
WT+aug. networks trained on 1% of CIFAR10.
In Table.
4 we present the results of freezing thebackbone and fine-tuning only the final linear classifier on the CIFAR100 dataset.
Surprisingly thewinning ticket not only outperforms the dense (1%), and dense+aug.
network, but it also outperformsthe dense model trained on 100% of CIFAR10.
This indicates that the sparse sub-networks identifiedat the least data size learns generalizable representations of the input image.
5.4 Layer-wise representation similarity
[Figure]
We use Centered Kernel Alignment (CKA) [40] to study the similarity of internal representation structure across different models in Fig. 8.
CKA computes the normalized Hilbert-Schmidt independencecriterion between two representations either within or across networks, averaged across sampleswithin a mini-batch.
It is better suited to measure similarity between high dimensional representations where the dimensions far exceed the number of data points.
We visualize the intra-layer wisesimilarity across three models: dense network trained on 100% data, along with dense+Aug ∗ , and WT+Aug ∗ networks trained on 1% CIFAR10.
We see that the sparse winning ticket exhibits a largerspan of similar initial layers unlike the others which exhibit more intra-block similarity.
It is possiblethat the increased layer-wise similarity is directly related to the denser residual connections observedin Sec. 5.2.
To probe this, we plot the ratio of norms of the output from the residual branch andthe main branch in Fig. A higher value would indicate stronger propagation of information fromthe residual branch while a lower value would indicate the vice-versa. Surprisingly, these ratios arelower in the sparse network when compared to the dense network, which potentially suggests that thesparse network is in fact learning globally generalizable features. It is expected that a network trainedon the complete 100% data is sufficiently more generalizable than one trained on fewer number ofsamples.
Therefore, we compare the WT, Dense (+ Aug*) networks trained only on 1% data to anetwork trained on the complete 100% data, and we notice higher layer-wise similarity in the case ofthe sparse network.
These observations go in hand to further support our empirical analysis regardingthe generalizability of the learned representations.
Conclusion
In this work, we hypothesized that winning tickets identified via magnitude pruning would be moreeffective in low-data regimes than their dense counter parts.
Based on extensive experiments onsub-sampled versions of the CIFAR10 dataset, we find that with decreased training data, the winningticket gets sparser and when combined with augmentations considerably outperforms the originaldense network.
We also verify that the sparsity of the winning tickets helps it avoid memorizationand prevents over-fitting, by evaluating performance on distributionally shifted datasets.
We showthat IMP continues to show improvements in performance when combined with other data-efficienttraining strategies for low data regimes.
We further evaluate the generalizability of the approach ondiverse datasets and simulated imabalanced datasets which have 50-100 examples per class.
Finally,we analyze why sparse winning tickets show superior performance on low-data regimes.
Our findingsindicate that their performance can be attributed to combined properties of lower network capacityand connectivity, and help the network learn more generalizable representations.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[N/A](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Please findcode in Supplemental Material, Relevant datasets are cited in Sec.
3.5(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Augmentation is important to find better Winning Tickets
We can see from 3a that at 1% data size, the performance gap between WT+aug., Dense+aug. issignificantly higher than their corresponding non-augmented versions.
To verify the importance ofaugmentation during lottery ticket finding, we select the WT identified via Base-augment strategy,reinitialize the network to θ r (refer Sec. 3.1) and train using the best augmentation strategy at eachdata size.
From Table.
5, we can see that the winning ticket identified via Base-augment cannotmatch the performance of the WT+aug* even when trained later with the same best augmentationstrategy (indicated by WT(Base)+Aug* ).
Therefore, augmentation plays a significant role duringticket finding.
[Table] [Table]
B Adversarial Robustness of Winning Tickets
To evaluate the network’s performance on adversarial attacks, we perform the FGSM [41] one stepattack with an epsilon value of 8 / 255 .
Similar to Sec.
4.2, Fig.
9 presents adversarial robustnessof models trained on CIFAR10 subsets across different augmentation strategies and sparsity levels.
Unlike the other cases (i.e synthetic and domain shifts), the sparse networks perform worse thantheir corresponding dense counterparts across all data sizes.
We hypothesize that since the IMPprocedure only retains the maximum magnitude weights, they would directly affect the gradients,hence increasing the effects of the adversarial attack.
To verify the same, we compute the averageratio of norms of the remaining weights between WT+aug. and Dense+aug across layers.
Table.
6clearly indicates that the winning tickets though sparse, have a higher magnitude of weights across alldata sizes (ratio > 1).
At the least data sizes (2%, 1%), the absolute value of the remaining weights isroughly 1.3 times its dense counterpart thereby increasing the magnitude of the adversarial attack fora given epsilon value.
C Long-tailed CIFAR100 Data Distribution
We visualize the long-tailed data distribution of the sampled CIFAR100 dataset for varying λ valuesin Fig.
The number of samples per class varies from 5-500 samples/class when λ = 0.01 upto50-500 samples/class when λ = 0.1.
D Generalization to complex datasets
Besides the experiments in Sec. 4.4, we train a ResNet-18 model on 2% (or 10 samples/class) and 1%(or 5 samples/class) of CIFAR100 to evaluate our proposal with more classes.
Additionally, we have
[Figure]
also included results after 10 pruning iterations on a subset of ImageNet [42] (5% or 50 samples/class).
Table.
7 summarize these results and show that sparse winning tickets always outperforms the densenetwork, and justifies our primary claim that sparsity improves data-efficient performance.