F INDING W INNING T ICKETS WITH L IMITED ( OR N O ) S UPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
The lottery ticket hypothesis argues that neural networks contain sparse subnetworks, which, if appropriately initialized (the winning tickets ), are capable of matching the accuracy of the full network when trained in isolation.
Empirically made in different contexts, such an observation opens interesting questions about the dynamics of neural network optimization and the importance of their initializations.
However, the properties of winning tickets are not well understood, especially the importance of supervision in the generating process.
In this paper, we aim to answer the following open questions: can we ﬁnd winning tickets with few data samples or few labels?
can we even obtain “good” tickets without supervision?
We ﬁnd that winning tickets found in these scenarios are, perhaps surprisingly, competitive with winning tickets generated on the full ImageNet dataset when evaluated on ImageNet classiﬁcation task.
I NTRODUCTION
Recently, Frankle & Carbin (2019) have observed that sparse subnetworks of over-parameterized neural networks could achieve good predictions when trained in isolation, as long as they are appropriately initialized; these “lucky” starting points have been termed winning tickets .
Building these tickets is typically achieved by pruning the weights with lowest magnitude of an over-parametrized network that has been trained to convergence, before resetting the remaining weights to their initial values, or at some point early in training, and repeating the procedure.
The properties of these effective subnetworks and their initialization are however not well understood yet.
For example, the original lottery ticket hypothesis argued that these winning tickets emerge from the network initialization.
Yet, original experiments were performed on classiﬁcation tasks with shallow architectures and small datasets, and further experiments on deeper models and more challenging problems suggest that they may rather appear later, though still early, during training (Frankle et al., 2019).
Recent works (Morcos et al., 2019; Yu et al., 2019; Zhou et al., 2019) have studied other properties of the winning tickets generation process: Morcos et al.
(2019) have shown that winning tickets can be transferred across datasets with similar natural image statistics, and Yu et al.
(2019) have exposed the existence of lottery tickets to other domains, such as text and reinforcement learning.
In this paper, we expand on this line of work and empirically answer other important open questions on winning tickets and the data-labels distribution on which they are generated.
In particular, we want to know if “good” tickets can be obtained when few data samples, or few labels, or even no labels, are available.
Answering these questions is important since it may speed-up the winning ticket generation process, which is computationnally expensive, and also open new perspectives about generating sparse subnetworks that may be trained efﬁciently on new tasks.
To this effect, we design several experiments that isolate the impact of the data and label distributions on the quality of the resulting winning tickets.
First, we analyze the extent to which winning tickets are label-dependent by generating “label-agnostic” winning tickets with self-supervised tasks (Gidaris et al., 2018; Doersch & Zisserman, 2017).
Then, we evaluate the impact of reducing the numbers of samples per class or the number of classes used for winning ticket generation.
Finally, we evaluate the performance of semi-supervised approaches to winning ticket generation.
Most of our experiments are conducted on ImageNet; we remark indeed that deep networks trained on smaller datasets such as CIFAR-10 are already sparse at convergence, making pruning less challenging, and conclusions drawn about lottery tickets potentially misleading if this effect is not accounted for.
Overall, our experiments show that winning tickets are surprisingly robust to many of these data/label distribution changes.
Indeed, using only 10% of the dataset or removing entirely the labels still leads to effective winning tickets.
As a result, we manage to extract winning tickets 5 × faster by training on a subset of data, with no modiﬁcations to the core algorithm.
This is of particular interest because ﬁnding winning tickets is highly data, labels and resource-demanding since it requires training a network to convergence at least once (and generally dozens of times in practice).
Our paper makes the following contributions: (i)
We combine self-supervised learning with winning tickets generation, showing that good winning tickets can be found without labels.
(ii) We show that ﬁnding winning tickets can be accelerated by a factor 5 × on ImageNet by using only a subset of the data.
(iii)
We also show that using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading.
R ELATED WORK
Pruning.
Pruning is an approach to model compression (Han et al., 2015) and regularization (LeCun et al., 1990) in which weights or nodes/ﬁlters are removed, typically by clamping them to zero.
It is an active research which has primarily focused on pruning an already trained network (Han et al., 2015; Li et al., 2017) or pruning while training (Prakash et al., 2019).
In particular, the iterative pruning during training of Han et al.
(2015) has been extended to continuous pruning (Guo et al., 2016), layer-wise pruning (Dong et al., 2017) and with weight sharing (Ullrich et al., 2017).
Pruning during training has been considered with (cid:96) 0 regularization (Louizos et al., 2018), binary convolution (Rastegari et al., 2016) or using the hashing trick for weight sharing (Chen et al., 2015).
There are several ways to prune a network; among them are unstructured pruning (LeCun et al., 1990; Han et al., 2015) which corresponds to discarding individual connections or structured pruning that operates by removing entire channels or layers (see Liu et al.
for a review of the different methods)
The lottery ticket hypothesis focuses on unstructured pruning early in training, which has the advantage of signiﬁcantly reducing the training cost of the resulting model.
The lottery tickets hypothesis.
The lottery tickets hypothesis (Frankle & Carbin, 2019)
explores the possibility of pruning early in training by revealing that some sparse subnetworks inside neural networks can reach accuracy matching that of the full network when trained in isolation.
Setting the weights of the sparse architecture appropriately is critical to reach good performance.
Frankle & Carbin (2019) provide a proof of concept of the lottery ticket hypothesis on small vision benchmarks while Frankle et al.
(2019) conduct further experiments with deeper networks, which result in the introduction of rewinding and a consequent revision to the original hypothesis.
Liu et al.
(2019) question the importance of weights resetting and observe for moderate pruning rates and without rewinding that the pruned architecture alone is responsible for successful training.
Zhou et al.
(2019) conduct ablation studies on the lottery tickets hypothesis and show, among others, the importance of the signs of the reset weights.
Similar to our paper, these works aim at better understanding winning tickets properties; however none of them investigate their relationship with the data and labels on which they are trained, though Yu et al.
(2019) investigate lottery tickets in reinforcement learning problems.
Closer to our work, Morcos et al.
(2019) show that winning tickets initializations transfer across different image classiﬁcation datasets, thus suggesting that winning tickets do not entirely overﬁt to the particular data distribution on which they are found.
Here, we follow up on these observations by measuring the importance of labels and data size to the quality of winning tickets.
Learning without supervision.
As we study situations with limited access to labels, our work is also related to self-supervised learning where a network is trained on a pretext task that does not require any manual annotations.
Two main broad types of self-supervised learning approaches appear in the litterature.
The ﬁrst one consists of methods where the pretext task is created by manipulating the input data.
This includes predicting relative spatial location, colorizing grayscale images or predicting the rotation applied to an image (Doersch et al., 2015; Wang & Gupta, 2015;
Zhang et al., 2016; Noroozi & Favaro, 2016; Pathak et al., 2017; Gidaris et al., 2018; Tian et al., 2019; H´enaff et al., 2019).
The second one is composed of methods where images are treated as different instances that should be discriminated from one another (Dosovitskiy et al., 2016; Bojanowski & Joulin, 2017; Wu et al., 2018; Caron et al., 2018).
Representations learnt using self-supervision are most often evaluated via transfer learning to a supervised task.
The better the pre-training with self-supervised learning, the better the performance on the transfer task.
In this work, we propose a new way of exploiting signal learned with self-supervision by ﬁndind winning tickets.
B ACKGROUND ON L OTTERY T ICKETS AND I MPLEMENTATION
In this section, we provide some background on the lottery tickets hypothesis and more particularly about the procedure to ﬁnd winning tickets.
We also give more details about our implementation.
3. E XTRACTING AND TRAINING SUBNETWORKS
We detail in this section the approach of Frankle et al.
(2019) to extract sparse and trainable subnetworks from an over-parameterized network.
These subnetworks, or winning tickets, are generated with a magnitude-based unstructured iterative pruning process (Han et al., 2015) and a weight resetting scheme (Frankle & Carbin, 2019; Frankle et al., 2019).
Subnetwork and dataset.
A subnetwork or winning ticket ( W, m ) is represented by the association of a mask m in { 0 , 1 } d and weights W in R d .
The convolutional network, or convnet, function associated with a subnetwork ( W, m ) is denoted by f m (cid:12) W , where (cid:12) is element-wise product.
We consider a training set { ( x 1 , y 1 ) , . . .
, ( x N , y N ) } of N pairs of images and their labels.
Magnitude-based unstructured iterative pruning.
Han et al.
(2015) propose an algorithm to prune networks by estimating which weights are important.
This approach consists of compressing networks by alternatively minimizing a loss L g and pruning the network parameters with the smallest magnitude, hence progressively reducing the network size.
We refer to the training objective L g as the winning ticket generation task.
More speciﬁcally, at each pruning iteration, the network is ﬁrst trained to minimize the following problem:
[Equation]
thus arriving at weights W ∗ .
The mask m is updated by setting to zero the elements already masked plus the smallest elements of {| W ∗ [ j ] | | m [ j ] (cid:54) = 0 } .
This method operates as a post-processing step to prune an already-trained network while preserving good accuracy at high compression rates.
Weight resetting.
In addition to weight pruning, Frankle & Carbin (2019) propose to also ﬁnd a good initialization for each subnetwork, such that it may be retrained from scratch.
On small-scale computer vision datasets and with shallow architectures, they indeed show that sub-architectures found with iterative magnitude pruning can be retrained from the start, as long as their weights are reset to their initial values.
Further experiments, however, have shown that this observation does not exactly hold for more challenging benchmarks such as ImageNet (Frankle et al., 2019; Liu et al., 2019).
Speciﬁcally, Frankle et al.
(2019) found that resetting weights to their value “early in training” can still lead to good trainable subnetworks.
Formally, at each pruning iteration, the subnetwork is reset to weights W k obtained after k weight updates from the ﬁrst pruning iteration.
Winning ticket generation task.
Most studies about the lottery tickets hypothesis evaluate winning tickets in the same setting (i.e. task and dataset) used to ﬁnd them.
In other words, the resulting winning tickets performance is measured by re-training them on the same objective L g that was used in the generation process (Eq. 1).
In this paper, however, we vary the generation loss L g while the evaluation remains constant.
For example, to generate “label-agnostic” winning tickets (i.e. that do not depend on labels), L g might correspond to a self-supervised loss.
We evaluate winning tickets by reporting their performance after training on label classiﬁcation on the full dataset.
3. I MPLEMENTATION
Lottery tickets.
We follow the lottery tickets setup of Morcos et al.
At each pruning iteration, we globally pruned 20% of the remaining weights.
The last fully-connected and batch- norm layers parameters are left unpruned.
We apply up to 30 pruning iterations to reach extreme pruning rates where only 0 .
1% of the weights remain.
Late resetting is set to 3 × 1 , 3 M samples, which corresponds to 3 epochs on full ImageNet.
More details about the rewinding parameter are in the Appendix B.
We use the “random” baseline of Morcos et al.
(random mask and weights).
Datasets and models.
We use CIFAR-10 (Krizhevsky, 2009) or ImageNet (Deng et al., 2009) datasets.
On ImageNet, we consider AlexNet (Krizhevsky et al., 2012) and ResNet-50 (He et al., 2016) architectures.
On CIFAR-10 we use ResNet-18 and the modiﬁed VGG-19 (Simonyan & Zisserman, 2014) of Morcos et al.
(multilayer perceptron (MLP) is replaced by a fully connected layer).
Models are trained with an (cid:96) 2 penalization of the weights.
Details about optimization schemes are in Appendix A.
We run each experiment with 6 (CIFAR- 10 ) or 3 (ImageNet) random seeds, and report the mean and standard error of the accuracy (tables of results are in Appendix E).
E XPERIMENTAL STUDY
In this section, we look at how “label-agnostic” winning tickets transfer to label-dependent task.
Then, we investigate if we can ﬁnd good performing winning tickets with a reduced number of samples and/or classes.
Finally, we also question if the “label-agnostic” winning tickets are complementary to the tickets found on a subset only of data through a semi-supervised generation task.
4. L ABEL - AGNOSTIC WINNING TICKETS
In this section, we are interested in the properties of “label-agnostic” winning tickets, i.e. found with self-supervised tasks, w.r.t.
to a target supervised task.
We compare their performance with winning tickets found with supervision to measure the importance of labels in the winning tickets generation.
Experimental setting.
We consider 2 self-supervised methods: RotNet (Gidaris et al., 2018) and the Exemplar approach of Dosovitskiy et al.
(2016) following Doersch & Zisserman (2017) implementation.
RotNet consists in predicting the rotation which was applied to the input image among a set of 4 possible large rotations: { 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ } .
Exemplar is a classiﬁcation problem where each image and its transformations form a class, leading to as many classes as there are training examples.
We choose these 2 self-supervised tasks because they have opposite characteristics:
RotNet encourages discriminative features to data transformations and has a small number of classes, while
Exemplar encourages invariance to data transformations and its output space dimension is large.
Transferring label-agnostic winning tickets to a supervised task.
In Figure 1, we show the performance on label classiﬁcation of the label-agnostic winning tickets at different pruning ratios.
These tickets yield good accuracy on label classiﬁcation on both CIFAR-10 and ImageNet datasets.
Indeed, on CIFAR-10 (Figure 1a), when only 6 . 9% of the weights remain, the RotNet winning ticket performs as well as the labels winning ticket.
The label-agnostic tickets only fail for AlexNet where most of the weights are in the MLP used in the classiﬁer.
An explanation is that, since the MLP corresponds to the last layers of the network, it is greatly inﬂuenced by the labels.
We investigate this hypothesis in the set of experiments below.
Note that, however, on all the other models, the gap of performance between label-agnostic tickets and random masks increases when the network is severely pruned.
In the extreme pruning regime, the label-agnostic winning tickets perform as well as the ones generated with the supervised task.
Results in Appendix G suggest that the quality of the pruned mask itself is similar for labels and self-supervised subnetworks but the weights of selfsupervised winning tickets are not as good starting points as the ones from labels.
Overall, winning tickets are rather robust to the absence of labels.
Layerwise lottery tickets.
We further investigate the difference between label-agnostic and supervised lottery tickets in AlexNet by looking at their performance as we prune up to a given depth.
In
Figure 2, we verify that for the convolutional layers, the label-agnostic tickets match the performance of supervised tickets, and the performance drops only with the MLP.
Note that, even if this effect is particularly visible with an AlexNet, it is happening with most self-supervisedly trained network.
Indeed, it has been observed that most self-supervised approaches (Jing & Tian, 2019) produce good shallow and mid-level features but poor high level features.
This means that we should expect the
[Figure]
N e t v a l ac
[Figure]
[Figure]
quality of label-agnostic winning tickets to depend on the depth of the pruning.
In Appendix C, we conﬁrm this intuition with a ResNet- 50 , and show the difference between label-agnostic and supervised lottery tickets increases with depth.
4. L OW - SHOT WINNING TICKETS
In this section, we are interesting in the quality of winning tickets generated with only a few annotated images.
We explore several ways to extract a subset by varying its size or its distribution and study the impact of these choices on the subsequent winning tickets quality.
[Figure]
I m a g e N e t v a l acc
[Figure]
Experimental setting.
We consider either 1% , 10% or 30% of ImageNet, with 1% corresponding to approximately 13 k images.
When varying the dataset size, we sample the subset per class, i.e., we preserve all the classes.
When varying the number of classes, we sample a subset of classes and keep all the images for these classes, i.e., 10% of ImageNet means that we only have 100 classes.
Varying dataset size.
First, we generate winning tickets using a subset of the training data while preserving the same number of classes.
Results for CIFAR- 10 are in Appendix F. Note that generating winning tickets with 10% of ImageNet takes 5 × less time than with the full dataset.
In Figure 3a, we show the ImageNet validation accuracy for the winning tickets of a ResNet-50 generated with these different subsets.
In less extreme pruning scenarii, there is a gap of 1 − 4% between winning tickets generated with the full training set and 10% of it but the winning tickets behave in the same ballpark for all settings for extreme pruning, i.e., for more than 90% of weights pruned.
This experiment shows that winning tickets are not data hungry, especially for extreme pruning.
Varying the number of classes.
In Figure 3b, we analyze the impact of changing the joint datalabel distribution while keeping the subsets size constant with that of Figure 3b.
Speciﬁcally, we are interested in the impact of the number of classes used during the generation of the tickets.
We observe that reducing the number of labels has little effect on the quality of the winning tickets.
This observation reinforces the ﬁndings on label-agnostic generation, that winning tickets are robust to the absence or deterioration of the labels.
Even more surprisingly, we observe good performance with only 1% of the classes, i.e., 10 classes and 13 k images, leading to a 10 × faster generation.
[Figure]
of
[Figure]
Decorrelating label and image distributions.
A possible explanation for these positive results about label-agnostic and low-shot winning tickets is that winning tickets generated with any task related to the right target distribution lead to good tickets.
As a control experiment, we propose
[Figure]
to learn winning tickets on a classiﬁcation tasks where the labels have been shufﬂed, i.e., where the image and label distributions have been decorrelated.
In Figure 4, we show on CIFAR-10 the performance of winning tickets generated on randomly permuted labels.
As expected, decorrelating the label and the image distributions destroys the quality of the subsequent winning tickets, since they are not better than the adjusted random baseline (see Sec.
4.5 for details on this baseline).
4. S EMI - SUPERVISED WINNING TICKETS
In this experiment, we investigate if label-agnostic and low-shot winning tickets are complementary.
Experimental setting.
We ﬁnd winning tickets by training networks in the semi-supervised setting of Zhai et al.
(2019) with 10% of labeled images from ImageNet and the remaining 90% unlabeled.
In particular, the generation task L g corresponds to the sum of a label classiﬁcation loss applied to labeled examples only, and a RotNet loss applied to all data samples.
Results.
In Figure 5-a, we compare these winning tickets with the label-agnostic, the low-shot and the supervised ones.
Surprisingly, the low-shot tickets perform better than the label-agnostic ones.
More importantly, we observe that winning tickets generated in the semi-supervised setting perform better in general than both the label-agnostic and low-shot ones.
This shows that properties of both appraoches capture different statistics that add up to generate better winning tickets.
4. C OMPARAISON WITH WINNING TICKETS TRANSFERRED FROM OTHER DATASETS
In this experiment, we compare the winning tickets transferred from other datasets to those obtained without labels or limited number of data.
Experimental setting.
We follow the setting of Morcos et al.
and use Places365 (Zhou et al., 2017) and CIFAR- 10 which count respectively 1 .
8 M and 50 k training data samples andand 10 categories.
We consider these two datasets because they have different statistics, in particular
Places365 is very large while CIFAR- 10 is of comparable size with the subsets used for our low-shot experiments.
For both datasets, we generate winning tickets with supervision.
Results.
In Figure 5-b, we compare these lottery tickets with those transferred from other data distributions even if trained with more data.
Interestingly, winning tickets generated with only 10% of the images perform better than the tickets generated on Places365 with 14 × more data and 3 × more classes.
This suggests that lottery tickets are more sensitive to the data distribution than the number of samples or the use of labels to generate them.
[Figure]
w e i gh t m
a gn
it ud e
[Figure]
Somewhat surprisingly, we ﬁnd that prior to any pruning, a large proportion of the weights of a deep architecture trained on CIFAR-10 has converged naturally to zero during training.
For example, we observe in Figure 6 that ∼ 85% of the weights of a VGG-19 and ∼ 80% of that of a ResNet- 18 at convergence are zeroed (Appendix D shows results on more architectures).
As a result, it is trivial to prune these networks without any loss in accuracy.
Unstructured magnitude-based pruning acts here as training since are freezed to zero weights that were going to zero anyway
(Zhou et al., 2019).
Overall, while pruning on CIFAR-10 large networks originally tuned for ImageNet at rates above their natural level of sparsity ( ∼ 80% ) is still meaningful, analyzing pruning below this rate may not be conclusive.
We ﬁnd that in the random global pruning baseline (which can remove non zero weights) in Figure 1a, pruning at rates below the natural sparsity of the network degrades accuracy, while pruning of weights that are already zeroed has no effect.
Inconveniently, this performance gap carries over to higher pruning rates (in which we are interested in) and can lead to misleading interpretations.
For fair comparison, we adjust the random mask baseline in Figure 1a: we remove this effect by ﬁrst pruning the weights that naturally converge to zero after training. Then, we randomly mask the remaining non-zeroed weights to get different ﬁnal desired pruning rates. The remaining non-masked weights are randomly initialized. This baseline therefore corrects for the natural sparsity present in CIFAR-10 networks.
D ISCUSSION
Overall, our study empirically shows that we can ﬁnd winning tickets with good accuracy on label classiﬁcation with no supervision at all.
This observation suggests that winning tickets are, to some extent, surprisingly labels and task-independent.
We also ﬁnd that they are not very data-demanding since a dataset subset is sufﬁcient to generate them.
Finding winning tickets with a subset of ImageNet is 5 × faster than using the entire dataset, without modifying the original algorithm.
Such a speedup is especially crucial for extreme pruning rates that are computationally demanding.
Nonetheless, we ﬁnd several critical limitations to winning tickets in our study.
First, none of the tickets found with limited access to labels and or data matches the accuracy of tickets found with all the labeled data when considering moderate pruning rates (more than ∼ 10% of unpruned weights) on ImageNet.
Indeed, we consistently observe a decrease in performance compared to the full overparametrized network as soon as we prune the network.
Therefore, for these rates, winning tickets are certainly label and data dependant.
Second, we ﬁnd that pruning large modern architectures on
CIFAR-10 should be done with caution as these networks tend to be sparse at convergence, making unstructured pruning at rates below 80% particularly simple.
Third, we observe that winning tickets are particularly sensitive to the late resetting parameter (see Appendix B for a discussion about our choice of rewind parameter).
The deﬁnition of “early in training” is somehow ill-deﬁned: network weights change much more for the ﬁrst epochs than for the last ones.
Thus, by resetting weights early in their optimization, they still contain a vast amount of information.
Finally, we have observed winning tickets through the scope of label classiﬁcation for images with natural-looking statistics only.
Considering more self-supervised tasks and even different types of tasks (generative modeling for example) might complement furthermore this study.
A PPENDICES A H YPERPARAMETERS AND MODEL DETAILS
We detail in this appendix the different hyperparameters used in our experiments.
For all trainings on ImageNet, we use stochastic gradient descent with a momentum of 0 .
We use Adam optimizer on CIFAR-10.
On ImageNet, unless speciﬁed otherwise, we perform standard data augmentation consisting in croppings of random sizes and aspect ratios and horizontal ﬂips Krizhevsky et al.
On CIFAR-10, we use horizontal ﬂips and croppings of ﬁxed size on a 2 -padded input image.
We use PyTorch version 1.0 for all our experiments.
On ImageNet, we use the “Dali” library for fast data augmentation implementation 1 .
• ImageNet Labels - full dataset - AlexNet : we train for 90 epochs with a total batch-size of 4096 distributed over 8 GPUs ( 512 samples per GPU), learning rate of 0 . 4 , weight decay of 0 . 0001 .
We decay the learning rate by a factor 10 at epochs 30 and 60 .
• ImageNet Labels - full dataset - ResNet-50 : we train for 90 epochs with a total batch-size of 1536 distributed over 16 GPUs ( 96 samples per GPU), learning rate of 0 . 1 , weight decay of 0 . 0001 .
We decay the learning rate by a factor 10 at epochs 50 , 65 and 80 .
• ImageNet Labels - 30% dataset : we train for 200 epochs with a total batch-size ofdistributed over 16 GPUs ( 96 samples per GPU), learning rate of 0 .
1 warmed up during the ﬁrst 5 epochs, weight decay of 0 . 0003 .
We decay the learning rate by a factor 10 at epochs 140 , 160 and 180 .
• ImageNet Labels - 30% of classes : we train for 200 epochs with a total batch-size ofdistributed over 16 GPUs ( 96 samples per GPU), learning rate of 0 .
03 warmed up during the ﬁrst 5 epochs, weight decay of 0 .
We decay the learning rate by a factor 10 at epochs 140 , 160 and 180 .
• ImageNet Labels - 10% dataset : we train for 200 epochs with a total batch-size ofdistributed over 8 GPUs ( 96 samples per GPU), learning rate of 0 .
1 warmed up during the ﬁrst 5 epochs, weight decay of 0 .
We decay the learning rate by a factor 10 at epochs 140 , 160 and 180 .
• ImageNet Labels - 10% of classes : we train for 200 epochs with a total batch-size ofdistributed over 8 GPUs ( 96 samples per GPU), learning rate of 0 .
3 warmed up during the ﬁrst 5 epochs, weight decay of 0 . 0003 .
We decay the learning rate by a factor 10 at epochs 140 , 160 and 180 .
• ImageNet Labels - 1% dataset : we train for 1000 epochs with a total batch-size ofdistributed over 8 GPUs ( 96 samples per GPU), learning rate of 0 . 1 , weight decay of 0 .
We decay the learning rate by a factor 10 at epochs 700 , 800 and 900 .
We follow Zhai et al.
and use random color data augmentation during training.
• ImageNet Labels - 1% of classes : we train for 1000 epochs with a total batch-size of 768 distributed over 8 GPUs ( 96 samples per GPU), learning rate of 0 .
01 , weight decay of 0 .
We decay the learning rate by a factor 10 at epochs 700 , 800 and 900 .
We follow Zhai et al.
and use random color data augmentation during training.
• ImageNet Semi-supervised RotNet : We reproduce the semi-supervised method of Zhai et al.
and follow precisely their hyperparameter.
We train for 200 epochs with a total batch-size of 2048 distributed over 32 GPUs ( 64 samples per GPU), learning rate of 0 .
1 warmed up during the ﬁrst 5 epochs, weight decay of 0 . 0003 .
We decay the learning rate by a factor 10 at epochs 140 , 160 and 180 .
• ImageNet RotNet - ResNet-50 : we train for 90 epochs with a total batch-size ofdistributed over 16 GPUs ( 96 samples per GPU), learning rate of 1 warmed up during the ﬁrst 5 epochs, weight decay of 0 .
We decay the learning rate by a factor 10 at epochs 50 , 65 and• ImageNet Exemplar - ResNet-50 : we train for 30 epochs with a total batch-size of 1536 distributed over 32 GPUs ( 48 samples per GPU), learning rate of 1 , weight decay of 0 .
We decay the learning rate by a factor 10 at epochs 20 and 25 .
We follow the
Exemplar implementation based on triplet margin loss from Doersch & Zisserman (2017).
As the goal of the task is to learn invariance to data transformation, we use data color augmentation and random small rotations on top of standard data augmentation scheme.
• ImageNet RotNet - AlexNet : we train for 90 epochs with a total batch-size of 8192 distributed over 16 GPUs ( 512 samples per GPU), learning rate of 0 .
5 warmed up during the ﬁrst 2 epochs, weight decay of 0 .
We decay the learning rate by a factor 10 at epochs 30 and 60 .
• ImageNet Exemplar - AlexNet : we train for 30 epochs with a total batch-size ofdistributed over 16 GPUs ( 256 samples per GPU), learning rate of 0 . 1 , weight decay of 0 . 0001 .
We decay the learning rate by a factor 10 at epoch 20 .
We follow the Exemplar implementation based on triplet margin loss from Doersch & Zisserman (2017).
We use data color augmentation and random small rotations on top of standard data augmentation scheme.
• Places Labels - ResNet-50 : we train for 90 epochs with a total batch-size of 2048 distributed over 16 GPUs ( 128 samples per GPU), learning rate of 0 . 1 , weight decay of 0 . 0001 .
We decay the learning rate by a factor 10 at epochs 50 , 65 and 80 .
• CIFAR-10 Labels & RotNet - VGG-19 & ResNet-18 : we train for 160 epochs with a total batch-size of 512 on 1 GPU, learning rate of 0 .
001 , weight decay of 0 . 0001 .
We decay the learning rate by a factor 10 at epochs 80 and 120 .
• CIFAR-10 Exemplar - VGG-19 & ResNet-18 : we train for 180 epochs with a total batchsize of 512 on 1 GPU, learning rate of 0 .
0003 , weight decay of 0 . 0001 .
We decay the learning rate by a factor 10 at epochs 120 .
As the goal of the task is to learn invariance to data transformation, we use data color augmentation and random small rotations on top of standard data augmentation scheme.
• CIFAR-10 Randomly permuted labels - VGG-19 & ResNet-18 : we train for 160 epochs with a total batch-size of 512 on 1 GPU, learning rate of 0 .
001 , weight decay of 0 . 0001 .
We decay the learning rate by a factor 10 at epochs 80 and 120 .
We do not apply data augmentation.
B L ATE RESETTING PARAMETER
We follow Frankle et al.
and use late resetting (or rewind ) for the winning tickets generation process.
Indeed, before re-training a winning ticket, we reset its weights to their value “early in training” of the full over-parameterized network.
In our work, we set the late resetting parameter to 1 epoch on CIFAR-10.
However, when dataset size, total number of epochs, mini-batch sizes or learning rate vary, it becomes more complicated to choose a rewind criterion that guarantees a fair comparison between all settings.
A choice can be to rewind at a point where “the same amount of information” has been processed.
Thus, in our work, we choose to set the rewind parameter to 3 × 1 , 280 , 000 samples for all our experiments on ImageNet, which corresponds to 3 epochs on full
ImageNet.
We describe in Table 1 to what this rewind parameter corresponds to in terms of number of epoch, number of data samples seen, number of gradient update and percentage of total training for our different experiments.
Moreover, we show in Figure 7 the performance of winning tickets generated using 10% of ImageNet with different values of rewind.
Each of the considered value corresponds to keeping one of the criteria (number of epoch, number of data samples seen, number of gradient update or percentage of total training) ﬁxed compared to the rewind parameter on full
ImageNet (ﬁrst row of Table 1).
# epochs # samples
[Table]
[Table] [Table]
training
[Table]
% ∼ 3 ×
[Table] [Table]
% ∼
[Table]
×
[Table]
[Table] [Table]
[Figure] [Figure]
tickets generated by pruning partly or entirely (all) a network with 2 generation tasks: labels classiﬁcation or RotNet.
AlexNet is used in top row and ResNet- 50 in the bottom row.
We also show for reference results when the network layers are randomly pruned.
C L AYERWISE WINNING TICKETS
In this appendix we show more results on the layerwise winning tickets experiment (see 4.1).
We generate winning tickets by pruning only the n ﬁrst convolutional layers of a network, and leaving the remaining of the network unpruned.
On AlexNet (ﬁrst row of Figure 8), we consider 4 situations.
From left to right in the ﬁrst row of Figure 8), we prune the ﬁrst convolutiona layer; up to the third convolutiona layer; up to the ﬁfth convolutional layer; or the whole network.
For ResNet-50, from left to right in the bottom row of Figure 8), we prune the ﬁrst 11 convolutiona layers; the ﬁrst 24 convolutiona layers; the ﬁrst 43 convolutional layers; or the whole network.
Consistently with observations from the self-supervised literature, self-supervised winning tickets are competitive with labels ones when only the shallow and mid-level layers are pruned.
Indeed, these layers are not specialized yet for the labels classiﬁcation tasks and many works have shown that representations learned are similar at these depths.
D S PARSE TRAINED NETWORKS ON CIFAR-
In this appendix, in Figure 9, we provide results on more architectures about the proportion of weights zeroed during training on CIFAR-10 compared to ImageNet.
Note that we do not consider the batch-norm layers parameters, nor the parameters of the last fully-connected layer (since we do not prune it in our setting).
For all the considered VGGs we use the modiﬁed version of Morcos et al.
(2019), replacing the ﬁnal MLP by a fully connected layer. [Figure] i gh t
[Figure]
w e i gh t m
a gn
it ud e
[Figure]
w e i gh t m
a gn
it ud e
[Figure]
E E XACT NUMBERS FROM
In Tables 2 3 4 5 6 7 8, we report accuracies for each setting presented in the main paper.
We report mean and standard errors for our experiments which we run with 3 different seeds for ImageNet and
Places- 365 and 5 different seeds for CIFAR- 10 .
F V ARYING DATASET SIZE ON CIFAR-
We choose to present results with ImageNet only in the main paper because varying the dataset size of an already small dataset such as CIFAR-10 is less conclusive.
Moreover, as noted before 4.5, studying sparsity on CIFAR- 10 should be done precautiously since trained deep networks are already sparse.
In this Appendix, we present results on CIFAR- 10 in Table 10.
With VGG- 19 architecture, we observe that decreasing the dataset size still leads to reasonable winning tickets.
However, with
{+ResNet- 
+}
we observe an important performance gap of winning tickets generated with a fraction of
[Table]
[Table]
the dataset compared to using the entire dataset.
Indeed, when only only 10% of CIFAR- 10 (images) the generated winning tickets are at random level.
G R ANDOM RE - INITIALIZATION OF WINNING TICKETS
In this Appendix
[Table]
we present results with random re-initialization for winning tickets found with labels or with RotNet self-supervised task on both ImageNet and CIFAR- 10 in Figure 11.
On ImageNet 11b, consistently with the experiments of Frankle et al.
(2019), we observe that resetting the weights accordingly is crucial to get high accuracy.
Indeed, on both ResNet- 50 and AlexNet, for labels, the subnetworks that are reset to their weights early in training (dark blue plain line) perform signiﬁcantly better than subnetworks randomly re-initialized (dark blue dashed line).
Interestingly, this is not the case for RotNet winning tickets: starting from original weights (pink plain line) gives only a very slight boost (or even no boost at all) of performance compared to randomly re-initialization (pink dashed line).
Overall, labels or RotNet subnetworks perform in the same ball park when randomly re-initialized, but using the original weights gives a large boost of performance for labels but not for RotNet.
Thus, it suggests that the information carried by the pruned mask itself
[Table]
[Table]
is similar for labels and RotNet subnetworks but the weights of the RotNet winning tickets are not as good starting points as the weights from labels winning tickets.
On CIFAR-10 11a, up to a certain level of sparsity (that roughly corresponds to the natural level of sparsity of the network), using random re-initialization or weights “early in training” gives similar performance.
However, for more extreme pruning rates, using a particular set of weights gives signiﬁcantly better performance than random re-initialization.
Our results are consistent with the ﬁndings of Liu et al.
(Figure 7.a of their paper), Frankle et al.
(Appendix A of their paper)
Morcos et al.
(Appendix A2 of their paper).
[Table]
[Table]
[Table]
[Figure]
[Figure]