T HE S OONER THE B ETTER : I NVESTIGATING S TRUCTURE OF E ARLY W INNING L OTTERY T ICKETS
Anonymous authors
Paper under double-blind review
A BSTRACT
The recent success of the lottery ticket hypothesis by Frankle & Carbin (2018) suggests that small, sparsiﬁed neural networks can be trained as long as the network is initialized properly.
Several followup discussions on the initialization of the sparsiﬁed model have discovered interesting characteristics such as the necessity of rewinding (Frankle et al.
, importance of sign of the initial weights (Zhou et al.
(2019)), and the transferability of the winning lottery tickets (S. Morcos et al.
In contrast, another essential aspect of the winning ticket, structure of the sparsiﬁed model, has been little discussed.
To ﬁnd the lottery ticket, unfortunately, all the prior work still relies on computationally expensive iterative pruning.
In this work, we conduct an in-depth investigation of the structure of winning lottery tickets.
Interestingly, we discover that there exist many lottery tickets that can achieve equally good accuracy much before the regular training schedule even ﬁnishes.
We provide insights into the structure of these early winning tickets with supporting evidence.
1) Under stochastic gradient descent optimization, lottery ticket emerges when weight magnitude of a model saturates;
2) Pruning before the saturation of a model causes the loss of capability in learning complex patterns, resulting in the accuracy degradation.
We employ the memorization capacity analysis to quantitatively conﬁrm it, and further explain why gradual pruning can achieve better accuracy over the one-shot pruning.
Based on these insights, we discover the early winning tickets for various ResNet architectures on both CIFAR10 and ImageNet, achieving state-of-the-art accuracy at a high pruning rate without expensive iterative pruning.
In the case of ResNet50 on ImageNet, this comes to the winning ticket of 75 .
31% Top-1 accuracy at 80% pruning rate in only 22% of the total epochs for iterative pruning.
I NTRODUCTION
Deep Neural Networks (DNNs) achieve superior accuracy in a wide spectrum of applications through the use of very large and deep models (Goodfellow et al.
These high-capacity but complex models, however, pose a tremendous challenge for their deployment, particularly in resource-constrained edge environments.
Over the years, many techniques have been developed to compress the models to a compact counterpart to alleviate the computational costs.
Among these techniques, pruning less important parameters to obtain a compact sub-network has emerged to be a popular and efﬁcient approach (Cun et al.
; Han et al.
In search of these sub-networks, new intuitions are also built up for understanding the DNN working mechanism, one example of which is the recently proposed "lottery ticket hypothesis" by Frankle & Carbin (2018).
The lottery ticket hypothesis states that, once a sub-network is found to match the accuracy of the original neural network, the sub-network (i.e., lottery ticket) together with its initialized weights can be trained in isolation and still achieve accuracy comparable to the original network within a similar number of iterations.
This conjecture intrigues discussions on a series of topics, such as the importance of initialization scheme (Liu et al.
, Zhou et al.
, the role of over-parameterization in training (Frankle et al.
and even the transferability of the "winning ticket" (S. Morcos et al.
However, all these discussions start from the point that the winning ticket has been obtained after painfully long iterative pruning procedures, which often take up to thousands of epochs.
When and how a winning ticket can be found in the course of pruning procedures has not been studied; most of the prior works use a traditional way of repeated cycles of pruning and retraining, which makes such study less practical.
In this work, we provide insights to ﬁnd the winning tickets early .
We start with an interesting observation that there exist many lottery tickets that can achieve equally good accuracy much before the regular training schedule ﬁnishes.
This leads to our in-depth investigation of the weight magnitude; we discover that a model saturates early but not too early under stochastic gradient descent (SGD) optimization.
To understand this characteristic of the winning lottery tickets, we conjecture that pruning a premature model causes the loss of capability in learning complex patterns, leading to accuracy degradation.
We conﬁrm this conjecture with the empirical evidence as well as the quantitative analysis based on the memorization capacity.
Using this analysis framework, we further provide a reasoning behind the success of the gradual pruning over the one-shot pruning.
Based on these insights, we identify the early winning tickets for various ResNet architectures on both CIFAR10 and ImageNet, achieving state-of-the-art accuracy at a high pruning rate without expensive iterative pruning.
In the case of ResNet50 on ImageNet, this comes to the winning ticket of 75 .
31% Top-1 accuracy at 80% pruning rate obtained within only 22% of the total epochs for iterative pruning.
This promising outcome not only sheds light on understanding the optimization behavior of the pruned models but also enables performance gain for fast training of the pruned models.
R ELATED W ORK
The lottery ticket hypothesis was ﬁrst proposed in Frankle & Carbin (2018) where the presence of a trainable subnetwork that achieves compelling ﬁnal accuracy with the inherited initial values for the un-pruned connections is demonstrated.
This paper also argued that the initial weight of the original network is essential for maintaining good accuracy when the model is sparsiﬁed.
This claim has been extended to the over-parameterized neural networks on the larger datasets in Frankle et al.
with the notion of "rewinding"; the authors claim that rewinding of the weights not to the initial values but to the values after a few epochs can stabilize the accuracy of the winning lottery tickets.
As a follow-up work, Zhou et al.
(2019) studied the critical components of the lottery tickets such as zeros, signs and the super-mask.
Also, S. Morcos et al.
(2019) investigated the transferability of winning tickets obtained in one dataset to the network of a similar structure for the other datasets.
But none of these focused on the structure of the winning lottery tickets; it involves several repetitions of the full training schedule for iterative pruning and retraining, often taking hundreds to thousands of epochs.
In this work, we demonstrate that many winning tickets can be found in the early stage of the baseline training schedule, drastically reducing the computational effort to obtain them.
Most work on the lottery ticket hypothesis, including this work, rely on magnitude-based weight pruning for identifying unimportant weights to be pruned (usually via global sorting).
Such an intuitive method was ﬁrst proposed in Han et al.
and became popular.
Afterwards, more complex pruning methodologies have been presented to improve pruning performance, such as having different pruning criteria (Li et al.
(2017), Wen et al.
Liu et al.
(2017)) or different pruning granularity (Mao et al.
(2017), Molchanov et al.
While these attempts offer insights on training pruned models, there is little discussion about the interesting interplay between the pruning criteria and the structure of a model pruned by it.
In this work, we reveal that the pruned structure obtained based on the weight magnitude has tangible impact on the ﬁnal accuracy, and further propose a way to quantitatively distinguish good structures for pruning.
There have been various strategies to apply pruning to the neural networks.
Iterative pruning by Han et al.
(2015) involves several repetition of pruning (with gradual increase of the pruning rate) and retraining.
One can increase the repetition cycles arbitrarily large to achieve good accuracy at high pruning rate; Frankle & Carbin (2018) employed iterative pruning with hundreds to thousands of pruning epochs to match the baseline accuracy for the challenging neural networks.
On the other hand, gradual pruning introduced by Zhu & Gupta (2017) determines the pruning rate and frequency via a polynomial equation as a function of the starting and ending epochs as well as the target pruning rate.
Although it provides a systematic pruning schedule, there is lack of discussion about when to start the gradual pruning.
Lastly, Lee et al.
(2018) proposes a method of pruning weights at initialization.
This method takes most advantage in performance since a pruned model can be obtained without any expensive retraining procedures.
However, its effectiveness has not been demonstrated for the challenging neural networks on large datasets such as ImageNet.
In contrast, we propose a mechanism to identify the winning lottery ticket early in the course of baseline training so that we can avoid costly iterative pruning while maintaining the baseline accuracy.
S TRUCTURE OF E ARLY W INNING T ICKETS
In this section, we extend the lottery ticket hypothesis by Frankle & Carbin (2018); Frankle et al.
(2019) to discuss the early winning tickets.
The lottery ticket hypothesis can be summarized as: for a given network of f ( x ; w 0 (cid:12) m 0 ) with the initial weight w 0 and the mask of all ones m 0 , there exists a winning lottery ticket m f where | m f | / | m 0 | = 1 − p % ( p is the pruning rate) and training of f ( x ; w v (cid:12)
m f ) for 0 < v (cid:28) f achieves test accuracy comparable to the baseline f
( x ; w f (cid:12) m 0 ) .
There are two main components of a winning lottery ticket: the sparsiﬁed structure m f and the weight that initializes it, w v .
In the previous work, m f has been obtained only after expensive iterative pruning.
We characterize the structure of the early lottery ticket m s where s (cid:28) f , then propose a strategy for ﬁnding it early.
3. E XTENDED L OTTERY T ICKET E XPLORATION
Frankle et al.
(2019) suggests that a lottery ticket found after a baseline training can achieve the accuracy of the baseline model if it is initialized with the weight of the baseline model after a few epochs, which is called rewinding .
We extend this exploration toward different structures obtained at different epochs of the baseline training (via magnitude-based weight pruning).
Fig.
1 shows the validation accuracy after retraining of a ResNet20 model on CIFAR10 pruned at different epochs of its baseline training.
The same learning rate schedule of 0.1 reduced by 10x at epoch 120 and 160 is used for both the baseline training and retraining, and the total number of epochs is 200.
The rows ( = v ) correspond to the different rewinding epoch, whereas the columns ( = s ) correspond to the different epoch that we apply one-shot pruning (pruning rate = 80% ).
For example, the validation accuracy along the lottery ticket conﬁgurations of ( s = 200 , v = 0 ∼ 200) resembles the phenomena of "rewinding" observed in Frankle et al.
Interestingly, Fig. 1 further demonstrates that the winning tickets emerge at much earlier epochs of the baseline training; the lottery ticket conﬁgurations of ( s ≥ 100 , v ≥ 5) achieve almost the same accuracy as the accuracy of ( s = 200 , v = 200) .
The accuracy then gradually decreases as s ≤ 80 .
This result implies two important aspects: 1) a winning ticket can be found in the middle of the baseline training so that one can avoid expensive iterative pruning used in the prior work, and 2) the winning ticket, however, does not emerge arbitrarily early in the process of training.
In the following sections, we investigate the characteristics of these early winning tickets.
In particular, we focus on their structure, as the weight initialization is not the major factor provided a proper rewinding.
3. A NALYSIS ON W EIGHT M AGNITUDE
As the ﬁrst step of understanding the characteristics of the early winning tickets, we focus on the important quantity of pruning, weight magnitude .
At pruning, we determine the structure of the sparsiﬁed model based on the rank of the
[Figure]
weight magnitude (via global sorting).
Therefore, the change in the weight magnitude during training has large impact on the lottery ticket structure.
In Fig.
2a, we measure the standard deviation of weight ( W std ) for each layer in the course of training ResNet20 on
CIFAR10 (note: the mean of weight is typically near zero).
The same learning rate schedule is used as above.
The ﬁrst thing to note is that the change in W std has strong correlation with the learning rate change.
In particular, different layers show different rate of change in W std when the learning rate is 0 .
1 , but from the second learning rate (after epoch 120), all the weights follow a very similar decreasing trend.
This trend in W std can be understood via steps of stochastic gradient descent.
From the typical setting of weight update with momentum and weight decay, we have:
[Equation]
where w t , w g,t and v t are the weight, gradient and momentum at step t , respectively; m is the momentum factor, η is the learning rate and λ is the weight decay factor.
After n steps,
[Equation]
From this derivation, we can see that the two factors determine the mode of change in W std .
When learning rate is high, the gradient terms play the major role in weight update.
On the other hand, if the gradient activity becomes low, e.g., when the learning rate is low and the gradients oscillate around zero, we can further simplify Eq. 3.
Assume that m n v t approaches to zero when n is relatively large, w t + k − 1 ≈ w t , and (cid:80) nk =1 ((1 − ηλ ) n − k w g,t + k − 1 ) and (cid:80) nk =1 ( m n − k w g,t + k − 1 ) approach to zero as the gradients oscillate around zero, we have v t
+ i ≈ λw t 1 − m .
Then w t + n is approximated as,
[Equation]
Note that (1 − ηλ ) n ≈ 1 − nηλ since ηλ (cid:28) 1 and (cid:80) n − 1 i =
((1 − ηλ ) i ) ≈ n .
Thus,
[Equation]
In other words, when the gradient activity is low, the change in W std is dominated by the both the learning rate and the weight decay.
As an example, in Fig.
2a, W std is decreased with the slope following Eq.
5 when learning rate is low;
W std of layer 10 is decreased by 1.5e-3 in 782 updates of CIFAR10 (with λ = 1e-4 and m = 0 . 9 ), conﬁrming the slope from Eq.
These two modes governing the change of weight magnitude are critical for understanding the behavior of pruning.
In particular, the interplay between learning rate and weight decay causes the per-layer pruning rate to converge after enough number of epochs.
Fig.
2b shows the pseudo per-layer pruning rate, where we just measure the layer-wise pruning rate without really pruning out the weights in the model, for the same CIFAR10 training experiment.
As the ﬁgure shows, the per-layer pruning rates saturate around epoch 100, indicating that pruning before that would select the weights based on the premature model.
Moreover, although the weight magnitude is an important factor, we discovered that the ranking of individual weight does not play a critical role in winning lottery tickets.
Speciﬁcally, we empirically show that there exist many winning lottery tickets that are vastly different in terms of its sparse structure m v .
Fig.
3 shows the hamming distance of the sparse structure of the lottery tickets at different conﬁguration ( s, v ) .
Note that the two distant lottery tickets (e.g., ( s = 200 , v = 100) and ( s = 100 , v = 100) ) show large hamming distance of 0.159 (where x/y = 200 / 100 ) while achieving the equally good accuracy as shown in Fig.
1, indicating that the ranking of the weights itself can not explain the quality of the structure of the lottery ticket.
3. U NDERSTANDING I MPACT OF P RUNED S TRUCTURE
The weight magnitude analysis motivates us not to prune a model too early.
It is also implied that a distance-based metric might not reveal the winning structure of the model at different epochs.
To understand the early winning structures, we further investigate the impact of the sparsiﬁed structure on the ﬁnal accuracy.
Considering that a highly pruned network is likely to have limited learning capability, we make a conjecture that the accuracy degradation of a pruned model is due to the loss of capability for learning complex pattern if pruned too early.
Recently, Li et al.
(2019) reveals that the training behavior of a sufﬁciently over-parameterized model with non-linearity highly depends on the learning rate schedule, where a model tends to memorize the complex patterns when a small learning rate is applied while learning simple patterns with a large learning rate.
To validate this claim in the context of pruning, we construct an experiment where a model is pseudo -pruned at every epoch of the baseline training then retrained with large or small learning rate for just one epoch.
For the pseudo-pruned-then-retrained (PPR) model at each baseline epoch, we measure the validation accuracy recovered from the retraining.
Fig.4 shows the result of this experiment on CIFAR10 ResNet20.
When it is retrained with the large retraining learning rate ( = 0 . 1 ), in just 1-epoch retraining, the PPR models from all the baseline epochs achieve the accuracy matching with the baseline accuracy.
This indicates that those pruned models maintain the capability of learning the simple patterns.
Whereas, when the small retraining learning rate ( = 0 . 01 ) is used, the accuracy of the PPR models pruned at later epochs (epoch 100-200) is higher than the accuracy of the models from earlier epochs (epoch 20-60).
This reveals that the PPR models from different epoch exhibit varying capability of learning complex patterns.
In particular, the accuracy of the PPR models increases until around 100 epoch of the baseline training, then it saturates.
Note that this coincides with the epoch when the early winning tickets emerges in Fig.
Based on this observation, we hypothesize that the models pruned at 100 epoch of the baseline training or later will preserve the capability of learning complex patterns.
To validate this hypothesis, we conducted the memorization test proposed by Boo et al.
(2019), where a model is trained with training data of varying size with the randomized labels.
[Figure]
A model has "high memorization capacity" if it achieves high training accuracy for a large data size.
Fig.
5 (left) shows the training accuracy of the models pruned at different epochs, measured from 10 independent simulations
(the average capacity shown as solid lines, and min and max shown as the shaded regions).
As the training data size increases, the memorization capacity decreases.
The model pruned too early (i.e., at epoch 20) suffers higher degradation in the memorization capacity compared to the models pruned at later epochs (i.e., at epochs 120 or 200).
Also, Fig. 5 (right) shows that those models pruned at later than 100 epochs (i.e., the early winning tickets) exhibit similar memorization capacity.
This result not only conﬁrms our conjecture on the impact of pruned structure to the capability of learning complex patterns, but also leads us to employ the PPR accuracy check as a computationally reasonable heuristic to discover the early winning tickets.
3. U NDERSTANDING G RADUAL P RUNING
Gradual pruning is a popular pruning approach that applies pruning gradually over a period of training.
For example, gradual pruning proposed by Zhu & Gupta (2017) provides a systematic way to schedule iterative pruning as follows:
[Equation]
where s f
and s i are the ﬁnal and initial sparsity, and t is the time when pruning is applied.
Eq.
6 determines how much sparsity is applied at a certain time step t .
But it is still a user-hyper-parameter to decide when t 0 or how often ∆ t apply pruning.
Based on the insights we discussed in the previous section, we explain why gradual pruning helps to obtain better lottery tickets.
The reasoning behind the gradual pruning is that the model can be changed graciously if the pruning is applied gradually.
In terms of the early winning ticket analysis, there are two factors playing the critical role: 1) by applying low pruning rate in the beginning, the structure found at that pruning level can preserve the memorization capacity better, 2) once pruning is applied, the remaining weights of the pruned model is updated via SGD, granting a chance for the pruned model to adopt its weights toward better accuracy.
Thanks to these two factors, a structure with better memorization capacity can be found when the increased pruning rate is applied next time.
Table 1 conﬁrms this explanation using the CIFAR10 ResNet20 example.
In this experiment, we perform the memorization test for the 4-step gradual pruning as well as four 1-step pruning at the corresponding epoch for comparison.
The gap in memorization capacity is maintained across the different epochs, demonstrating that the memorization capacity is maintained thanks to the gradual application of pruning and the evolution of weights after pruning.
This suggests a strategy for gradual pruning where 1) we can use gradual pruning to reduce the loss of capability in learning complex patterns, and 2) by applying a smaller pruning rate in the beginning, we can start pruning early and ﬁnding the winning lottery tickets faster. The beneﬁt of this strategy combining gradual pruning with the early winning tickets
[Table]
E XPERIMENTS : F INDING W INNING T ICKET E ARLY
In this section, we demonstrate our strategy of ﬁnding the early winning tickets over popular neural networks on
CIFAR10 and ImageNet.
The detail experimental setup is described in Appendix A.
We perform the lottery ticket experiments of Sec.
3.1 for both one-shot and gradual pruning with the pruning rate of 80% .We also conduct the PPR accuracy check of Sec.
3.3 to predict from which epoch
the early winning ticket can be found.
By comparing the two results, we demonstrate that the proposed heuristic for ﬁnding early winning tickets works robustly across the networks and the datasets.
Furthermore, we showcase our gradual pruning strategy by comparing the performance in terms of the accuracy and the required pruning epochs with the existing lottery ticket approaches.
4. E XPERIMENTS ON CIFAR
Table 2 summarizes the lottery ticket experiments of ResNet20 and ResNet56 on CIFAR10 dataset.
In case of one-shot pruning, the winning tickets can be found from epoch 100.
In case of gradual pruning, the gradual pruning schedule can start from epoch 75 (which is earlier than the one-shot pruning).
Note that the gradual pruning can achieve better accuracy as it can preserve higher memorization capacity as discussed in Sec. 3.4.
Fig.
6a shows the pseudo-pruning curve that highlights the presence of the winning tickets from the epoch around 100, which is consistent with the results of the lottery ticket experiment in Table 2.
[Figure]
4. E XPERIMENTS ON I MAGE N ET
Table 3 summarizes the lottery ticket experiments on ImageNet dataset, and the predicted results of the early winning tickets from the pseudo-pruning are shown in Fig.
6b.
Similar to the CIFAR10 experiments, the results of the lottery ticket experiment matches with the results from the PPR accuracy check (which indicates epoch 45 for early winning tickets), demonstrating the robust behavior of the proposed strategy of ﬁnding the early winning tickets.
Furthermore, the gradual pruning at the early winning tickets achieves the accuracy near to the baseline (ResNet18: baseline= 69 .
7% vs ours= 69 .
24% , ResNet50: baseline= 75 .
7% vs ours= 75 .
31% ), showcasing the superior quality of the winning tickets discovered by the proposed gradual pruning strategy. [Table] [Table]
4. P ERFORMANCE G AIN FROM E ARLY W INNING T ICKET
To demonstrate the performance gain from our gradual pruning strategy, the pruning results of the proposed algorithm (GP + EWT) and the previous implementation of iterative pruning (IP) along with the winning ticket (WT) of Frankle &
Carbin (2018) are shown for the ResNet variants on CIFAR10 and ImageNet in Table 4.
Our GP+EWT algorithm consistently achieves high pruning rate with the number of pruning epochs even lower than the regular retraining epochs for all the models, i.e., 80% pruning with negligible accuracy degradation on ResNet50 for ImageNet.
In contrast, the iterative pruning approach (IP+WT) achieves the similar accuracy at the cost of more than 4 .
5 × increase in the total training epochs.
C ONCLUSION
In this work, we investigate the structure of the winning lottery ticket, which leads to the computationally efﬁcient discovery of the winning lottery tickets.
Based on a careful analysis of the characteristics of the structure of the winning lottery tickets, we proposed a computationally reasonable heuristic to identify when the early lottery tickets emerge.
Furthermore, we proposed a gradual pruning strategy incorporating the early lottery ticket analysis to achieve high accuracy at large pruning rate.
This results in the state-of-the-art accuracy on ResNet50 for 80% pruning only within 22% of the total epochs for iterative pruning.
[Table]
al
[Table]
R
A E XPERIMENTAL S ETUP A. E XPERIMENTS ON CIFAR
We try to ﬁnd the winning tickets early for ResNet20 (baseline accuracy: 92.5%) and ResNet56 (baseline accuracy: 94.25%) on CIFAR10 dataset.
All the baseline models are trained using SGD optimizer with momentum ( m = 0.9) and weight decay ( λ = 1e-4) for 200 epochs.
The initial learning rate for the ﬁrst 120 epochs is 0.1 and decrease to 0.1 × every 40 epochs.
For the lottery ticket experiments, the retraining schedule is the same as the baseline learning rate schedule.
Following convention, we do not prune the ﬁrst and the last layers of the ResNets.
A. E XPERIMENTS ON I MAGE N ET
We further try to ﬁnd the winning tickets early for ResNet18 (baseline accuracy: 69.7%) and ResNet50 (baseline accuracy: 75.7%) on ImageNet dataset.
All the baseline models are trained using SGD optimizer with momentum ( m = 0.9) and weight decay ( λ = 1e-4) for 90 epochs.
The initial learning rate for the ﬁrst 30 epochs is 0.1 and decrease to 0.1 × every 30 epochs.
For the lottery ticket experiments, the retraining schedule is the same as the baseline learning rate schedule.
Following convention, we do not prune the ﬁrst and the last layers of the ResNets.
B D ISCUSSION B. I MPACT OF S TRUCTURED S PARSITY ON L OTTERY T ICKET H YPOTHESIS
In this section, we expand our scope of analysis on the characteristics of the early winning tickets toward the sparsity obtained via structured pruning.
Structured pruning is a popular method that prunes weights by a group.
For example,
Liu et al.
(2017) proposes channel-pruning, where a group consists of weights along each channel.
The main motivation of looking at the structured sparsity is that there is a signiﬁcant discrepancy in the best pruning rate we can achieve between the element-wise pruning (i.e., group-size=1) and the structured pruning (i.e., group-size=channel).
For example, on CIFAR10-VGG19, the element-wise pruning achieves 95% of pruning rate whereas the channel-pruning achieves 70% of pruning rate for the same level of accuracy (Liu et al.
Although such discrepancy has been observed for quite a while, there is no in-depth investigation to understand why.
To ﬁnd out the reasoning behind it, we analyze the impact of structured sparsity with varying group sizes in the context of the lottery ticket hypothesis, as illustrated in Fig.7.
First, we revisit the extended lottery ticket experiments of Fig.
1 while applying group sparsity along the channel dimension (instead of pruning individual weights).
Fig.
8 shows the results when the group size ( gs ) is 8.
Comparing it with Fig.
1, there are two notable points; 1) there is > 1% accuracy degradation when the group size is increased from 1 to 8, and 2) when gs = 8 , the trend that the tickets drawn at epochs s > 100 achieve the higher accuracy disappears (as highlighted with a red rectangle in the ﬁgure).
(Note that the two ﬁgures achieve the similar accuracy when s < 60 (colored in yellow), but the tickets with high accuracy (colored in green) can only be seen in Fig.
The former observation makes sense as the more regularization on the sparse weight structure would result in the lower accuracy.
But the latter observation is quite surprising; it seems that a distinct behavior of the lottery ticket hypothesis (i.e., the opportunity of ﬁnding winning tickets given a proper initialization) is disrupted as a group structure is asserted on the sparsity.
We can ﬁnd a clue on this disrupted behavior by employing the memorization capacity analysis.
Table 5 shows the results of memorization capacity experiments on CIFAR10-ResNet20 with an increasing group size gs = { 1 , 2 , 4 , 8 , 16 , 32 , 64 } .
The memorization capacity degrades signiﬁcantly as the group size increases.
Intuitively, by asserting a large group size, the model’s expressivity is degraded and it becomes harder to learn the complex patterns, leading to accuracy loss.
Note that we had a similar observation when the tickets are drawn too early (cf., s < 60 in Fig. 1).
From this, we can hypothesize that the winning tickets disappear as the memorization capacity is degraded, which can be observed when the tickets are drawn too early or the group sparsity is forced.
Fig.
9 further demonstrates the impact of structured sparsity on the saturation of pruning accuracy.
For different group sizes, the ﬁgure shows the accuracy of CIFAR10-ResNet20 80% pruned at varying epochs.
It can be observed that the models with the larger group size not only achieve the lower accuracy but also converge at earlier epochs.
(E.g., the knee points for gs = 1 and gs = 16 are around 90 and 50 epochs, respectively.)
This experimental result supports our claim that the larger group size enforced in the sparse structure results in the earlier convergence of the pruned models that misses the winning tickets.
B. M ODE C ONNECTIVITY OF E ARLY W INNING T ICKETS
Mode connectivity is a phenomenon that SGD solutions are connected through paths of approximately equal loss (Draxler et al.
It provides the perspective of how well the models trained via a proposed method can generalize.
In the context of early winning tickets, we for the ﬁrst time reveal that the lottery tickets drawn early based on PPR are indeed connected from the one obtained by IMP , whereas the tickets chosen too early do not.
For the mode connectivity experiments, we draw three lottery tickets - one drawn too early (i.e., a premature ticket, PT), one drawn based on PPR (i.e., an early winning ticket, EWT), and one at the end of the training schedule (i.e., a winning ticket, WT).
Note that all the tickets are trained from the same initial weights and with the same retraining schedule.
Then we linearly interpolate PT and EWT with WT, and plot the test error for different interpolation coefﬁcient (fromto 1).
Fig.
10 shows the results of the mode connectivity experiments for ImageNet and CIFAR10.
For both datasets, the linearly interpolated models between PT and WT depict a barrier, indicating that PT and WT are disconnected.
Whereas, the linearly interpolated models between EWT and WT show consistently low test error.
Note that such linear mode connectivity is observed even when 1) the weights are sparse (i.e., 80% pruning rate) and 2) the mask distance is large (cf., Fig. 3).
This intriguing observation counters some observations of prior work; Draxler et al.
(2018) claims that it takes a careful search (instead of a simple linear interpolation) to ﬁnd a pass connecting the modes, and the existence of such a non-linear pass requires enough number of parameters.
Our novel observation of early lottery tickets provide a new aspect of sparse model training; the sparse structures mature early in the training, and they converge to a ﬂat local minimum even if its shape is vastly distant.
A deeper investigation about the sparse structure of the lottery tickets and the mode connectivity is an interesting future research topic.
C E ARLY W INNING T ICKET A LGORITHMS
In this section, we describe the detail algorithms for ﬁnding early winning tickets via one-shot and gradual pruning.