H OW I L EARNED TO S TOP W ORRYING AND L OVE R ETRAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Many Neural Network Pruning approaches consist of several iterative training and pruning steps, seemingly losing a signiﬁcant amount of their performance after pruning and then recovering it in the subsequent retraining phase.
Recent works of Renda et al.
and Le & Hua (2021) demonstrate the signiﬁcance of the learning rate schedule during the retraining phase and propose speciﬁc heuristics for choosing such a schedule for IMP (Han et al., 2015).
We place these ﬁndings in the context of the results of Li et al.
(2020) regarding the training of models within a ﬁxed training budget and demonstrate that, consequently, the retraining phase can be massively shortened using a simple linear learning rate schedule.
Improving on existing retraining approaches, we additionally propose a method to adaptively select the initial value of the linear schedule.
Going a step further, we propose similarly imposing a budget on the initial dense training phase and show that the resulting simple and efﬁcient method is capable of outperforming signiﬁcantly more complex or heavily parameterized state-of-the-art approaches that attempt to sparsify the network during training.
These ﬁndings not only advance our understanding of the retraining phase, but more broadly question the belief that one should aim to avoid the need for retraining and reduce the negative effects of ‘hard’ pruning by incorporating the sparsiﬁcation process into the standard training.
I NTRODUCTION
Modern Neural Network architectures are commonly highly over-parameterized (Zhang et al., 2016), containing millions or even billions of parameters, resulting in both high memory requirements as well as computationally intensive and long training and inference times.
It has been shown however (LeCun et al., 1989; Hassibi & Stork, 1993; Han et al., 2015; Gale et al., 2019; Lin et al., 2020;
Blalock et al., 2020) that modern architectures can be compressed dramatically by pruning , i.e., removing redundant structures such as individual weights, entire neurons or convolutional ﬁlters.
The resulting sparse models require only a fraction of storage and ﬂoating-point operations (FLOPs) for inference, while experiencing little to no degradation in predictive power compared to the dense model.
Although it has been observed that pruning might have a regularizing effect and be beneﬁcial to the generalization capacities (Blalock et al., 2020; Hoeﬂer et al., 2021), a very heavily pruned model will normally be less performant than its dense (or moderately pruned) counterpart.
One approach to pruning consists of removing part of a network’s weights from the model architecture after a standard training process, seemingly losing most of its predictive performance, and then retraining to compensate for that pruning-induced loss.
This can be done either One Shot, that is pruning and retraining only once, or the process of pruning and retraining can be repeated iteratively.
Although dating back to the early work of Janowsky (1989), this approach was most notably proposed by Han et al.
(2015) in the form of I TERATIVE M AGNITUDE P RUNING
(IMP).
In its full iterative form, for example formulated by Renda et al.
, IMP can require the original train time several times over to produce a pruned network, resulting in hundreds of retraining epochs on top of the original training procedure and leading to its reputation for being computationally impractical (Liu et al., 2020;
Ding et al., 2019; Hoeﬂer et al., 2021; Lin et al., 2020; Wortsman et al., 2019).
This as well as the belief that IMP achieves sub-optimal states (Carreira-Perpinán & Idelbayev, 2018; Liu et al., 2020) is one of the motivating factors behind methods that incorporate sparsiﬁcation into the training, which we refer to as pruning-stable following the conventions of Bartoldson et al.
Motivated by recent results of Li et al.
(2020) regarding the training of neural networks under constraints on the number of training iterations, we challenge these commonly held beliefs by rethinking the retraining phase of IMP within the context of Budgeted Training and demonstrate that it can be massively shortened by using a simple linearly decaying learning rate schedule.
We further demonstrate the importance of the learning rate scheme during the retraining phase and improve upon the results of Renda et al.
and Le & Hua (2021) by proposing a simple and efﬁcient approach to also choose the initial value of the learning rate, a problem which has not been previously addressed in the context of pruning.
We also propose likewise imposing a budget on the initial dense training phase of IMP, turning it into a method capable of efﬁciently producing sparse, trained networks without the need for a pretrained model by effectively leveraging a cyclic linear learning rate schedule.
The resulting method is able to outperform signiﬁcantly more complex and heavily parameterized state-of-the-art approaches, that aim to maintain pruning-stability by incorporating the sparsiﬁcation into the training, while using less computational resources.
Contributions.
The major contributions are as follows:
We empirically ﬁnd that the results of Li et al.
(2020) regarding the Budgeted Training of Neural Networks apply to the retraining phase of IMP, providing further context for the results of Renda et al.
and Le & Hua (2021).
Building on this, we ﬁnd that the runtime of IMP can be drastically shortened by using a simple linear learning rate schedule with little to no degradation in model performance.
We propose a novel way to choose the initial value of this linear schedule without the need to tune additional hyperparameters in the form of A DAPTIVE L INEAR L EARNING R ATE R ESTARTING (ALLR).
Our approach takes the impact of pruning as well as the overall retraining time into account, improving upon previously proposed retraining schedules on a variety of learning tasks.
By considering the initial dense training phase as part of the same budgeted training scheme, we derive a simple yet effective method in the form of B UDGETED IMP (BIMP) that can easily outperform most pruning-stable approaches given the same number of iterations to train a network from scratch
We believe that our ﬁndings not only advance the general understanding of the retraining phase, but more broadly question the belief that methods aiming for pruning-stability outperform methods that rely on ‘hard’ pruning and retraining both in terms of the quality of the resulting networks and in terms of the speed at which they are obtained.
We also hope that BIMP can serve as a modular and easily implemented baseline against which future approaches can be realistically compared.
Outline.
Section 2 contains a summary of existing literature and approaches regarding network pruning.
It also contains a reinterpretation of some of these results in the context of Budgeted
Training as well as a technical description of the methods we are proposing.
In Section 3 we will experimentally analyze and verify the claims made in the preceding section.
We conclude this paper with some relevant discussion in Section 4.
P RELIMINARIES AND M ETHODOLOGY
While the sparsiﬁcation of Neural Networks includes a wide variety of approaches, we will focus on the analysis of Model Pruning , i.e., the removal of redundant structures in a Neural Network.
We focus on performing unstructured pruning, that is the removal of individual weights, further also providing experiments for its structured counterpart, where entire groups of elements, such as convolutional ﬁlters, are removed.
We will also focus on approaches that start with a dense network and then either prune the network during training or after training as already discussed in the introduction.
For a full and detailed survey of pruning algorithms we refer the reader to Hoeﬂer et al.
Pruning-unstable methods are exempliﬁed by I
TERATIVE M AGNITUDE P RUNING (IMP) (Han et al., 2015).
In its original form, it ﬁrst employs standard network training, adding a common (cid:96) 2regularization term on the objective, and then removes all weights from the network with magnitude below a certain threshold.
The network at this point commonly loses some or even all of its learned predictive power, so it is then retrained for a ﬁxed number of epochs.
This prune-retrain cycle is usually repeated a number of times; the threshold at every pruning step is determined as the appropriate percentile such that, at the end of given number of iterations, a desired target sparsity is met.
Renda et al.
(2020) suggested the following complete approach: train a network for T epochs and then iteratively prune 20% percent of the remaining weights and retrain for T rt =
T epochs until the desired sparsity is reached.
For a goal sparsity of 98% and T = 200 original training epochs, the algorithm would therefore require 18 prune-retrain-cycles for a massive 3800 total retrain epochs.
2. R ETHINKING RETRAINING AS B UDGETED T RAINING
[Figure]
There has been some recent interest in the learning rate schedule used during retraining.
The original approach by Han et al.
(2015) is commonly referred to as F INE
T UNING (FT): suppose we train for T epochs using the learning rate schedule ( η t ) t ≤ T and retrain for T rt epochs per prune-retrain-cycle, then FT retrains the pruned network using a constant learning rate of η T , i.e., the last learning rate used during the original training.
Renda et al.
(2020) note that the learning rate schedule during retraining can have a dramatic impact on the predictive performance of the pruned network and propose L EARNING
R ATE R EWINDING (LRW), where one retrains the pruned network for T rt epochs using the last T − T rt learning rates η T − T rt +1 , . . .
, η T during each cycle.
Le & Hua (2021) further improved upon these results by proposing S CALED
L EARNING R ATE R ESTARTING (SLR), where the pruned network is retrained using a proportionally identical schedule, i.e., by compressing ( η t ) t ≤ T into the retraining time frame of T rt epochs with a short warm-up phase.
They also introduced C YCLIC L EARNING R ATE
R ESTARTING (CLR) based on the 1-cycle learning rate schedule of Smith & Topin (2017), where the original schedule (commonly a stepped one) is replaced with a cosine based one starting at the same initial learning rate η 1 , likewise including a short warm-up phase.
Figure 1 depicts the aforementioned schedules for a retraining budget of 60 epochs.
The motivation behind LRW was stated to be W EIGHT R EWINDING (WR) (Frankle et al., 2019), suggesting that its success is due to some connection to the Lottery Ticket Hypothesis.
Le & Hua (2021) already gave a more grounded motivation when introducing SLR by noting that its main feature is the “usage of large learning rates”.
1 By proposing CLR and motivating it through the 1-cycle learning rate schedule of Smith & Topin (2017), they also already demonstrated that there is no particular signiﬁcance to basing the learning rate schedule on the one used during the original training.
We think that the results of Li et al.
(2020) regarding the training of Neural Network within a ﬁxed iteration budget provide some relevant further context for the varying success achieved by these methods as well as indications on how one can further improve upon them, in particular when retraining is assumed to be signiﬁcantly shorter than the original training, that is when T rt (cid:28) T .
Li et al.
(2020) study training when a resource budget is given (Budgeted Training), commonly in the form of a ﬁxed number of epochs or iterations that the network will be trained for, instead of following the common assumption that training is executed until asymptotic convergence is achieved to some satisfactory degree.
Speciﬁcally, they empirically determine what learning rate schedules are best suited to achieve the highest possible performance within a given budget.
The two major takeaways from their results are as follow:
Compressing any given learning rate schedule to ﬁt within a speciﬁc budget signiﬁcantly outperforms simply truncating it once the budget is reached.
Li et al.
(2020) refer to this compression as B UDGET -A WARE C ONVERSION (BAC).
This clearly aligns with the ﬁndings of Le & Hua (2021) that SLR outperforms LRW, since SLR is simply the BAC of the original training schedule while LRW is a truncated version of it (albeit truncated ‘from the back’ instead of the front).
Certain learning rate schedules are more suited for a wider variety of budgets than others.
In particular, their results indicate that a cosine based schedule (and to an even stronger degree a linear one) performs well when a tight budget is given.
This provides an explanation for why CLR outperforms SLR when the original learning rate schedule follows more traditional recommendations.
Put succinctly, we claim that retraining should ﬁrst and foremost be considered under the aspect of Budgeted Training and that lessons derived in the latter setting are generally applicable in this context.
Motivated by the ﬁndings of Li et al.
, we therefore propose L INEAR L EARNING
R ATE R ESTARTING (LLR) leveraging a linear learning rate schedule during retraining: LLR linearly decays the learning rate during each retrain cycle from an initial value of η 1 to zero after a short warm-up phase.
This effectively results in a cyclic learning rate schedule when pruning and retraining in the iterative setting, which has previously been found to help generalization (Smith, 2015).
Going one step further, we also propose dynamically adapting the initial value of the retraining schedule by relating it not just to the initial learning rate during the original training but also to the impact of the previous pruning step, resulting in A DAPTIVE L INEAR L EARNING R ATE R ESTARTING (ALLR).
While previous works have focused on the actual schedule of the learning rate during retraining, the initial value has only implicitly been dealt with.
FT chooses the last learning rate value η T , which is typically the smallest.
On the other hand, SLR and CLR rely on the initial value corresponding to the maximum value η 1 of the original schedule, which the authors attribute the success of their methods to.
The initial value of LRW is implicitly chosen in proportion to the retraining time by truncating the original schedule from the back.
Existing works have shown that to ﬁnd minima that generalize well, the learning rate should exhibit a large-step and a small-step retraining regime (Jastrz˛ebski et al., 2017;
Li et al., 2019; You et al., 2019; Leclerc & Madry, 2020).
When choosing the initial value of the retraining schedule, the two characteristics of a prune-retrain cycle have to be taken into account: its length and the impact of pruning.
Given a tight retraining budget it might occur that large initial steps cannot be compensated adequately, while a too small learning rate (possibly over a long retraining) period might be insufﬁcient to recover large pruning-induced performance degradation.
An adaptive way of choosing the initial stepsize must therefore address the following question: how much of an increase in loss do we have to compensate for and do we have sufﬁcient time to properly perform both a large-step and small-step learning rate regime?
To that end, ALLR discounts the initial value η 1 by a factor d ∈ [0 , 1] to account for both the available retraining time (similar to LRW, where the magnitude of the initial learning rate naturally depends on T rt ) and the performance drop induced by pruning.
Since measuring the decrease in train accuracy would require an additional evaluation epoch and is thus undesirable, ALLR achieves this goal by ﬁrst measuring the relative L 2 -norm change in the weights due to pruning, that is after pruning a s ∈ ]0 , 1] fraction of the remaining weights, we compute the normalized distance between the weight vector W and its pruned version W p in the form of
[Equation]
where normalization by √ s ensures that d 1 can actually attain the full range of values in [0 , 1] .
We then determine d 2 = T rt /T to account for the length of the retrain phase and choose d η 1 as the initial learning rate for ALLR where d = max( d 1 , d 2 ) .
This approach effectively interpolates between the recommendations of Renda et al.
and Le & Hua (2021)
based on a computationally cheap proxy.
Appendix C.1 contains several ablation studies to justify our design choices.
We will verify our assumptions by ﬁrst comparing the performance of the various translation schemes that give heuristic guidances for how to choose a learning rate schedule during retraining, namely FT,
LRW, SLR, CLR, LLR and ALLR, against tuned versions of those same learning rate schedules in
Section 3.1 both for the One Shot as well as the iterative setting.
We then study to what degree the retraining phase of IMP can be shortened in the iterative setting compared to the recommendations of
Renda et al.
(2020) when using an appropriate learning rate schedule in Section 3.2.
2. P RUNING - STABILITY : TRYING TO AVOID RETRAINING
Pruning-stable algorithms are deﬁned by their attempt to ﬁnd a well-performing pruned model during the training procedure so that the ultimate ‘hard’ pruning step results in almost no drop in accuracy and the retraining phase becomes superﬂuous.
They do so by inducing a strong implicit bias during some otherwise standard training setup, either by gradual pruning, i.e., extending the pruning mask dynamically, or by employing regularization- and constraint-optimization techniques to learn an almost sparse structure throughout training.
Many methods also rely on some kind of ‘soft’ pruning for this, e.g., by zeroing out weights or strongly pushing them towards zero, but not fully removing them from the network architecture during training.
Let us brieﬂy summarize a variety of methods that have been proposed in this category over the last couple of years: LC (Carreira-Perpinán & Idelbayev, 2018) and GSM (Ding et al., 2019) both employ a modiﬁcation of weight decay and force the k weights with the smallest score more rapidly towards zero, where k is the number of parameters that will eventually be pruned and the score is the parameter magnitude or its product with the loss gradient.
Similarly, DNW (Wortsman et al., 2019) zeroes out the smallest k weights in the forward pass while still using a dense gradient.
CS (Savarese et al., 2020), STR (Kusupati et al., 2020) and DST (Liu et al., 2020) all rely on the creation of additional trainable threshold parameters, which are applied to sparsify the model while being regularly trained alongside the usual weights.
Here, the training objectives are modiﬁed via penalty terms to control the sparsiﬁcation.
GMP (Zhu & Gupta, 2017) follows a tunable pruning schedule which sparsiﬁes the network throughout training by dynamically extending and updating a pruning mask.
We will use a variant of it suggested by Gale et al.
Finally, based on this idea, DPF (Lin et al., 2020) maintains a pruning mask which is extended using the pruning schedule of Zhu &
Gupta (2017), but allows for error compensation by modifying the update rule to use the (stochastic) gradient of the pruned model while updating the dense parameters.
The two most commonly claimed advantages of pruning-stable methods compared to IMP are the following:
They result in a pruned model faster when training from scratch since they avoid the expensive iterative prune-retrain cycles.
Ding et al.
(2019) for example advertise that is there is “no need for time consuming retraining”, Liu et al.
(2020) try to “avoid the expensive pruning and ﬁne-tuning iterations”, Hoeﬂer et al.
(2021) state that sparsifying during training “is usually cheaper than the train-then-sparsify schedule”, Lin et al.
(2020) argue that IMP is “computationally expensive“ and “outperformed by algorithms that explore different sparsity masks instead of a single one”, Wortsman et al.
(2019) try to “train a sparse neural network without retraining or ﬁne-tuning” and Frankle & Carbin (2018)
(in the context of the Lottery Ticket Hypothesis) state that “iterative pruning is computationally intensive, requiring training a network 15 or more times consecutively”.
They produce preferable results either because they avoid ‘hard’ pruning or due to the particular implicit bias they employ.
Liu et al.
(2020) for example state that previous methods suffer from a “failure to properly recover the pruned weights” and Carreira-Perpinán & Idelbayev (2018) argue that learning the pruning set throughout training “helps ﬁnd a better subset and hence prune more weights with no or little loss degradation”.
While many of the previously listed methods perform well and achieve state-of-the-art results, so far little empirical evidence has been given to support that the claimed advantages of pruning-stability have actually been achieved or, more to the point, that the goal of pruning-stability is worth aiming for in the ﬁrst place.
To verify this, we propose B UDGETED IMP (BIMP), where the same lessons we previously derived from Budgeted Training for the retraining phase of IMP are applied to the initial training of the network.
More speciﬁcally, given a budget of T epochs, we simply train a network for some T 0 < T epochs using a linearly decaying learning rate schedule and then apply IMP with
ALLR on the output for the remaining T − T 0 epochs.
The resulting method is capable of obtaining a pruned model from a random initialization within any given budget T while still maintaining all of the key characteristics of IMP, most notably the fact that (1) we ‘hard’ prune and do not allow weights to recover in subsequent steps, (2) we do not impose any particular additional implicit bias besides a common weight decay term during either training or retraining, and (3) we follow a prescribed static training schedule with the exception of adapting
[Table]
the initial learning rate to the impact of the last pruning in the case of ALLR.
This clearly delineates
BIMP from the previously listed methods and allows us to compare them on equal terms by giving all methods the same budget of a total of T epochs, independent of whether they are spent on ‘normal’ training or retraining.
We remark that the implicit biases of many pruning-stable approaches can result in a substantial computational overhead that we are deliberately ignoring here by comparing methods on a per-epoch basis and therefore giving these methods the advantage in the comparison.
However, we include the images-per-second throughput of the individual algorithms, indicating that
BIMP is among the most efﬁcient approaches.
In Section 3.3 we thoroughly compare our proposed approach to previously listed pruning-stable methods in a fair setting by giving all methods the same computational budget in terms of total training epochs.
Finally, let us remark that there has been a signiﬁcant amount of attention on how to select the speciﬁc weights to be pruned.
Ranking weights for pruning based on the magnitude of their current values has established itself as the approach of choice (Lee et al., 2018), where speciﬁc criteria have been proposed that take the particular network architecture into consideration (Zhu & Gupta, 2017;
Gale et al., 2019; Evci et al., 2020; Lee et al., 2020).
We have veriﬁed some of these results in
Appendix C.2 and will stick to the simple global selection criterion used by Han et al.
(2015) for
BIMP.
E XPERIMENTAL R ESULTS
Let us outline the general methodological approach to computational experiments in this section, including datasets, architectures and metrics.
Experiment-speciﬁc details are found in the respective subsections.
We note that, given the surge of interest in pruning, Blalock et al.
(2020) proposed experimental guidelines in the hope of standardizing the experimental setup.
We aim to follow these guidelines whenever possible.
All experiments performed throughout this computational study are based on the PyTorch framework (Paszke et al., 2019), using the original code of the methods whenever possible.
All results and metrics were logged and analyzed using Weights & Biases (Biewald, 2020).
We plan on publicly releasing our implementations and general setup for the sake of reproducibility and will make it available during any submission of this work.
We perform extensive experiments on image recognition datasets such as ImageNet (Russakovsky et al., 2015), CIFAR-10/100 (Krizhevsky et al., 2009), the semantic segmentation tasks COCO (Lin et al., 2014) and CityScapes (Cordts et al., 2016) as well as neural machine translation (NMT) on WMT16 (Bojar et al., 2016).
In particular, we employed ResNets
(He et al., 2015), Wide ResNets (WRN) (Zagoruyko & Komodakis, 2016), VGG (Simonyan & Zisserman, 2014), the transformerbased MaxViT (Tu et al., 2022) architecture, as well as PSPNet (Zhao et al., 2016) and DeepLabV (Chen et al., 2017) in the case of CityScapes and COCO, respectively. For NMT, we used a T transformer (Raffel et al., 2020) available through HuggingFace (Wolf et al., 2020).
Exact parameters can be found in Appendix A, where we also deﬁne what can be considered a ‘standard’ training setup for each setting that we rely on whenever not otherwise speciﬁed.
The focus of our analysis will be the tradeoff between the model sparsity and the ﬁnal test performance, being the accuracy in the case of image classiﬁcation, the common mIoU (mean intersection over union) for segmentation, and the BLEU score (Post, 2018) for NMT.
As a secondary measure, we will also consider the theoretical [Figure] speedup (Blalock et al., 2020) induced by the sparsity, see Appendix A for full details.
We use a validation set of 10% of the training data for hyperparameter selection.
3. L EARNING RATE SCHEDULES DURING RETRAINING
Table 1 contains part of the results regarding the comparison between FT, LRW, SLR, CLR and our proposed approaches LLR and ALLR for ImageNet in the One Shot setting.
First of all, we ﬁnd that retraining after pruning is in fact a Budgeted Training scenario, as the insights for normal dense training (Li et al., 2020) transfer to the retraining case.
This is further observable when comparing the translated schedules to versions of constant, stepped exponential, cosine and linear learning rate schedules, where the initial learning rate after pruning was tuned using a grid search (cf.
Appendix B.1, also containing additional results, exact parameter grids and the results for other datasets).
In general, linear and cosine based schedules clearly outperform the constant and stepped ones, with a slight advantage of LLR over CLR.
However, for short retraining times and for the medium sparsity range, the ﬁxed restarting schedules
CLR and LLR fail to yield results competitive to FT and LRW, since a too large initial learning rate is detrimental given a restricted retraining budget.
ALLR is able to consistently improve upon previous approaches, which becomes especially noticeable in the small retraining budget regime.
We think that ALLR is a suitable drop-in replacement when performing retraining.
We similarly observe the strength of ALLR in Figure 2, depicting the highest achievable test accuracy for each number of total retraining epochs, including both the One Shot as well as iterative pruning case.
Appendix B.1 includes the full results on different tasks and datsets, longer retraining budgets as well as the structured pruning setting, where we remove convolutional ﬁlters based on their respective norm, similar to Li et al.
3. B UDGETING THE RETRAINING PHASE
In this part we will treat the number of retrain epochs per prune-retrain cycle T rt as well as the total amount of such cycles J as tunable hyperparameters for IMP and try to determine the tradeoff between the predictive performance of the ﬁnal pruned network and the total number of retrained epochs J · T rt .
As a baseline performance for a pruned network, we will use the approach suggested by Renda et al.
(2020) as it serves as a good benchmark for the current potential of IMP.
In Figure 3 we present the envelope of the results for ResNet-56 trained on CIFAR-10 with target sparsities of 90%, 95%, and 98%, respectively.
The parameters for the retraining phase were optimized using a grid search over T rt ∈ { 10 , 15 , 20 , ..., 60 } and J ∈ { 1 , 2 , . .
, 6 } using LLR.
We ﬁnd that IMP is capable of achieving what has previously been considered its full potential with signiﬁcantly less than the total number of retraining epochs usually budgeted for its iterative form.
More concretely, for all three levels of sparsity, 90%, 95%, and 98%, IMP meets the baseline laid out by Renda et al.
(2020) after only around 100 epochs of retraining, instead of requiring the 2000, 2800, and 3600 epochs used to establish that baseline, respectively.
In fact, given the superiority of a [Table] linear learning rate schedule over a more commonly used stepped one in the case of budgeted training (Li et al., 2020), LLR clearly continues and exceeds the stepped learning rate schedule baseline.
Additional results and exact parameter grids are included in Appendix B.
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
Given that we have just established that the retraining phase of IMP takes well to enforcing a budget when using an appropriate learning rate schedule and that Li et al.
(2020) already established that ‘normal’ training can be signiﬁcantly shortened through a linear learning rate schedule, it is reasonable to assume that the same holds for the original training phase without strongly impacting the pruning and retraining part and therefore the ultimate product of IMP.
To verify this, we trained ResNet-56 on CIFAR-10 using a linearly decaying learning rate schedule from between 5% up to 100% of 200 epochs, which we consider the ‘full’ training, and then apply IMP with LLR on the resulting network both One Shot and iteratively for target sparsities of 90%, 95%, and 98%.
Figure 6 shows the results for the iterative setting, where we retrain for three cycles ofepochs each.
We can see that IMP takes well to budgeting the initial training period, both in the One Shot and in the iterative setting, with the target sparsity seemingly having little inﬂuence on how much the initial training can be compressed.
3. T HE EFFICACY OF PRUNING - STABILITY
We conclude by comparing the performance of BIMP to pruning-stable approaches.
To that end, we train models on ImageNet, CIFAR-100 and CIFAR-10 and give all methods an equal budget of epochs (200 for CIFAR) to derive a pruned model from a randomly initialized one.
For BIMP, we employ ALLR and treat the initial training length T 0 as well as the number of prune-retrain-cycles as hyperparameters, ensuring that the overall epoch budget T is not exceeded.
The hyperparameters of each of the pruning-stable methods were likewise tuned using manually deﬁned grid searches, resorting to the recommendations of the original publications whenever possible, see Appendix B.1.
For GMP, GSM, DPF, DNW, and LC we give the methods prespeciﬁed sparsity levels, the same as given to BIMP.
Tuning the remaining methods in a way that allows for a fair comparison however is signiﬁcantly more difﬁcult, since none of them allow to clearly specify a desired level of sparsity but instead require tuning additional hyperparameters as part of the grid search.
Despite our best efforts, we were only able to cover part of the desired sparsity range using STR and DST.
For CS in the case of ImageNet, we were unable to tune the hyperparameters.
In addition, we noticed that each of these method can have some variance in the level of sparsity achieved even for ﬁxed hyperparameters, so we list the standard deviation of the ﬁnal sparsity with respect to the random seed initialization.
We note that in the original works, LC and GSM were applied to pretrained models.
To allow a fair comparison, we applied LC and GSM to both randomly initialized as well as pretrained models and chose the best results for each sparsity, giving them a larger budget than the others.
Further, we noticed that some pruning-stable methods can proﬁt from retraining.
For the sake of a fair comparison, we hence retrain all methods excluding BIMP for 10 (30 in the case of CIFAR) epochs using FT and use the accuracy reached after retraining when it exceeds the original one.
All results are averaged over multiple seeds with standard deviation indicated.
Table 2 now reports the ﬁnal test performance, theoretical speedup, and actually achieved sparsity of all methods for CIFAR-10 and ImageNet, where we defer full results to Appendix B.4.
The results show that BIMP is able to outperform many of the pruning-stable methods considered here.
For
ImageNet, only DNW is consistently outperforming BIMP, albeit at the price of needing roughly twice as long for training, cf.
image-per-second throughput.
Surprisingly, despite broad hyperparameter grid search, most methods seem to be in disadvantage compared to BIMP, with DPF being a both efﬁcient and strong competitor.
BIMP obtains these results within the same number of overall training epochs and that we are ignoring both the computational overhead of the implicit bias of some of the more involved methods as well as the additional retraining times required despite their claimed pruning-stability.
In Appendix B.4 we also provide some additional context by analyzing the impact that the length of the initial training period in relation to the retraining length has on the performance of BIMP when a ﬁxed budget its given, distinguishing it from the results presented in Figure 6.
We have also included an ablation study (cf.
Appendix C.3) where we compare BIMP to several modiﬁcations of
GMP not previously suggested in the literature, since GMP is the closest in design to BIMP out of all pruning-stable methods considered here.
Most notably this includes variants of GMP with the same pruning selection scheme used by BIMP as well as both a global and a cyclical linear learning rate schedule.
D ISCUSSION AND O UTLOOK
The learning rate is often considered to be the single most important hyperparameter in Deep Learning which nevertheless still remains poorly understood, certainly from a theoretical perspective but also still from an empirical one.
Our work therefore provides an important building block in which we established that, counter to the often explicitly stated belief that IMP is inefﬁcient, many signiﬁcantly more complex and sometimes strenuously motivated methods are outperformed by perhaps the most basic of approaches when proper care is taken of the learning rate.
Despite providing a strong retraining alternative with ALLR, we emphasize that the main goal of this work is not to suggest yet another acronym and claim that it is the be-all and end-all of network pruning, but to instead (a) hopefully focus the efforts of the community on ﬁrst understanding more basic questions before suggesting convoluted novel methods and (b) to emphasize that IMP can serve as a strong, easily implemented, and modular baseline.
We think the modularity here is of particular importance, as individual aspects can easily be exchanged or modiﬁed, e.g., when, what, how, and how much to prune or how to retrain, in order to formulate rigorous ablation studies.
R EPRODUCIBILITY
Reproducibility is of utmost importance for any comparative computational study such as this.
All experiments were based on the PyTorch framework and use publicly available datasets.
The implementation of the ResNet-56 and ResNet-18 network architecture is based on https: //github.com/JJGO/shrinkbench/blob/master/models/cifar_resnet.py and https://github.com/charlieokonomiyaki/pytorch-resnet18-cifar10/ blob/master/models/resnet.py , respectively, the implementation of the WideResNet network architecture is based on https://github.com/meliketoy/wide-resnet.
pytorch , the implementation of the VGG-16 network architecture is based on https://github.com/jaeho-lee/layer-adaptive-sparsity/blob/main/ tools/models/vgg.py and the implementation of the Resnet-50 network architecture is taken from https://pytorch.org/vision/stable/index.html .
Regarding the pruning methods, the code was taken from the respective publications whenever possible.
Regarding the different variants of magnitude pruning such as ERK or
U NIFORM +, we closely followed the implementation of Lee et al.
(2020) available at https://github.com/jaeho-lee/layer-adaptive-sparsity/ .
For metrics such as the theoretical speedup, we relied on the implementation in the ShrinkBench -framework of Blalock et al.
(2020), see https://github.com/JJGO/shrinkbench .
We also plan on publicly releasing the code used for our implementations and general setup for the sake of reproducibility and will make it available during any submission of this work.
A T ECHNICAL DETAILS AND T RAINING S ETTINGS A. T ECHNICAL DETAILS AND GENERAL TRAINING SETTINGS
We deﬁne pruning-stability , theoretical speedup as well as several pruning selection criteria for
IMP, i.e. different criteria that are used to select weights for pruning.
We will analyze the impact of such criteria under state-of-the-art retraining schedules in Appendix C.2.
Further, Table 3 shows the default training settings used throughout this work.
Deﬁnition A.1 (Bartoldson et al.
Let t pre and t post be the test accuracy before pruning and after pruning the trained model, respectively.
We deﬁne the pruning-stability of a method as
[Equation]
Pruning-stable methods are sparsiﬁcation algorithms that learn a sparse solution throughout training such that ∆ stability ≈ 1 .
For example, methods that perform the forward-pass using an already sparsiﬁed copy of the parameters (e.g. DNW by Wortsman et al., 2019), will have ∆ stability = 1 , since the ‘hard’ pruning step only consists of an application of the present pruning mask, which has no further effect.
Methods that actively drive certain parameter groups towards zero more rapidly (such as Carreira-Perpinán & Idelbayev, 2018; Ding et al., 2019) will have a pruning-stability close to 1 , since the projection of (magnitude) pruning at the end of training will perturb the parameters only slightly.
Crucial to our analysis are the tradeoffs between the model sparsity, the ﬁnal test accuracy and the theoretical speedup induced by the sparsity (Blalock et al., 2020).
Theoretical speedup is a metric measuring the ratio in FLOPs needed for inference comparing the dense and sparse model.
More precisely, let F d be the number of FLOPs the dense model needs for inference, and let similarly be
F s the same number for the pruned model, given some sparsity s .
The theoretical speedup is deﬁned as F d /F s and depends solely on the position of the zero weights within the network and layers, not on the numerical values of non-zero parameters.
While we focused on the model sparsity in the main body of the text, we include plots regarding the theoretical speedup here in the appendix, as suggested by Blalock et al.
IMP in its original form treats all trainable parameters as a single vector and computes a global threshold below which parameters are removed, independent of the layer they belong to.
This simple approach, which we will refer to as G LOBAL , has been subject to criticism for not determining optimal layer-dependent pruning rates and for being inconsistent (Liu et al., 2020).
Fully-connected layers for example have many more parameters than convolutional layers and are therefore much less sensitive to weight removal (Han et al., 2015; Carreira-Perpinán & Idelbayev, 2018).
Further, it has been observed that the position of a layer can play a role in whether that layer is amenable to pruning: often ﬁrst and last layers are claimed to be especially relevant for the classiﬁcation performance (Gale et al., 2019).
On the other hand, in which layers pruning takes place signiﬁcantly impacts the sparsity-induced theoretical speedup (Blalock et al., 2020).
Lastly, the non-negative homogeneity of modern ReLU-based neural network architectures (Neyshabur et al., 2015) would also seem to indicate a certain amount of arbitrariness to this heuristic selection rule, or at least a strong dependence on the network initialization rule and optimizer used, as weights can be rescaled to force it to fully remove all parameters of a layer, destroying the pruned network without having affected the output of the unpruned network.
Determining which weights to remove is hence crucial for successful pruning and several methods have been designed to address this fact.
Zhu & Gupta (2017) introduced the U NIFORM allocation, in which a global sparsity level is enforced by pruning each layer to exactly this sparsity.
Gale et al.
(2019) extend this approach in the form of U NIFORM + by (a) keeping the ﬁrst convolutional layer dense and (b) pruning at most 80% of the connections in the last fully-connected layer.
Evci et al.
(2020) propose a reformulation of the E RD
˝
OS -R ÉNYI KERNEL (ERK) (Mocanu et al., 2018) to take the layer and kernel dimensions into account when determining the layerwise sparsity distribution.
In particular, ERK allocates higher sparsity to layers with more parameters.
Finally, Lee et al.
propose L AYER -A
DAPTIVE M AGNITUDE - BASED P RUNING (LAMP), an approach which takes
[Table]
an (cid:96) 2 -distortion perspective by relaxing the problem of minimizing the output distortion at time of pruning with respect to the worst-case input.
We note that we follow the advice of Evci et al.
and Dettmers & Zettlemoyer (2019) and do not prune biases and batch-normalization parameters, since they only amount to a negligible fraction of the total weights, however keeping them has a very positive impact on the performance of the learned model.
Further, for the computations involving
GMP, we follow the recommendations of Gale et al.
and use the enhanced pruning selection criterion U NIFORM +.
We will compare these approaches in Appendix C.2 with a focus on the impact of the retraining phase.
Since Le & Hua (2021) found that SLR can be used to obtain strong results even when pruning convolutional ﬁlters randomly, i.e., by assigning random importance scores to the ﬁlters instead of using the magnitude criterion or others, we are interested in understanding the importance of the retraining technique when considering different sparsity distributions.
B E XTENDED RESULTS AND COMPLETE TABLES B. L EARNING RATE SCHEDULES DURING RETRAINING
This section contains the complete results regarding the performance of different learning rate schedules for retraining.
For ﬁxed schedules, i.e., FT, LRW, SLR, CLR, LLR and ALLR, only the weight decay parameter is tuned and the best conﬁguration reported.
For the tuned schedules, i.e., constant, stepped (BAC), cosine and linear, we tune the weight decay, the initial value of the learning rate as well as the length of warm-up as follows.
For ResNet-18 and ResNet-56 trained on CIFAR-10, weight decay is tuned using a grid search over 1e-4, 2e-4, and 5e-4 and for the tuned schedules in the lower half of the table the initial value is chosen using a grid search over { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 } , where in the iterative case we use the same value for each cycle, and the warm-up is tuned over either zero or ten percent of the retraining budget.
For WideResNet on
CIFAR-100, weight decay is tuned over 2e-4, and 5e-4, the initial value is chosen using a grid search over { 0 . 0008 , 0 . 004 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 5 } , where we vary the warmup length between zero and ten percent of the retraining budget.
For ResNet-50 on ImageNet, we only present the results for ﬁxed schedules, while keeping the weight decay ﬁxed at 0.0001.
Table 4 and Table 5 show the full results for ResNet-56 trained on CIFAR-10 in the One Shot and iterative setting, respectively.
Table 6, Table 7 and Table 8 show the One Shot results for ResNettrained on CIFAR-10, WideResNet trained on CIFAR-100 and ResNet-50 trained on ImageNet, respectively.
schedules (below) for IMP in the One Shot setting for target sparsity of 50%, 90%, 95%, 99% and a retrain time of 2.5%, 5%, 10%, and 25% of the initial training budget.
The ﬁrst , second , and third best values for the translation schemes are highlighted.
Results are averaged over two seeds with the standard deviation indicated.
[Table] [Table]
[Table] [Table]
[Table]
±0.
CLR 71.96 ±0.
73.30 ±0.
74.24 ±0.
74.38 ±0.
LLR 72.07 ±0.
73.41 ±0.
74.23 ±0.
74.12 ±0.
ALLR 72.96 ±0.
74.02 ±0.
74.71 ±0.
74.43 ±0.
Model sparsity 90%
Budget: 2.22% 5.55% 11.11% 22.22%
FT 56.75 ±0.
61.60 ±0.
64.61 ±0.
65.90 ±0.
LRW 56.75 ±0.
61.61 ±0.
64.60 ±0.
70.41 ±0.
SLR 67.19 ±0.
69.45 ±0.
70.80 ±0.
71.24 ±0.
CLR 68.72 ±0.
70.60 ±0.
71.51 ±0.
71.68 ±0.
LLR 68.90 ±0.
70.48 ±0.
71.53 ±0.
71.45 ±0.
ALLR 69.56 ±0.
71.19 ±0.
71.99 ±0.
71.76 ±0.
Model sparsity 95%
Budget: 2.22% 5.55% 11.11% 22.22%
FT 27.24 ±0.
40.91 ±0.
48.56 ±0.
51.69 ±0.
LRW 27.24 ±0.
40.91 ±0.
48.56 ±0.
62.55 ±0.
SLR 61.42 ±0.
64.60 ±0.
66.31 ±0.
66.93 ±0.
CLR 63.35 ±0.
66.07 ±0.
67.34 ±0. 67.44 ±0.
LLR 63.58 ±0.
66.02 ±0.
67.20 ±0.
67.34 ±0.
ALLR 63.75 ±0.
66.22 ±0.
67.43 ±0.
67.75 ±0.
[Table] [Table]
±0.
77.13 ±0.
77.22 ±0.
LLR 75.82 ±0.
76.57 ±0. 77.22 ±0. 77.21 ±0. ALLR 76.70 ±0.
77.21 ±0.
77.37 ±0.
77.31 ±0.
Model sparsity 85%
Budget: 1% 2.5% 5% 10%
FT 71.49 ±0.
72.97 ±0.
73.61 ±0.
74.14 ±0.
LRW 71.49 ±0.
72.97 ±0.
73.61 ±0.
74.14 ±0.
SLR 74.02 ±0.
75.15 ±0.
75.75 ±0.
75.86 ±0.
CLR 74.85 ±0.
76.11 ±0.
76.55 ±0.
76.59 ±0.
LLR 75.05 ±0.
76.05 ±0.
76.56 ±0.
76.59 ±0.
ALLR 75.79 ±0.
76.55 ±0.
76.85 ±0.
76.74 ±0.
Model sparsity 90%
Budget: 1% 2.5% 5% 10%
FT 60.03 ±1.
64.56 ±0.
66.62 ±0.
68.42 ±0.
LRW 60.03 ±1.
64.56 ±0.
66.62 ±0.
68.42 ±0.
SLR 71.44 ±0.
73.03 ±0.
73.87 ±0.
74.14 ±0.
CLR 72.62 ±0.
74.25 ±0.
74.97 ±0.
75.10 ±0.
LLR 72.82 ±0.
74.27 ±0. [Table] [Table] 59 ±0.
CLR 75.49 ±0.
76.68 ±0.
76.11 ±0.
76.52 ±0.
LLR 75.66 ±0.
76.78 ±0.
76.10 ±0.
76.80 ±0.
ALLR 76.45 ±0.
77.11 ±0.
77.01 ±0.
77.15 ±0.
Model sparsity 85%
Budget: 2 × 1 % 2 × 2 .
5 % 3 × 1 % 3 × 2 .
FT 71.07 ±0.
72.85 ±0.
72.14 ±0.
73.12 ±0.
LRW 71.07 ±0.
72.85 ±0.
72.14 ±0.
73.12 ±0.
SLR 73.62 ±0.
75.05 ±0.
73.95 ±0.
74.85 ±0.
CLR 74.68 ±0.
76.09 ±0.
75.34 ±0.
75.95 ±0.
LLR 74.89 ±0.
76.22 ±0.
75.47 ±0.
76.21 ±0.
ALLR 75.43 ±0.
76.45 ±0.
76.14 ±0.
76.53 ±0.
Model sparsity 90%
Budget: 2 × 1 % 2 × 2 .
5 % 3 × 1 % 3 × 2 .
FT 59.72 ±1.
64.38 ±0.
61.75 ±0.
64.96 ±0.
LRW 59.72 ±1.
64.38 ±0.
61.75 ±0.
64.96 ±0.
SLR 71.04 ±0.
73.04 ±0.
72.12 ±0.
73.50 ±0.
CLR 72.39 ±0.
74.33 ±0.
73.31 ±0.
74.47 ±0.
LLR 72.58 ±0.
74.36 ±0.
73.69 ±0.
74.78 ±0.
ALLR 72.89 ±0.
74.65 ±0.
73.94 ±0.
74.84 ±0.
[Table]
Budget: 3.33% 6.66% 16.66%
FT 62.24 ±0.
62.31 ±0.
62.56 ±0.
LRW 62.24 ±0.
62.31 ±0.
62.75 ±0.
SLR 60.82 ±0.
61.32 ±0.
62.11 ±0.
CLR 61.66 ±0.
62.56 ±0.
62.99 ±0.
LLR 61.62 ±0.
62.64 ±0.
63.11 ±0. ALLR 62.53 ±0. 62.78 ±0. 63.08 ±0.
Model sparsity 80%
Budget: 3.33% 6.66% 16.66%
FT 59.84 ±0.
60.30 ±0.
61.08 ±0.
LRW 59.84 ±0.
60.31 ±0.
61.80 ±0.
SLR 60.04 ±0.
60.91 ±0.
61.69 ±0.
CLR 61.02 ±0.
61.74 ±0.
62.69 ±0.
LLR 61.24 ±0.
61.85 ±0.
62.62 ±0.
ALLR 61.68 ±0.
62.05 ±0.
62.55 ±0.
[Table]
[Table]
57.25 ±0.
57.96 ±0.
57.74 ±0.
58.05 ±0.
LLR 57.37 ±0.
58.06 ±0.
58.18 ±0.
58.49 ±0.
ALLR 57.67 ±0.
58.09 ±0.
58.04 ±0.
58.30 ±0.
Model sparsity 80%
Budget: 0.66% 1.66% 3.33% 5%
FT 49.59 ±0.
51.08 ±0.
52.17 ±0.
52.77 ±0.
LRW 49.59 ±0.
51.08 ±0.
52.17 ±0.
54.34 ±0.
SLR 55.88 ±0.
57.07 ±0.
57.41 ±0.
57.87 ±0.
CLR 56.68 ±0.
57.68 ±0.
57.84 ±0.
58.16 ±0.
LLR 56.78 ±0.
57.53 ±0.
57.95 ±0.
58.31 ±0.
ALLR 56.89 ±0.
57.40 ±0.
57.57 ±0.
58.05 ±0.
Model sparsity 90%
Budget: 0.66% 1.66% 3.33% 5%
FT 32.54 ±0.
35.85 ±0.
39.14 ±0.
41.00 ±0.
LRW 32.54 ±0.
35.85 ±0.
39.14 ±0.
45.57 ±0.
SLR 54.76 ±1.
56.43 ±0.
56.50 ±0.
56.53 ±0.
CLR 55.96 ±0.
56.77 ±0.
56.89 ±0.
57.34 ±0.
[Table]
[Table]
.68 ±0.
CLR 58.18 ±0.
57.07 ±0.
58.05 ±0.
57.93 ±0.
LLR 58.09 ±0.
58.13 ±0.
58.07 ±0.
58.47 ±0.
ALLR 58.28 ±0.
58.28 ±0.
58.34 ±0.
58.41 ±0.
Model sparsity 80%
Budget: 2 × 0 .
66 % 2 × 1 .
66 % 3 × 0 .
66 % 3 × 1 . 66 %
FT 50.05 ±0.
51.25 ±0.
49.71 ±0.
51.22 ±0.
LRW 50.05 ±0.
51.25 ±0.
49.71 ±0.
51.22 ±0.
SLR 56.61 ±0.
56.70 ±0.
56.45 ±0.
55.35 ±1.
CLR 57.36 ±0.
57.01 ±0.
57.43 ±0.
57.49 ±0.
LLR 57.50 ±0.
57.14 ±0.
57.22 ±0.
57.65 ±0.
ALLR 57.64 ±0.
57.93 ±0.
57.61 ±0.
58.04 ±0.
Model sparsity 90%
Budget: 2 × 0 .
66 % 2 × 1 .
66 % 3 × 0 .
66 % 3 × 1 . 66 %
FT 33.09 ±0.
36.22 ±0.
33.70 ±0.
37.69 ±0.
LRW 33.08 ±0.
36.22 ±0.
33.70 ±0.
37.69 ±0.
SLR 55.37 ±1.
55.89 ±0.
55.47 ±0.
54.53 ±0.
CLR 56.44 ±
[Table]
[Table]
[Table]
[Table]
LLR
[Table]
[Table]
[Table]
ALLR
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
FT
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
LLR
[Table]
[Table]
[Table]
ALLR
[Table]
[Table]
[Table]
[Table] [Table] 66.23 ±0. 68.54 ±0.
CLR 67.94 ±0.
70.14 ±0.
68.25 ±0.
70.18 ±0.
LLR 68.08 ±0.
70.17 ±0.
68.41 ±0.
70.38 ±0.
ALLR 68.11 ±0.
70.17 ±0.
68.95 ±0.
71.90 ±0.
Model sparsity
[Table]
Budget:
[Table]
[Table]
[Table]
[Table]
FT 54.
[Table]
[Table] [Table]
[Table]
LRW
[Table] [Table]
[Table] [Table]
[Table]
SLR 63.
[Table]
[Table]
64.59 ±0.
[Table]
CLR 66.07 ±0.
68.78 ±0.
66.71 ±0.
69.00 ±0.
LLR 66.40 ±0.
68.75 ±0.
66.87 ±0.
[Table]
ALLR 66.
[Table] [Table]
67.86 ±0.
70.47 ±0.
[Table]
[Table]
CLR 74.42 ±0. 76.06 ±0. 75.14 ±0. 76.28 ±0.
LLR 74.57 ±0.
76.25 ±0.
75.38 ±0.
76.47 ±0.
ALLR 75.62 ±0.
76.61 ±0.
76.17 ±0.
76.59 ±0.
Model sparsity 40%
Budget: 2 × 1 % 2 × 2 .
5 % 3 × 1 % 3 × 2 .
FT 60.97 ±0.
66.62 ±0.
63.67 ±0.
67.77 ±0.
LRW 60.97 ±0.
66.62 ±0.
63.67 ±0.
67.77 ±0.
SLR 72.48 ±0.
74.51 ±0.
73.50 ±0.
74.66 ±0.
CLR 73.89 ±0.
75.70 ±0.
74.94 ±0.
76.17 ±0.
LLR 74.29 ±0.
75.91 ±0.
75.19 ±0.
76.26 ±0.
ALLR 74.99 ±0.
76.38 ±0.
75.68 ±0.
76.39 ±0.
Model sparsity 45%
Budget: 2 × 1 % 2 × 2 .
5 % 3 × 1 % 3 × 2 .
FT 55.23 ±0.
63.10 ±0.
58.66 ±1.
64.53 ±1.
LRW 55.23 ±0.
63.10 ±0.
58.66 ±1.
64.53 ±1.
SLR 71.81 ±0.
74.01 ±0.
72.85 ±0.
74.41 ±0.
CLR 73.44 ±0.
75.37 ±0.
74.43 ±0.
75.71 ±0.
LLR 73.66 ±0.
75.52 ±0.
74.61 ±0.
75.84 ±0.
ALLR 74.40 ±0.
75.90 ±0.
75.07 ±0.
76.01 ±0.
Model sparsity 50%
Budget: 2 × 1 % 2 × 2 .
5 % 3 × 1 % 3 × 2 .
FT 48.40 ±1.
58.29 ±0.
53.10 ±1.
60.36 ±1.
LRW 48.40 ±1.
58.29 ±0.
53.10 ±1.
60.36 ±1.
SLR 71.08 ±0.
73.30 ±0.
72.30 ±0.
73.88 ±0.
CLR 72.74 ±0.
74.74 ±0.
73.83 ±0.
75.61 ±0.
LLR 73.05 ±0.
75.01 ±0.
74.21 ±0.
75.55 ±0.
ALLR 73.77 ±0.
75.54 ±0.
74.54 ±0.
75.53 ±0.
B. B UDGETING THE RETRAINING PHASE
Similarly to Figure 3 in the main part
[Figure]
Figure 4 and Figure 5 display the envelope when retraining with LLR and ALLR, respectively.
Here, the weight decay values, including those used for the baseline, were individually tuned for each datapoint using a grid search over 1e-4, 2e-4 and 5e-4.
All results are averaged over two seeds with max-min-bands indicated.
Although LLR and ALLR show slightly different behaviour
[Figure]
we note that both reach the performance of the baseline with signiﬁcantly less retraining epochs than required to establish that baseline.
[Figure]
[Figure]
B. B UDGETING THE INITIAL TRAINING
Figure 6 and Figure 7 show the behaviour of One Shot as well as iterative pruning when budgeting the initial training, where retraining is performed with LLR and ALLR, respectively.
Here, the initial training is budgeted between 5% up to 100% of 200 epochs, which we consider the ‘full’ training.
The initial training follows a linearly decaying learning rate schedule which starts from 0.1.
After initial training, IMP is applied One Shot (retraining for 30 epochs) or iteratively (3 cycles ofepochs each).
The individual datapoints are given by the length of the initial training, namely 10, 25, 50, 75, 100, 125, 150, 175 and 200 training epochs.
We keep weight decay ﬁxed at 5e-
[Figure]
[Figure]
B. C OMPARISON TO STATE - OF - THE - ART PRUNING - STABLE METHODS
Table 22 lists all methods that take part in the comparative study between BIMP and state-of-the-art pruning-stable methods (cf.
Section 3.3), where we added IMP as a method of reference when listing the computational overhead when compared to regular (dense) training.
In the following subsections, we list the exact hyperparameter grids used for each pruning-stable method.
B.4.
R ES N ET -
ON CIFARFor each method (including BIMP) we tune weight decay over 1e-4, 5e-4, 1e-3 and 5e-3 and keep momentum ﬁxed at 0.9.
Since momentum plays a crucial part in the justiﬁcation of GSM, we tune it over 0.9, 0.99 and 0.995.
Since the learning rate schedule might need additional tuning, we vary the initial learning rate between 0.05, 0.15, 0.1 and 0.2 for all methods except CS.
The decay of the schedule follows the same pattern as listed in Table 3, where BIMP clearly follows its own cyclic learning rate schedule.
Since CS required the broadest grid, we ﬁxed the learning rate schedule to the one in Table 3.
Otherwise, we used the following grids.
[Table] [Table]
[Table]
GSM
Momentum: { 0 . 9 , 0 .
LC
We only tune the weight decay and, similar to the recommendation by Carreira-Perpinán & Idelbayev (2018), increase it over time as λ 0 · 1 . 1 j , where j is increased over time and λ 0 is the initial weight decay.
For the retraining phase we deactivate weight decay.
DPF
As for GMP, we tune the number of pruning steps, i.e., { 20 , 100 } , and the weight decay.
DNW
We only tune the weight decay, since there are no additional hyperparameters.
CS
As recommended by Savarese et al.
(2020), we ﬁx the temperature β at 300.
Otherwise, we perform the following grid search.
We set the mask initialization s∈ {− 0 .
3 , − 0 .
25 , − 0 .
2 , − 0 .
1 , − 0 .
25 , 0 . 3 } and the (cid:96) 1 penalty λ to { 1e-8, 1e-7 } .
STR
We tune the initial threshold value s init ∈ {− 100 , − 50 , − 5 , − 2 , − 1 , 0 , 5 , 50 , 100 } .
In an extended grid search, we also used weight decays in { 5e-05, 1e-4 } and varied s init ∈ {− 40 , − 30 , − 20 , − 10 } .
DST
We tune the sparsity-controlling regularization parameter α ∈ { 5e-6, 1e-5, 5e-5, 1e-4, 5e-4 } .
In an extended grid search, we used weight decays in { 0, 1e-4 } and tuned α over { 1e-7, 5e-7, 1e-6 } .
B.4.
W IDE R ES N ET ON CIFAR-
For each method (including BIMP) we tune weight decay over 1e-4, 2e-4 and 5e-4 and keep momentum ﬁxed at 0.9.
Since momentum plays a crucial part in the justiﬁcation of GSM, we tune it over 0.9 and 0.95.
We vary the initial learning rate between 0.05, 0.1 and 0.15 for all methods (including CS).
The decay of the schedule follows the same pattern as listed in Table 3, where BIMP clearly follows its own cyclic learning rate schedule.
Otherwise, we used the following grids.
[Table]
[Table]
Equally distributed pruning steps: { 20 , 100 } .
GSM
Momentum: { 0 . 9 , 0 .
LC
We only tune the weight decay and, similar to the recommendation by Carreira-Perpinán & Idelbayev (2018), increase it over time as λ 0 · 1 . 1 j , where j is increased over time and λ 0 is the initial weight decay.
For the retraining phase we deactivate weight decay.
DPF
As for GMP, we tune the number of pruning steps, i.e., { 20 , 100 } , and the weight decay.
DNW
We only tune the weight decay, since there are no additional hyperparameters.
CS
As recommended by Savarese et al.
(2020), we ﬁxed the temperature β at 300, but increased it to 500 upon noticing the pruning instability of CS on this dataset.
Otherwise, we perform the following grid search.
We set the mask initialization s∈ {− 0 .
3 , − 0 .
25 , − 0 .
2 , − 0 .
1 , − 0 .
05 , − 0 .
03 , − 0 .
01 , − 0 .
005 , − 0 .
003 , − 0 .
001 , 0 } and the (cid:96) 1 penalty λ to { 1e-9, 1e-8, 1e-7 } .
STR
We tune the initial threshold value s init ∈ {− 5000 , − 3000 , − 2000 , − 1000 , − 500 } .
In an extended grid search, we also varied s init ∈ {− 200 , − 150 , − 100 , − 80 , − 50 , − 40 , − 25 , − 10 , 0 } .
DST
We tune the sparsity-controlling regularization parameter α ∈ { 5e-6, 1e-5, 5e-5, 1e-4, 5e-4 } .
In an extended grid search, we tuned α over { 1e-6, 3e-6, 8e-6, 3e-5, 8e-5, 2e
[Table]
B.4. R ES N ET -50 ON I MAGE N ET
If not otherwise speciﬁed, we ﬁx the weight decay at 1e-4 and momentum at 0.9.
Since momentum plays a crucial part in the justiﬁcation of GSM, we tune it over 0.9 and 0.95.
We keep the learning rate schedule ﬁxed for all methods cf.
Table 3.
Note that for GSM and LC, we choose the best value training from scratch or using a pretrained model.
In the latter case, we apply the methods for 10,or 40 epochs.
Otherwise, we used the following grids.
[Table] [Table]
[Table]
Equally
[Table] [Table]
Momentum: { 0 . 9 , 0 . 95 } .
Weight decay: { 1e-4, 1e-5 } .
[Table]
Weight decay: { 1e-4, 1e-5 } .
[Table]
Equally distributed pruning steps: { 9 , 18 } .
DNW
Weight decay: { 1e-4, 1e-5 } .
STR
Weight decay: { 1e-5, 2e-5, 3e-5, 4e-5, 1e-4 } .
s init : {− 20 , − 15 , − 13 , − 10 , − 5 , − 4 , − 2 } .
[Table]
Weight decay: { 1e-4, 1e-5 } .
α : { 1e-7, 5e-7, 1e-6, 2e-6, 5e-6, 8e-6, 1e-5, 1e-4 }
[Table]
R ESULTS
Figure 8 and Figure 9 show the accuracy increase by retraining and the pruning stability for CIFARand CIFAR-100, respectively.
Although some methods can beneﬁt from additional retraining, we note that almost all methods are indeed pruning-stable, i.e., the ultimate removal of weights incurs practically no decrease in test accuracy.
Similar to the ﬁgures in the previous section, Figure 10 displays the test accuracy achieved by IMP when budgeting the initial training phase.
However, instead of using a ﬁxed One Shot or iterative procedure, we dynamically set the retraining length to the full amount of training epochs, namely 200, subtracted by the initial training length.
The retraining length is separated into prune-retrain cycles of length 20, each being retrained using LLR.
Hence, every experiment runs for the duration of exactly 200 epochs and this is exactly the setting of BIMP.
We keep the initial learning rate value as well as weight decay ﬁxed at 0.1 and 5e-4, respectively.
The experiment shows that BIMP is relatively robust against varying the initial training length, making it an easily tunable hyperparameter.
In the high sparsity regime, namely for sparsity 99%, we note that shortening the initial training is even beneﬁcial to the ﬁnal test accuracy.
Clearly, shortening the initial training time results in more prune-retrain cycles, which are beneﬁcial in the high sparsity regime.
Table 23 and Table
[Figure]
[Figure] [Figure]
[Table] [Table]
[Table] [Table]
[Table]
C A BLATION STUDIES C. ALLR: T HE INITIAL VALUE OF THE LEARNING RATE SCHEDULE
We analyze the impact of our choices regarding the design of ALLR.
First of all, we justify the usage of the proxy to determine the initial learning rate.
Recall that ALLR discounts the initial value η 1 by a factor d ∈ [0 , 1] to account for both the available retraining time (similar to LRW) and the actual increase in loss induced by pruning.
It does so by choosing d = max( d 1 , d 2 ) , where
[Equation]
measures the relative L 2 -norm change in the weights due to pruning and d 2 = T rt /T accounts for the length of the retrain phase in comparison to the original training length.
The motivation behind choosing these factors lies in handling different retraining scenarios.
When choosing the initial value, two aspect have to be taken into account.
The number of retraining epochs available, T rt , might be very limited.
A large initial learning rate might yield too large oscillations of the loss, from which we cannot recover in a too short retraining timeframe.
To ﬁnd highly generalizing minima, we need both phases: a large-step and a small-step retraining regime (Jastrz˛ebski et al., 2017;
Li et al., 2019; You et al., 2019; Leclerc & Madry, 2020).
If T rt is too small, we do not have enough time to do both.
This is the advantage of LRW:
The magnitude of the initial learning rates depends directly on T rt .
The pruning-induced decrease in accuracy might be very small, depending on the fraction of weights we remove.
Pruning only a small fraction of the weights will most likely have little impact on the loss: in a highly over-parameterized network, removing a small fraction of the weights will not drive the parameters far away from the current (local) optimum.
In that case, convergence is accelerated by performing small learning rate steps
We expect that especially in the higher sparsity regime, where the loss increases dramatically by pruning, a larger initial learning rate is desirable to being able to compensate the drop
in accuracy and approach the optimum faster. In other words, we seek to choose the learning rate to initially be as large as possible (but bounded by the largest learning rate used throughout training), where as large as possible means taking two factors into account: How much increase in loss do we have to compensate and do we have enough time to properly perform both a large-step regime and a small-step regime?
The metric d 2 is an immediate proxy to the duration of retraining phase, motivated by LRW.
Note that we clip d 2 to not become larger than 1.
When we are allowed to retrain at least as long as the initial training, we prefer the initial value of the original training, which is the maximum value in the problem setting.
In any other case, we take a fraction of it.
However, regarding the metric d 1 , we choose to measure the drop in L 2 -norm induced by pruning instead of measuring the actual drop in accuracy, since the latter is only available after performing an entire forward pass on the train dataset.
To investigate whether this replacement is justiﬁed, we compare ALLR to an accuracy drop based variant of it, namely AccALLR .
AccALLR works exactly like ALLR, but we instead choose d 1 as follows:
[Equation]
where Acc (Φ) denotes the train accuracy of model Φ and Φ d , Φ s denote the dense and sparse model, respectively.
We clip d 1 between 0 and 1.
The metric d 1 measures the relative drop in accuracy compared to the dense model.
It hence requires a complete forward pass of the sparse model on the training data before commencing retraining.
For CIFAR-10, Table 25 shows that despite AccALLR having a slight advantage over the ’less-informed’ ALLR, these often lie within the margin of the standard deviation.
The discounting factor of ALLR taking the norm-drop into account is indeed a well-functioning proxy for the pruning-induced drop in accuracy.
Similarly, Table 26 shows the comparison on ImageNet in the iterative setting. Especially for moderate sparsities such as 70% we see that the accuracy is a more precise metric when determining the initial learning rate, however it becomes unprecise at higher sparsities. High distortions of the parameters will lead to the accuracy dropping entirely to that of a random classiﬁer, resulting in the largest possible learning rate being taken as the initial value.
ALLR gives a more robust estimate. [Table]
[Table]
We further show the impact of the two factors d 1 and d 2 of ALLR by considering all four cases of selectively disabling a subset of the metrics, that is, we compare ALLR to ALLRd1 (ALLR only using d 1 ), ALLRd2 (ALLR only using d 2 ) and LLR (which is the same as applying no discounting factor at all) when training ResNet-56 on CIFAR-10.
Table 27 displays the performance of the four different variations when testing against the different scenarios as outlined above, i.e., with low to high sparsity (80-98%) and short to long retraining time (2-20% budget), where we stick to the One Shot setting.
First of all, we observe ALLR consistenly performs best or second best among all variants.
Note that the discounting factor d = max( d 1 , d 2 ) of ALLR is attained at either d 1 or d 2 and ALLR always matches the performance of the better d 1 - or d 2 -disabling variant.
Further, we observe that ALLR behaves exactly as designed and addresses the different retraining scenarios as outlined above.
In the low sparsity, short retraining regime, both ALLR variants disabling one of the two discounting factor perform equally well and are superior than selecting a large initial learning rate as LLR does.
When the retraining time is increased, larger initial learning rates are more suited despite low sparsity, as visible in the marginalized difference to LLR.
With further decreasing sparsity and increasing retraining time, the effect of ALLRd1 would vanish to the beneﬁt of ALLRd2.
On the other hand, in the high sparsity regime, we notice that it is beneﬁcial to start with a high initial learning rate and it is not sufﬁcient to only include the length of the retraining time, as ALLRd2 shows with a difference of ten percent in test accuracy to its competitors.
LRW would behave similarly in this particular case.
Overall, ALLR addresses these issues by accounting both for the increase in loss as well as the available retraining time.
[Table]
C. P RUNING SELECTION CRITERIA
We compare the original G LOBAL pruning criterion of IMP to the previously introduced proposed alternatives.
In the case of ResNet-56 on CIFAR-10 (Figure 11) and VGG-16 on CIFAR-10 (Figure 14), we report the weight decay conﬁg with highest accuracy, where we optimized over the values 1e-4, 5e-4 and 1e-3.
For WideResNet on CIFAR-100 (Figure 12) and ResNet-50 on ImageNet (Figure 13) we relied on a weight decay value of 1e-4 for both architectures. The CIFAR-10 and CIFARresults are averaged over three seeds and max-min-bands are indicated. For ImageNet, the results are based on a single seed.
We tested both FT (Han et al., 2015) and SLR (Le & Hua, 2021) to see whether the learning rate scheme during retraining has any impact on the performance of the pruning selection scheme.
Surprisingly the simple global selection criterion performs at least on par with the best out of all tested methods at any sparsity level for every combination of dataset and architecture tested here when considering the sparsity of the pruned network as the relevant measure.
Using SLR during retraining compresses the results by equalizing performance, but otherwise does not change the overall picture.
We note that the results on CIFAR-100 using FT largely track with those reported by
Lee et al.
(2020), with the exception of the strong performance of the global selection criterion.
Apart from slightly different network architectures, we note that they used signiﬁcantly more retraining epochs, e.g., 100 instead of 30, and that they use AdamW (Loshchilov & Hutter, 2019) instead of
SGD.
Comparing the impact different optimizers can have on the pruning selection schemes seems like a potentially interesting direction for future research.
While the sparsity-vs.-performance tradeoff has certainly been an important part of the justiﬁcation of modiﬁcations to global selection criterion, let us also directly address two further points that are commonly made in this context.
First, the global selection criterion has previously been reported to suffer from a pruning-induced collapse at very high levels of sparsity in certain network architectures that is avoided by other approaches.
This phenomenon has been studied in the pruning before training literature and was coined layer-collapse by Tanaka et al.
(2020), who hypothesize that it can be avoided by using smaller pruning steps since gradient descent restabilizes the network after pruning by following a layer-wise magnitude conservation law.
To verify whether these observations also hold in the pruning after training setting, we trained a VGG-16 network on CIFAR-10, as also done by Lee et al.
(2020), both in the One Shot and in the iterative setting.
The results are reported in
Figure 14 and show that layer collapse is clearly occurring for both FT and SLR for the global selection criterion at sparsity levels above 99% in the One Shot setting, but disappears entirely when pruning iteratively.
This indicates that layer collapse, while a genuine potential issue, can be avoided even using the global selection criterion.
We also remark that SLR needs less prune-retrain-cycles to avoid layer-collapse than FT, possibly indicating that the retraining strategy impacts the speed of restabilization of the network in the hypothesis posed by Tanaka et al.
The second important aspect to consider is that layer-dependent selection criteria are also intended to address the inherent tradeoff not just between the achieved sparsity of the pruned network and its performance, but also the theoretical computational speedup.
Figure 11, Figure 12 and Figureinclude plots highlighting the achieved performance in relation to the theoretical speedup.
The key takeaway here is that for both the ResNet-56 and the WideResNet network architecture, there is overall surprisingly little distinction between all ﬁve tested methods, with Uniform+ and ERK taking the lead and the global selection criterion performing well to average.
For the ResNet-50 architecture however a much more drastic separation occurs, with Uniform performing the best, followed by
Uniform+
and then the global selection criterion.
Overall, the picture is signiﬁcantly less clear.
However, despite its simplicity, the global approach performs on par with respect to managing the accuracy vs. speedup tradeoff, where we observe that for ResNet-50 on ImageNet it even outperforms methods such as LAMP and ERK regarding both objectives, performance and speedup.
[Figure] [Figure]
[Figure] [Figure]
C. C OMPARING BIMP TO GMP
We note that BIMP has a fair number of similarities to GMP (Zhu & Gupta, 2017) and we will therefore seek the direct comparison and run an ablation study.
In particular, both methods effectively prune and retrain (although that terminology is commonly reserved to IMP) at distinct predetermined points during the overall training process.
Let us start by highlighting the particular design decisions by which BIMP and GMP differ:
BIMP ‘hard’ prunes while GMP soft prunes.
In the original implementation of Zhu & Gupta (2017), pruned weights are zeroed out during the forward and backward pass, but the mask is recomputed at the next pruning step, hence allowing for previously pruned weights to recover.
However, it is unclear for example how the momentum buffer of SGD or the memory of optimizers like Adam (Kingma & Ba, 2014) are supposed to take this soft pruning into consideration, but we believe that this aspect overall contributes very little to explain the different performance of GMP and BIMP.
BIMP has a constant pruning frequency after an initial, possibly longer, training period, while GMP has equally distributed pruning points throughout the training.
We likewise believe that this difference contributes very little to differentiate the two methods and in fact think that our results support that the initial training period of BIMP should be thought of as just another cycle, i.e., one can essentially prune at the same time steps as GMP, assuming enough time between the pruning steps.
Both BIMP and GMP rely on magnitude pruning, but we have decided to use the simple G LOBAL selection criterion of Han et al.
(2015) for this paper while GMP was originally suggested using the U NIFORM selection, see Appendix C.2 for a distinction.
The version of GMP included in the main part of this text in fact relies on U NIFORM +, as suggested by Gale et al.
, but since the selection scheme is one of the most important components of any pruning algorithm, we have included a version of GMP using the same scheme as BIMP in this section.
BIMP relies on a cyclic linear learning rate schedule where the cycles obviously coincide with the points at which the network is pruned, while GMP can use any kind of learning rate scheme that would commonly be employed for that kind of architecture and data set, i.e., normally not a cyclic one.
Zhu & Gupta
2017) note that GMP can be quite sensitive to the learning rate.
In the main body of the text we have relied on a common stepped learning rate schedule.
Given the importance of the learning rate schedule, we have also tested a version of GMP using both a global linearly decaying schedule, as suggested by Li et al.
2020), as well as the same cyclic learning rate schedule we have suggested for BIMP, i.e., the cycles are chosen to exactly end at the next pruning step.
We have relied on the simple exponential pruning schedule suggested by Renda et al.
) for BIMP while GMP relies on a particular schedule deﬁned by a cubic polynomial that effectively leads to pruning larger amounts initially and progressively smaller amounts later in training when compared to BIMP.
While we think that the pruning schedule probably has a signiﬁcant impact on the performance of the pruning method and can possibly interact with the learning schedule in particularly interesting ways, we have so far not attempted exchanging the schedule in BIMP for either the one GMP or a novel one
In Table 28 and Table 29 we have included the previously mentioned modiﬁcations of GMP and compared them to BIMP with LLR and ALLR, where all versions of GMP use the global selection criterion also used by BIMP.
We use weight decay values of 1e-4 and 5e-4, tune the initial value of the learning rate over 0.05, 0.1, 0.15, 0.2 and further vary the amount of pruning steps for GMP between 20 and 40, while we vary the initial training length for BIMP between 20, 60 and 100.
All results are averaged over two seeds with standard deviations indicated.
We note that the exchanging the selection criterion has the most drastic impact and closes most of the gap to BIMP in CIFAR-10 but still falling short for CIFAR-100.
The impact of the learning rate schedule is more nuanced: using schedule that linearly decays throughout the training gives a slight increase in performance, as suggested by Li et al.
but while the cyclic schedule falls short for
CIFAR-10 it considerably boosts the performance of GMP for CIFAR-100.
Overall we think that there is a fair amount of nuance here that deserves further exploration and will probably require a more diverse testbed to draw any deﬁnitive conclusions.
[Table] [Table]