D YNAMIC M ODEL P RUNING WITH F EEDBACK
A BSTRACT
Deep neural networks often have millions of parameters.
This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference.
We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance).
We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models.
Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.
I NTRODUCTION
Highly overparametrized deep neural networks show impressive results on machine learning tasks.
However, with the increase in model size comes also the demand for memory and computer power at inference stage—two resources that are scarcely available on low-end devices.
Pruning techniques have been successfully applied to remove a signiﬁcant fraction of the network weights while preserving test accuracy attained by dense models.
In some cases, the generalization of compressed networks has even been found to be better than with full models (Han et al., 2015; 2017; Mocanu et al., 2018).
The sparsity of a network is the number of weights that are identically zero, and can be obtained by applying a sparsity mask on the weights.
There are several different approaches to ﬁnd sparse models.
For instance, one-shot pruning strategies ﬁnd a suitable sparsity mask by inspecting the weights of a pretrained network (Mozer & Smolensky, 1989; LeCun et al., 1990; Han et al., 2017).
While these algorithms achieve a substantial size reduction of the network with little degradation in accuracy, they are computationally expensive (training and reﬁnement on the dense model), and they are outperformed by algorithms that explore different sparsity masks instead of a single one.
In dynamic pruning methods, the sparsity mask is readjusted during training according to different criteria (Mostafa & Wang, 2019; Mocanu et al., 2018).
However, these methods require ﬁne-tuning of many hyperparameters.
We propose a new pruning approach to obtain sparse neural networks with state-of-the-art test accuracy.
Our compression scheme uses a new saliency criterion that identiﬁes important weights in the network throughout training to propose candidate masks.
As a key feature, our algorithm not only evolves the pruned sparse model alone, but jointly also a (closely related) dense model that is used in a natural way to correct for pruning errors during training.
This results in better generalization properties on a wide variety of tasks, since the simplicity of the scheme allows us further to study it from a theoretical point of view, and to provide further insights and interpretation.
We do not require time sparsity low med high t
Pruning before training
Pruning after training
Dynamic pruning during training
Incremental pruning during training
Scheme
Gradient
Pruning
Comments
Pruning before training (cid:101) g
( (cid:101) w t ) (on pruned model) once at the beginning + reaches local minima mask m 0 ﬁxed throughout - ﬁnding suitable masks is challenging
Pruning after training g ( w t ) (on full model) once at the end - does not reach local minima mask m T ﬁxed at the end - ﬁne-tuning of pruned model required
Incremental p. during training (cid:101) g
( (cid:101) w t ) (on partially pruned m.) several times during training + adaptive incrementally, m t +1 ≤ m t - cannot recover from premature pruning
Dynamic p. during training g ( (cid:101) w t ) (on pruned model) dynamically adapting mask m t
+ adaptive every few iterations + can recover from premature pruning tuning of additional hyperparameters, and no retraining of the sparse model is needed (though can further improve performance).
Contributions.
• A novel dynamic pruning scheme, that incorporates an error feedback in a natural way
Sec.
and ﬁnds a trained sparse model in one training pass.
Sec.
• We demonstrate state-of-the-art performance (in accuracy and sparsity),
Sec.
ourperforming all previously proposed pruning schemes
Sec.
• We complement our results by an ablation study that provides further insights.
Sec.
and convergence analysis for convex and non-convex objectives
Sec.
R ELATED WORK
Previous works on obtaining pruned networks can (loosely) be divided into three main categories.
Pruning after training.
Training approaches to obtain sparse networks usually include a three stage pipeline—training of a dense model, one-shot pruning and ﬁne-tuning—e.g., (Han et al., 2015).
Their results (i.e., moderate sparsity level with minor quality loss) made them the standard method for network
Pruning during training.
Zhu & Gupta (2017) propose the use of magnitude-based pruning and to gradually increase the sparsity ratio while training the model from scratch.
A pruning schedule determines when the new masks are computed (extending and simplifying (Narang et al., 2017)).
He et al.
(2018) (SFP) prune entire ﬁlters of the model at the end of each epoch, but allow the pruned ﬁlters to be updated when training the model.
Deep Rewiring (DeepR) (Bellec et al., 2018) allows for even more adaptivity by performing pruning and regrowth decisions periodically.
This approach is computationally expensive and challenging to apply to large networks and datasets.
Sparse evolutionary training (SET) (Mocanu et al., 2018)
simpliﬁes prune–regrowth cycles by using heuristics for random growth at the end of each training epoch and NeST (Dai et al., 2019) by inspecting gradient magnitudes.
Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang, 2019) implements a prune– redistribute–regrowth cycle where target sparsity levels are redistributed among layers, based on loss gradients (in contrast to SET, which uses ﬁxed, manually conﬁgured, sparsity levels).
Sparse
Momentum (SM) (Dettmers & Zettlemoyer, 2019) follows the same cycle but instead using the mean momentum magnitude of each layer during the redistribute phase.
SM outperforms DSR on ImageNet for unstructured pruning by a small margin but has no performance difference on
CIFAR experiments.
Our approach also falls in the dynamic category but we use error compensation mechanisms instead of hand crafted redistribute–regrowth cycles.
Pruning before training.
Recently—spurred by the lottery ticket hypothesis (LT) (Frankle & Carbin, 2019)—methods which try to ﬁnd a sparse mask that can be trained from scratch have attracted increased interest.
For instance, Lee et al.
(2019) propose SNIP to ﬁnd a pruning mask by inspecting connection sensitivities and identifying structurally important connections in the network for a given task.
Pruning is applied at initialization, and the sparsity mask remains ﬁxed throughout training.
[Figure]
Note that Frankle & Carbin (2019); Frankle et al.
(2019) do not propose an efﬁcient pruning scheme to ﬁnd the mask, instead they rely on iterative pruning, repeated for several full training passes.
Further Approaches.
Srinivas et al.
; Louizos et al.
(2018) learn gating variables (e.g. through (cid:96) 0 regularization) that minimize the number of nonzero weights, recent parallel work studies ﬁlter pruning for pre-trained models (You et al., 2019).
Gal et al.
; Neklyudov et al.
Molchanov et al.
(2017) prune from Bayesian perspectives to learn dropout probabilities during training to prune and sparsify networks as dropout weight probabilities reach 1.
Gale et al.
(2019) extensively study recent unstructured pruning methods on large-scale learning tasks, and ﬁnd that complex techniques (Molchanov et al., 2017; Louizos et al., 2018) perform inconsistently.
Simple magnitude pruning approaches achieve comparable or better results (Zhu & Gupta, 2017).
M ETHOD
We consider the training of a non-convex loss function f : R d → R .
We assume for a weight vector w ∈ R d to have access to a stochastic gradient g ( w ) ∈
R
d such that E [ g ( w )]
= ∇ f ( w ) .
This corresponds to the standard machine learning setting with g ( w ) representing a (mini-batch) gradient of one (or several) components of the loss function.
Stochastic Gradient Descent (SGD) computes a sequence of iterates by the update rule
[Equation]
for some learning rate γ t .
To obtain a sparse model, a general approach is to prune some of the weights of w t , i.e., to set them to zero.
Such pruning can be implemented by applying a mask m ∈ { 0 , 1 } d to the weights, resulting in a sparse model (cid:101) w t := m (cid:12) w t , where (cid:12) denotes the entry-wise (Hadamard) product.
The mask could potentially depend on the weights w t (e.g., smallest magnitude pruning), or depend on t (e.g., the sparsity is incremented over time).
Before we introduce our proposed dynamic pruning scheme, we formalize the three main existing types of pruning methodologies (summarized in Figure 1).
These approaches differ in the way the mask is computed, and the moment when it is applied.
Pruning before training.
A mask m 0 (depending on e.g. the initialization w 0 or the network architecture of f ) is applied and (SGD) is used for training on the resulting subnetwork (cid:101) f ( w ) :
= f
( m 0 (cid:12) w ) with the advantage that only pruned weights need to be stored and updated 2 , and that by training with
SGD a local minimum of the subnetwork (cid:101) f
(but not of f —the original training target)
can be reached.
In practice however, it remains a challenge to efﬁciently determine a good mask m 0 and a wrongly chosen mask at the beginning strongly impacts the performance.
broad themes loosely, with slight variations in detail.
For the sake of clarity we omit a too technical and detailed discussion here.
When training on (cid:101) f ( w ) , it sufﬁces to access stochastic gradients of (cid:101)
f
( w ) , denoted by (cid:101) g ( w ) , which can potentially be cheaper be computed than by naively applying the mask to g ( w ) (note (cid:101) g ( w )
= m 0 (cid:12) g ( w ) ). Pruning after training (one-shot pruning).
A dense model is trained, and pruning is applied to the trained model w T .
As the pruned model (cid:101) w T = m T (cid:12) w T is very likely not at a local optimum of f , ﬁne-tuning (retraining with the ﬁxed mask m T ) is necessary to improve performance.
Pruning during training (incremental and dynamic pruning).
Dynamic schemes change the mask m t every (few) iterations based on observations during training (i.e. by observing the weights and stochastic gradients).
Incremental schemes monotonically increase the sparsity pattern, fully dynamic schemes can also reactivate previously pruned weights.
In contrast to previous dynamic schemes that relied on elaborated heuristics to adapt the mask m t , we propose a simpler approach:
Dynamic pruning with feedback (DPF, Algorithm 1).
Our scheme evaluates a stochastic gradient at the pruned model (cid:101) w t = m t (cid:12) w t and applies it to the (simultaneously maintained) dense model w t :
[Equation]
Applying the gradient to the full model allows to recover from “errors”, i.e. prematurely masking out important weights: when the accumulated gradient updates from the following steps drastically change a speciﬁc weight, it can become activated again (in contrast to incremental pruning approaches that have to stick to sub-optimal decisions).
For illustration, observe that (DPF) can equivalently be written as
[Equation]
where e t := (cid:101) w t − w t is the error produced by the compression.
This provides a different intuition of the behavior of (DPF), and connects it with the concept of error-feedback (Stich et al., 2018; Karimireddy et al., 2019).
3 We illustrate this principle in Figure 2 and give detailed pseudocode and further implementation details in Appendix A.1.
The DPF scheme can also be seen as an instance of a more general class of schemes that apply (arbitrary) perturbed gradient updates to the dense model.
For instance straight-through gradient estimators (Bengio et al., 2013) that are used to empirically simplify the backpropagation can be seen as such perturbations.
Our stronger assumptions on the structure of the perturbation allow to derive non-asymptotic convergence rates in the next section, though our analysis could also be extended to the setting in (Yin et al., 2019) if the perturbations can be bounded.
C ONVERGENCE A NALYSIS
We now present convergence guarantees for (DPF).
For the purposes of deriving theoretical guarantees, we assume that the training objective is smooth, that is (
cid:107)∇ f ( w ) − ∇ f ( v ) (cid:107) ≤ L (cid:107) w − v (cid:107) , ∀ w , v ∈ R d , for a constant L > 0 , and that the stochastic gradients are bounded E (cid:107) g
( (cid:101) w t ) (cid:107) 2 ≤ Gfor every pruned model (cid:101) w t = m t ( w t ) (cid:12) w t .
The quality of this pruning is deﬁned as the parameter δ t ∈ [0 , 1] such that
[Equation]
Pruning without information loss corresponds to (cid:101) w t = w t , i.e., δ t = 0 , and in general δ t ≤ 1 .
Convergence on Convex functions.
We ﬁrst consider the case when f is in addition µ -strongly convex, that is (cid:104)∇ f ( v ) , w − v (cid:105) ≤ f ( w ) − f ( v ) − µ 2 (cid:107) w − v (cid:107) 2 , ∀ w , v ∈ R d .
While it is clear that this assumption does not apply to neural networks, it eases the presentation as strongly convex functions have a unique (global) minimizer w (cid:63) :=
arg min w ∈ R d f ( w ) .
Theorem 4.1.
Let f be µ -strongly convex and learning rates given as γ t = 4 µ ( t +2) .
Then for a randomly chosen pruned model (cid:101) u of the iterates { (cid:101) w 0 , . .
, (cid:101) w T } of DPF , concretely (cid:101) u
= (cid:101) w t with probability p t = 2( t +1)
( T +1)
( T +2) , it holds that—in expectation over the stochasticity and the selection of (cid:101) u :
[Equation]
The rightmost term in (2) measures the average quality of the pruning.
However, unless δ t → 0 or (cid:107) w t (cid:107) → 0 for t → ∞ , the error term never completely vanishes, meaning that the method converges only to a neighborhood of the optimal solution (this not only holds for the pruned model, but also for the jointly maintained dense model, as we will show in the appendix).
This behavior is expected, as the global optimal model w (cid:63) might be dense and cannot be approximated well by a sparse model.
For one-shot methods that only prune the ﬁnal (SGD) iterate w T at the end, we have instead:
[Equation]
as we show in the appendix.
First, we see from this expression that the estimate is very sensitive to δ T and w T , i.e. the quality of the pruning the ﬁnal model.
This could be better or worse than the average of the pruning quality of all iterates.
Moreover, one looses also a factor of the condition number Lµ in the asymptotically decreasing term, compared to (2).
This is due to the fact that standard convergence analysis only achieves optimal rates for an average of the iterates (but not the last one).
This shows a slight theoretical advantage of DPF over rounding at the end.
Convergence on Non-Convex Functions to Stationary Points.
Secondly, we consider the case when f is a non-convex function and show convergence (to a neighborhood) of a stationary point.
Theorem 4.2.
Let learning rate be given as γ = c √ T , for c = (cid:113) f
[Equation]
w 0 ) − f ( w (cid:63) ) LG.
Then for pruned model (cid:101) u chosen uniformly at random from the iterates { (cid:101) w 0 , . . .
, (cid:101) w T } of DPF , concretely (cid:101) u := (cid:101) w t with probability p t
=
T
+1 , it holds—in expectation over the stochasticity and the selection of
[Equation]
u
[Equation]
Extension to Other Compression Schemes.
So far we put our focus on simple mask pruning schemes to achieve high model sparsity.
However, the pruning scheme in Algorithm 1 could be replaced by an arbitrary compressor C : R d → R d , i.e., (cid:101) w t = C ( w t ) .
Our analysis extends to compressors as e.g. deﬁned in (Karimireddy et al., 2019; Stich & Karimireddy, 2019), whose quality is also measured in terms of the δ t parameters as in (1).
For example, if our objective was not to obtain a sparse model, but to produce a quantized neural network where inference could be computed faster on low-precision numbers, then we could deﬁne C as a quantized compressor.
One variant of this approach is implemented in the Binary Connect algorithm (BC)
(Courbariaux et al., 2015) with prominent results, see also (Li et al., 2017) for further insights and discussion.
E XPERIMENTS
We evaluated DPF together with its competitors on a wide range of neural architectures and sparsity levels.
DPF exhibits consistent and noticeable performance beneﬁts over its competitors.
5. E XPERIMENTAL SETUP
Datasets.
We evaluated DPF on two image classiﬁcation benchmarks: (1) CIFAR-10 (Krizhevsky & Hinton, 2009)
(50K/10K training/test samples with 10 classes), and (2) ImageNet (Russakovsky et al., 2015)
(1.28M/50K training/validation samples with 1000 classes).
We adopted the standard data augmentation and preprocessing scheme from He et al.
(2016a)
; Huang et al.
(2016); for further details refer to Appendix A.2.
Models.
Following the common experimental setting in related work on network pruning (Liu et al., 2019; Gale et al., 2019; Dettmers & Zettlemoyer, 2019; Mostafa & Wang, 2019), our main experiments focus on ResNet (He et al., 2016a) and WideResNet (Zagoruyko & Komodakis, 2016).
However, DPF can be effectively extended to other neural architectures, e.g., VGG (Simonyan &
Zisserman, 2015), DenseNet (Huang et al., 2017).
We followed the common deﬁnition in He et al.
(2016a); Zagoruyko & Komodakis (2016) and used ResNet-
a , WideResNet- a - b to represent neural network with a layers and width factor b .
Baselines.
We considered the state-of-the-art model compression methods presented in the table below as our strongest competitors.
We omit the comparison to other dynamic reparameterization methods, as DSR can outperform DeepR (Bellec et al., 2018) and SET (Mocanu et al., 2018) by a noticeable margin (Mostafa & Wang, 2019).
Implementation of DPF.
Compared to other dynamic reparameterization methods (e.g. DSR and
SM) that introduced many extra hyperparameters, our method has trivial hyperparameter tuning
[Table]
overhead.
We perform pruning across all neural network layers (no layer-wise pruning) using magnitude-based unstructured weight pruning (inherited from Han et al.
We found the best preformance when updating the mask every 16 iterations (see also Table 11) and we keep this value ﬁxed for all experiments (independent of the architecture or task)
[Table]
Unlike our competitors that may ignore some layers (e
[Table]
g. the ﬁrst convolution and downsampling layers in DSR)
[Table]
we applied DPF (as well as the One-shot P+FT and Incremental baselines) to all convolutional layers while keeping the last fully-connected layer 4 , biases and batch normalization layers dense.
Lastly, our algorithm gradually increases the sparsity s t of the mask from 0 to the desired sparsity using the same scheduling as in (Zhu & Gupta, 2017); see Appendix A.2.
Training schedules.
For all competitors, we adapted their open-sourced code and applied a consistent (and standard) training scheme over different methods to ensure a fair comparison.
Following the standard training setup for CIFAR-10, we trained ResNet- a for 300 epochs and decayed the learning rate by 10 when accessing 50% and 75% of the total training samples (He et al., 2016a; Huang et al., 2017); and we trained WideResNet- a - b as Zagoruyko & Komodakis (2016) for 200 epochs and decayed the learning rate by 5 when accessing 30% , 60% and 80% of the total training samples.
For ImageNet training, we used the training scheme in (Goyal et al., 2017) for 90 epochs and decayed learning rate by 10 at 30 , 60 , 80 epochs.
For all datasets and models, we used mini-batch SGD with Nesterov momentum (factor 0 . 9 ) with ﬁne-tuned learning rate for DPF.
We reused the tuned (or recommended) hyperparameters for our baselines (DSR and SM), and ﬁne-tuned the optimizer and learning rate for One-shot P+FT, Incremental and SNIP.
The mini-batch size is ﬁxed to 128 for CIFAR-10 and 1024 for ImageNet regardless of datasets, models and methods.
5. E XPERIMENT R ESULTS
CIFAR-10.
Figure 3 shows a comparison of different methods for WideResNet-28-2.
For low sparsity level (e.g. 50%), DPF outperforms even the dense baseline, which is in line with regularization properties of network pruning.
Furthermore, DPF can prune the model up to a very high level (e.g.
99%), and still exhibit viable performance.
This observation is also present in Table 1, where the results of training different state-of-the-art DNN architectures with higher sparsities are depicted.
DPF shows reasonable performance even with extremely high sparsity level on large models (e.g.
WideResNet-28-8 with sparsity ratio 99.9%), while other methods either suffer from signiﬁcant quality loss or even fail to converge.
Because simple model pruning techniques sometimes show better performance than complex techniques (Gale et al., 2019), we further consider these simple models in Table 2.
While DPF outperforms them in almost all settings, it faces difﬁculties pruning smaller models to extremely high sparsity ratios (e.g. ResNet-20 with sparsity ratio 95% )
.This seems however to be an artifact of ﬁne-tuning, as DPF with extra ﬁne-tuning convincingly outperforms all other methods regardless of the network size.
This comes to no surprise as schemes like One-shot P+FT and Incremental do not beneﬁt from extra ﬁne-tuning, since it is already incorporated in their training procedure and they might become stuck in local minima.
On the other hand, dynamic pruning methods, and in particular DPF, work on a different paradigm, and can still heavily beneﬁt from ﬁne-tuning.
Figure 13 (in Appendix A.3.4) depicts another interesting property of DPF.
When we search for a subnetwork with a (small) predeﬁned number of parameters for a ﬁxed task, it is much better to run
Besides that a large fraction of the mask elements already converge during training (see e.g. Figure 4 below), not all mask elements converge.
Thus DPF can still beneﬁt from ﬁne-tuning on the ﬁxed sparsity mask.
[Figure]
The results we presented for each model consider some reasonable pruning ratios (we prune more aggressively for deeper and wider neural networks)
[Table]
and readers can refer to Table 4 (in Appendix A.3.1) for a complete overview.
[Table]
DPF on a large model (e.g. WideResNet-28-8) than on a smaller one (e.g. WideResNet-28-2).
That is, DPF performs structural exploration more efﬁciently in larger parametric spaces.
Note that One-shot P+FT prunes the dense model and uses extra ﬁne-tuning itself.
The Dense, Incremental and DPF train with the same number of epochs from scratch.
The extra ﬁne-tuning procedure considers the model checkpoint at the end of the normal training, uses the same number of training epochs (60 epochs in our case) with tuned optimizer and learning rate.
Detailed hyperparameters tuning procedure refers to Appendix A.2.
[Table]
ImageNet.
We compared DPF to other dynamic reparameterization methods as well as the strong
Incremental baseline in Table 3.
For both sparsity levels ( 80% and 90% ), DPF shows a signiﬁcant improvement of top-1 test accuracy with fewer or equal number of parameters
[Table] [Table]
D ISCUSSION
Besides the theoretical guarantees, a straightforward beneﬁt of DPF over one-shot pruning in practice is its ﬁne-tuning free training process.
Figure 12 in the appendix (Section A.3.3) demonstrates the trivial computational overhead (considering the dynamic reparameterization cost) of involving DPF to train the model from scratch.
Small number of hyperparameters compared to other dynamic reparameterization methods (e.g. DSR and SM) is another advantage of DPF and Figure 11 further studies how different setups of DPF impact the ﬁnal performance.
Notice also that for DPF, inference is done only at sparse models, an aspect that could be leveraged for more efﬁcient computations.
Empirical difference between one-shot pruning and DPF.
From the Figure 2 one can see that
DPF tends to oscillate among several local minima, whereas one-shot pruning, even with ﬁne tuning, converges to a single solution, which is not necessarily close to the optimum.
We believe that the wider exploration of DPF helps to ﬁnd a better local minima (which can be even further improved by ﬁne-tuning, as shown in Table 2).
We empirically analyzed how drastically the mask changes between each reparameterization step, and how likely it is for some pruned weight to become non-zero in the later stages of training.
Figure 4 shows at what stage of the training each element of the ﬁnal mask becomes ﬁxed.
For each epoch, we report how many mask elements were ﬂipped starting from this epoch.
As an example, we see that for sparsity ratio 95%, after epoch 157 (i.e. for 43 epochs left), only 5% of the mask elements were changing.
This suggests that, except for a small percentage of weights that keep oscillating, the mask has converged early in the training.
In the ﬁnal epochs, the algorithm keeps improving accuracy, but the masks are only being ﬁne-tuned.
A similar mask convergence behavior can be found in Appendix (Figure 7) for training ResNet-20 on CIFAR-
[Figure]
DPF does not ﬁnd a lottery ticket.
The LT hypothesis (Frankle & Carbin, 2019) conjectures that for every desired sparsity level there exists a sparse submodel that can be trained to the same or better performance as the dense model.
In Figure 5 we show that the mask found by DPF is not a LT, i.e., training the obtained sparse model from scratch does not recover the same performance.
The (expensive) procedure proposed in Frankle & Carbin (2019); Frankle et al.
(2019) ﬁnds different masks and achieves the same performance as DPF for mild sparsity levels, but DPF is much better for extremely sparse models (99% sparsity).
[Figure]
Extension to structured pruning.
The current state-of-the-art dynamic reparameterization methods only consider unstructured weight pruning.
Structured ﬁlter pruning 6 is either ignored (Bellec et al., 2018; Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019) or shown to be challenging (Mostafa & Wang, 2019) even for the CIFAR dataset.
In Figure 6 below we presented some preliminary results on CIFAR-10 to show that our DPF can also be applied to structured ﬁlter pruning schemes.
DPF outperforms the current ﬁlter-norm based state-of-the-art method for structured pruning (e
[Figure]
g.
SFP (He et al., 2018)) by a noticeable margin.
Figure 16 in Appendix A.4.3 displays the transition procedure of the sparsity pattern (of different layers) for WideResNet-28-2 with different sparsity levels.
DPF can be seen as a particular neural architecture search method, as it gradually learns to prune entire layers under the guidance of the feedback signal.
We followed the common experimental setup as mentioned in Section 5 with (cid:96) 2 norm based ﬁlter selection criteria for structured pruning extension on CIFAR-10.
We do believe a better ﬁlter selection scheme (Ye et al., 2018;
He et al., 2019; Lym et al., 2019) could further improve the results but we leave this exploration for the future work.
[Figure]
The reported are over
A CKNOWLEDGEMENTS
We acknowledge funding from SNSF grant 200021_175796, as well as a Google Focused Research
Award.
A. I MPLEMENTATION D ETAILS
We implemented our DPF in PyTorch (Paszke et al., 2017).
All experiments were run on NVIDIA
Tesla V100 GPUs.
Sparse tensors in our implementation are respresented as the dense tenors multiplied by the corresponding binary masks.
Datasets We evaluate all methods on the following standard image classiﬁcation tasks:
• Image classiﬁcation for CIFAR-10 (Krizhevsky & Hinton, 2009).
Dataset consists of a training set of 50 K and a test set of 10 K color images of 32 × 32 pixels, as well as 10 target classes.
We adopt the standard data augmentation and preprocessing scheme (He et al., 2016a; Huang et al., 2016).
• Image classiﬁcation for ImageNet (Russakovsky et al., 2015).
The ILSVRC 2012 classiﬁcation dataset consists of 1 . 28 million images for training, and 50 K for validation, with 1 K target classes.
We use ImageNet-1k (Deng et al., 2009) and adopt the same data preprocessing and augmentation scheme as in He et al.
(2016a;b); Simonyan & Zisserman (2015).
Gradual Pruning Scheme For Incremental baseline, we tuned their automated gradual pruning scheme s t =
s f
+ ( s i − s f ) (cid:0) 1 − t − t 0 n ∆ t (cid:1) 3 to gradually adjust the pruning sparsity ratio s t for t ∈ { t 0 , . . .
, t 0 + n ∆ t } .
That is, in our setup, we increased from an initial sparsity ratio s i = 0 to the desired target model sparsity ratio s f over the epoch ( n ) when performing the second learning rate decay, from the training epoch t 0 = 0 and with pruning frequency ∆ t = 1 epoch.
In our experiments, we used this gradual pruning scheme over different methods, except One-shot P+FT, SNIP, and the methods (DSR, SM) that have their own ﬁne-tuned gradual pruning scheme.
Hyper-parameters tuning procedure We grid-searched the optimal learning rate, starting from the range of { 0 . 05 , 0 . 10 , 0 . 15 , 0 . 20 } .
More precisely, we will evaluate a linear-spaced grid of learning rates.
If the best performance was ever at one of the extremes of the grid, we would try new grid points so that the best performance was contained in the middle of the parameters.
We trained most of the methods by using mini-batch SGD with Nesterov momentum.
For baselines involving ﬁne-tuning procedure (e.g. Table 2), we grid-searched the optimal results by tuning the optimizers (i.e. mini-batch SGD with Nesterov momentum, or Adam) and the learning rates.
The optimal hyper-parameters for DPF
The mini-batch size is ﬁxed to 128 for CIFAR-10 and 1024 for ImageNet regardless of datasets and models.
For CIFAR-10, we trained ResNet- a and VGG for 300 epochs and decayed the learning rate by 10 when accessing 50% and 75% of the total training samples (He et al., 2016a; Huang et al., 2017); and we trained WideResNet- a - b as Zagoruyko & Komodakis (2016) for 200 epochs and decayed the learning rate by 5 when accessing 30% , 60% and 80% of the total training samples.
The optimal learning rate for ResNet- a , WideResNet- a - b and VGG are 0 . 2 , 0 .
1 and 0 .
2 respectively; the corresponding weight decays are 1 e − 4 , 5 e − 4 and 1 e − 4 respectively.
For ImageNet training, we used the training scheme in Goyal et al.
(2017) for 90 epochs, where we gradually warmup the learning rate from 0 .
1 to 0 .
4 and decayed learning rate by 10 at 30; 60;epochs.
The used weight decay is 1 e − 4 .
A. A DDITIONAL R ESULTS FOR U NSTRUCTURED P RUNING A.3. C OMPLETE R ESULTS OF U NSTRUCTURED P RUNING ON CIFAR-
Table 4 details the numerical results for training SOTA DNNs on CIFAR-10.
Some results of it reconstruct the Table 1 and Figure 3.
[Table]
A.3. U NDERSTANDING THE T RAINING D YNAMICS AND L OTTERY T ICKET E FFECT
Figure 7 and Figure 8 complements Figure
[Table]
and details the training dynamics (e.g. the converge of δ and masks) of DPF from the other aspect.
Figure 9 compares the training dynamics between DPF and Incremental (Zhu & Gupta, 2017), demonstrating the fact that our scheme enables a drastical reparameterization over the dense parameter space for better generalization performance.
[Figure] [Figure]
[Figure]
Figure 10 in addition to the Figure 5 (in the main text) further studies the lottery ticket hypothesis under different training budgets (same epochs or same total ﬂops).
The results of DPF also demonstrate the importance of training-time structural exploration as well as the corresponding implicit regularization effects.
Note that we do not want to question the importance of the weight initialization or the existence of the lottery ticket.
Instead, our DPF can provide an alternative training scheme to compress the model to an extremely high compression ratio without sacriﬁcing the test accuracy, where most of the existing methods still meet severe quality loss (including Frankle & Carbin (2019);
Liu et al.
; Frankle et al.
A.3. C OMPUTATIONAL O VERHEAD AND THE I MPACT OF H YPER - PARAMETERS
In Figure 11, we evaluated the top-1 test accuracy of a compressed model trained by DPF under different setups, e.g., different reparameterization period p , different sparsity ratios, different mini-batch
[Figure]
sizes, as well as whether layer-wise pruning or not.
We can witness that the optimal reparameterization (i
[Figure]
e p = 16 ) is quite consistent over different sparsity ratios and different mini-batch sizes, and we used it in all our experiments.
The global-wise unstructured weight pruning (instead of layer-wise weight pruning) allows our DPF more ﬂexible to perform dynamic parameter reallocation, and thus can provide better results especially for more aggressive pruning sparsity ratios.
However, we also need to note that, for the same number of compressed parameters (layerwise or globalwise unstructured weight pruning), using global-wise pruning leads to a slight increase in the amount of
MACs, as illustrated Table
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
(whether layerwise) vs. sparsity ratio.
The heatmap displays the test accuracy.
We use mini-batch size 128 w/ reparameterization period p = 16 .
[Table]
w/o layerwise reparameterization
[Table]
w/ layerwise reparameterization
[Table]
Figure 12 demonstrates the trivial computational overhead of involving DPF to gradually train a compressed model (ResNet-50) from scratch (on ImageNet).
Note that we evaluated the introduced reparameterization cost for dynamic pruning, which is independent of (potential) signiﬁcant system speedup brought by the extreme high model sparsity.
Even though our work did not estimate the practical speedup, we do believe we can have a similar training efﬁciency as the values reported in Dettmers & Zettlemoyer (2019).
[Figure]
A.3. I MPLICIT N EURAL A RCHITECTURE S EARCH
DPF can provide effective training-time structural exploration or even implicit neural network search.
Figure 13 below demonstrates that for the same pruned model size (i
[Figure]
e. any point in the x -axis), we can always perform “architecture search” to get a better (in terms of generalization) pruned model, from a larger network (e
[Figure]
g. WideResNet-28-8) rather than the one searched from a relatively small network (e
[Figure]
g. WideResNet-28-4)
[Figure]
A. A DDITIONAL R ESULTS FOR S TRUCTURED P RUNING A.4. G ENERALIZATION PERFORMANCE FOR CIFAR-
Figure 14 complements the results of structured pruning in the main text (Figure 6), and Tabledetails the numerical results presented in both of Figure 6 and Figure 14.
[Figure]
T o p - 1 t e s t a cc u r a c y
[Figure]
t
[Figure]
t a
[Figure]
r
[Figure]
c y
[Figure]
[Table]
A.4. U NDERSTANDING THE L OTTERY T ICKET E FFECT
Similar to the observations in Section 6 (for unstructured pruning), Figure 15 instead considers structured pruning and again we found DPF does not ﬁnd a lottery ticket.
The superior generalization performance of DPF cannot be explained by the found mask or the weight initialization scheme.
[Figure]
A.4. M ODEL S PARSITY V ISUALIZATION
Figure 16 below visualizes the model sparsity transition patterns for different model sparsity levels under the structured pruning.
We can witness that due to the presence of residual connection, DPF gradually learns to prune the entire residual blocks.
[Table]
l o c k 2 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t li n e a r .
w e i g h t
[Table]
names 22222#
[Table]
f
[Table] [Table]
c o n v 1 .
w e i
[Table]
h t b l o c k 1 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t li n e a r .
w e i g h t
[Table] [Table] [Table] [Table]
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 0 .
c o n v 2 . w e i g h t b l o c k 1 . l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t b l o c k 1 . l a y e r . c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 2 . c o n v 2 . w e i g h t b l o c k 1 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 2 . l a y e r . 2 . c o n v 1 . w e i g h t
b l o c k 2 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 2 .
w e i g h t b l o c k 3 . l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t b l o c k 3 . l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t li n e a r .
w e i g h t
[Table]
[Table] [Table]
c o n
[Table]
h
[Table]
b l o c k
[Table]
l a
[Table]
e r
[Table] [Table]
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 1 . w e i g h t b l o c k 2 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 1 . w e i g h t b l o c k 3 . l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t li n e a r .
w e i g h t
[Table]
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 0 .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 0 .
c o n v 2 . w e i g h t b l o c k 1 . l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 1 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 0 . c o n v 1 . w e i g h t b l o c k 2 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 2 .
l a y e r . 0 . c o n v _ d o w n s a m p l e . w e i g h t b l o c k 2 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 2 .
l a y e r . c o n v 2 . w e i g h t b l o c k 2 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 2 . l a y e r . 2 . c o n v 2 . w e i g h t
b l o c k 2 .
l a y e r . c o n v 1 . w e i g h t b l o c k 2 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v 1 . w e i g h t b l o c k 3 .
l a y e r . 0 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 0 .
c o n v _ d o w n s a m p l e .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r . 2 .
c o n v 2 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 1 .
w e i g h t
b l o c k 3 .
l a y e r .
c o n v 2 .
w e i g h t li n e a r .
w e i g h t
[Figure]
names
[Figure] [Figure]
m s
[Figure]
B M ISSING P ROOFS
In this section we present the proofs for the claims in Section 4.
First, we give the proof for the stongly convex case.
Here we follow Lacoste-Julien et al.
(2012) for the general structure, combined with estimates from the error-feedback framework (Stich et al., 2018;
Stich & Karimireddy, 2019) to control the pruning errors.
Proof of Theorem
[Equation]
By deﬁnition of (DPF)
[Equation]
w t +1 =
w t − γ t g ( (cid:101) w t ) , hence,
E (cid:104)
[Equation]
By strong convexity,
[Equation]
f ( (cid:101) w t ) − f ( w (cid:63) )) , and with (cid:107) a + b (cid:107) 2 ≤ 2 (cid:107) a (cid:107) 2 + 2 (cid:107) b (cid:107) 2 further
[Equation]
and with (cid:104) a , b (cid:105) ≤ 12 α (cid:107) a (cid:107)
+ α 2 (cid:107) b (cid:107) 2 for a , b ∈ R d and α > 0 ,
[Equation]
where the last inequality is a consequence of L -smoothness.
Combining all these inequalities yields
E
[Equation]
as µ ≤ L .
Hence, by rearranging and multiplying with a weight λ t > 0 : λ t E
[Equation]
f
[Equation]
By plugging in the learning rate, γ t = 4 µ ( t +2) and setting λ t =
( t + 1) we obtain λ t E ( f ( (cid:101)
w
[Equation] [Equation]
[Equation] [Equation]
−
[Equation] [Equation]
[Equation] [Equation]
[Equation]
µ
[Equation] [Equation]
By summing from t = 0 to t =
T these λ t -weighted inequalities, we obtain a telescoping sum:
T (cid:88) t
[Equation]
Hence, for Λ T :=
[Equation]
Finally, using (cid:107) (cid:101) w t − w t (cid:107) 2 = δ t (cid:107) w t (cid:107) 2 by (1), shows the theorem.
Before giving the proof of Theorem 4.2, we ﬁrst give a justiﬁcation for the remark just below
Theorem 4.1 on the one-shot pruning of the ﬁnal iterate.
We have by L -smoothness and (cid:104) a , b (cid:105) ≤
12 α (cid:107) a (cid:107)
+ α 2 (cid:107) b (cid:107) 2 for a , b ∈ R d and α > 0 for any iterate w t :
[Equation]
Furthermore, again by L -smoothness,
[Equation]
as standard SGD analysis gives the estimate E (cid:107) w T − w (cid:63) (cid:107) 2 = O (cid:16) G 2 µ 2 T (cid:17) , see e.g. Lacoste-Julien et al.
Combining these two estimates (with w t = w T ) shows the claim.
Furthermore, we also claimed that also the dense model converges to a neighborhood of optimal solution.
This follows by L -smoothness and (4): For any ﬁxed model w t
we have the estimate (4), hence for a randomly chosen (dense) model u (from the same distribution as the sparse model in
Theorem 4.1) we have
[Equation]
Lastly, we give the proof of Theorem 4.2, following Karimireddy et al.
Proof of Theorem 4.2.
By smoothness, and (cid:104) a , b (cid:105) ≤ 12 (cid:107) a (cid:107) 2 + 12 (cid:107) b (cid:107) 2 for a , b ∈ R d ,
[Equation]
and by rearranging
[Equation]
Summing these inequalities from t = 0 to t =
T gives
T +
[Equation]
Finally, using (cid:107) (cid:101) w t − w t (cid:107) 2 = δ t (cid:107) w t (cid:107) 2 by (1), and plugging in the stepsize γ that minimizes the right hand side shows the claim.