P ROSPECT P RUNING : F INDING T RAINABLE W EIGHTS AT I NITIALIZATION USING M ETA -G RADIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Pruning neural networks at initialization would enable us to ﬁnd sparse models that retain the accuracy of the original network while consuming fewer computational resources for training and inference.
However, current methods are insufﬁcient to enable this optimization and lead to a large degradation in model performance.
In this paper, we identify a fundamental limitation in the formulation of current methods, namely that their saliency criteria look at a single step at the start of training without taking into account the trainability of the network.
While pruning iteratively and gradually has been shown to improve pruning performance, explicit consideration of the training stage that will immediately follow pruning has so far been absent from the computation of the saliency criterion.
To overcome the short-sightedness of existing methods, we propose Prospect Pruning (ProsPr), which uses meta-gradients through the ﬁrst few steps of optimization to determine which weights to prune.
ProsPr combines an estimate of the higherorder effects of pruning on the loss and the optimization trajectory to identify the trainable sub-network.
Our method achieves state-of-the-art pruning performance on a variety of vision classiﬁcation tasks, with less data and in a single shot compared to existing pruning-at-initialization methods.
I NTRODUCTION
Pruning at initialization —where we remove weights from a model before training begins—is a recent and promising area of research that enables us to enjoy the beneﬁts of pruning at training time, and which may aid our understanding of training deep neural networks.
Frankle & Carbin (2019) provide empirical evidence for the existence of sparse sub-networks that can be trained from initialization and achieve accuracies comparable to the original network.
These “winning tickets” were originally found in an iterative process where, in each iteration, the network is trained to full convergence followed by pruning a subset of the weights by magnitude.
The values of the remaining weights are then rewound to their value at initialization, and the process is repeated iteratively until the desired sparsity level is achieved.
This process, known as Lottery Ticket Rewinding (LTR), is very compute-intensive and is prone to failures.
For instance, Frankle et al.
(2020) show better results by rewinding weights not all the way back to initialization, but to early stages of training instead.
LTR is especially prone to failure for more difﬁcult problems (e.g., training on ImageNet), where we must rewind weights to their state several epochs into training.
A recent line of work proposes alternative practical solutions to identify these sub-networks before training begins, without the cost of retraining the network iteratively Lee et al.
Wang et al.
(2020); de Jorge et al.
; Tanaka et al.
This class of methods uses gradients to assess the importance of neural network weights.
These gradients are often known as Synaptic Saliencies and are used to estimate the effect of pruning a single parameter in isolation on various objectives, typically the loss function.
This objective is not so different from classical pruning-at-convergence methods, but the gradients for a well-trained model are small; therefore these methods must inspect higher-order metrics such as the Hessian to estimate the pruning effect (LeCun et al., 1990; Hassibi & Stork, 1993).
Pruning at initialization is desirable because the beneﬁts of pruning (in terms of memory and speed) can be reaped during training, rather than only at inference/deployment time.
However, the performance of prune-at-init methods remains poor: the degradation in accuracy is still signiﬁcant compared to training the full model and LTR, making these methods impractical for many real-world problems (Frankle et al., 2021).
In this paper, we identify a fundamental limitation in the objective formulation of current methods, namely that saliency criteria do not take into account the fact that the model is going to be trained after the pruning step.
If our aim was to simply prune a subset of weights without affecting the loss, then these saliency criteria are estimating the correct objective.
However, this estimate does not take into account that we are going to train the weights after we prune them.
We need a metric that captures the trainability of the weights during the optimization steps, rather than a single myopic estimate.
Many methods attempt to overcome this by pruning gradually and/or adding training steps between iterative pruning steps (Zhu & Gupta, 2018; You et al., 2020; de Jorge et al., 2021).
Although this approach has been shown to be effective, it is expensive and cumbersome in practice and ultimately is an indirect approximation to the trainability criteria we are looking to incorporate into our objective.
In this paper, we propose Prospect Pruning ( ProsPr ), a new pruning-at-init method that learns from the ﬁrst few steps of optimization which parameters to prune.
We explicitly formulate our saliency criteria to account for the fact that the network will be trained after pruning.
More precisely, ProsPr uses meta-gradients by backpropagating through the ﬁrst few model updates in order to estimate the effect the initial pruning parameters have on the loss after a few gradient descent steps.
Effectively this enables us to account for both higher-order effects of pruning weights on the loss, as well as the trainability of individual weights.
Similar to other methods we apply pruning to initialization values of weights and train our models from scratch.
In summary, our contributions are:
• We identify a key limitation in prior saliency criteria for pruning neural networks—namely that they do not explicitly incorporate trainability-after-pruning into their criteria.
• We propose a new pruning-at-init method, ProsPr , that uses meta-gradients over the ﬁrst few training steps to bridge the gap between pruning and training.
• We show empirically that ProsPr achieves higher accuracy compared to existing pruningat-init methods 1 .
Unlike other methods, our approach is single shot in the sense that the pruning is applied to the network initial weights in a single step.
B ACKGROUND
In this section we review the key concepts that our method builds upon.
We delay comparisons to other pruning techniques in the literature to Section 5.
Classic post-training pruning methods aim to identify and remove network weights with the least impact on the loss (LeCun et al., 1990; Hassibi & Stork, 1993).
They typically use the Taylor expansion of the loss with respect to parameters to deﬁne a saliency score for each parameter: δ L ≈ ∇ θ L (cid:62) δ θ + 12 δ θ (cid:62) H δ θ , where H = ∇ 2 θ L is the Hessian matrix.
When the network has converged, the ﬁrst-order term in the expansion is negligible, and hence these methods resort to using H .
Lee et al.
(2018) introduce SNIP, and show that the same objective of minimizing the change in loss can be used at initialization to obtain a trainable pruned network.
At initialization, the ﬁrst-order gradients ∇ θ in the local quadratic approximation are still signiﬁcant, so higher-order terms can be ignored.
Hence the computation of the parameter saliencies can be done using backpropagation.
The Taylor expansion approximates the effect of small additive perturbations to the loss.
To better approximate the effect of removing a weight, Lee et al.
(2018) attach a multiplicative all-one mask to the computation graph of each weight.
This does not change the forward-pass of the network, but it enables us to form the Taylor expansion around the mask values, rather than the weights, to estimate the effect of changing the mask values from 1 to 0 .
More speciﬁcally, SNIP computes the saliency scores according to:
[Equation]
with
[Equation]
  1 ... 1 ..

 c   w 11 ... w 1 N .. w Mw MN 

W init ⊗
W∇ W 0 L ( W 0 , D 0 )
W...
W M −∇ W M − 1 L ( W M − 1 , D M − 1 )
W M ∇ c L ( W M , D M )
D M
B a c kp r o p aga t e t h r o u g h e n t i re c o m pu t a t i o n g r a ph w i
t h re s p ec t t o c [Figure] where m is the number of weights in the network, c ∈ { 0 , 1 } m is the pruning mask (initialised toabove), D is the training dataset, w are the neural network weights, L is the loss function, and (cid:12) is the Hadamard product.
These saliency scores are computed before training the network, using one (or more) mini-batches from the training set.
The global Top-K weights with the highest saliency scores are retained ( c j = 1 ), and all other weights are pruned ( c j = 0 ), before the network is trained.
Our method, to be introduced in Section 3, also relies on computing the saliency scores for each mask element, but uses a more sophisticated loss function to incorporate the notion of trainability.
O UR M ETHOD : P ROS P R
In this section we introduce our method, Prospect Pruning (ProsPr).
We note that for the problem of pruning at initialization, the pruning step is immediately followed by training.
Therefore, pruning should take into account the trainability of a weight, instead of only its immediate impact on the loss before training.
In other words, we want to be able to identify weights that are not only important at initialization, but which may be useful for reducing the loss during training .
To this end, we propose to estimate the effect of pruning on the loss over several steps of gradient descent at the beginning of training, rather than the changes in loss at initialization.
More speciﬁcally, ProsPr models how training would happen by performing multiple (M) iterations of backpropagation and weight updates—like during normal training.
We can then backpropagate through the entire computation graph, from the loss several steps into training, back to the original mask, since the gradient descent procedure is itself differentiable. Once the pruning mask is computed, we rewind the weights back to their values at initialization and train the pruned network. The gradient-of-gradients is called a meta-gradient .
This algorithm is illustrated visually in Figure 1.
The higher-order information in the meta-gradient includes interactions between the weights during training. When pruning at initialization, our ultimate goal is to pick a pruned model, A, which is more trainable than an alternative pruned model
B. That means we want the loss L (ˆ y A , y ) to be lower than L (ˆ y B , y ) at convergence (for a ﬁxed pruning ratio).
Finding the optimal pruning mask is generally infeasible since the training horizon is long (i.e., evaluation is costly) and the space of possible pruning masks is large.
Unlike other methods that must compute the saliency scores iteratively, we can use the meta-gradients to compute the pruning mask in one shot .
This picks a line in loss-space, which more closely predicts the eventual actual loss. This is because it smooths out over more steps, and takes into account interactions between weights in the training dynamics. Crucially, in the limit of large M, the match to the ultimate objective is exact.
3. S ALIENCY S CORES VIA M ETA -G RADIENTS
We now introduce ProsPr formally.
After initialising the network weights randomly to obtain w init , we apply a weight mask to the initial weights,
[Equation]
This weight mask contains only ones, c = 1 , as in SNIP (Lee et al., 2018), and represents the connectivity of the corresponding weights.
We then sample M +1 batches of data D i ∼ D train ( i ∈ { 0 , . . .
, M } ; M ≥ 1 ) for the pruning step, and perform M weight updates 2 ,
[Equation]
Then, we compute a meta-gradient that backpropagates through these updates.
Speciﬁcally, we compute the gradient of the ﬁnal loss w.r.t.
the initial mask,
[Equation]
Using the chain rule, we can write out the form of the meta-gradient beginning from the last step:
[Equation]
) repeating for each step until we reach the zero’th step whose gradient is trivial,
[Equation]
In practice, we can compute the meta-gradients by relying on automatic differentiation software such as PyTorch (Paszke et al., 2019).
However, care must be taken to ensure that weights at each step are kept in memory so that the entire computation graph, including gradients, is visible to the automatic differentiation software.
The saliency scores are now given by
[Equation]
with
[Equation]
where w M is a function of c .
Equation (12) stands in contrast to SNIP, where the saliency is computed using the loss at c · w init rather than w M .
The saliency scores are then used to prune the initial weights w init : the ones with the highest saliency scores are retained ( c j = 1 ), and all other weights are pruned ( c j = 0 ).
Finally, the network is trained with the pruned weights ˆ w init .
Algorithm 1 summarises the proposed method, ProsPr.
3. F IRST -O RDER A PPROXIMATION
Taking the meta-gradient through many model updates (Equation (6)) can be memory intensive: in the forward pass, all gradients of the individual update steps need to be retained in memory to then be able to backpropagate all the way to the initial mask.
However, we only need to perform a few steps 3 at the beginning of training so in practice we can perform the pruning step on CPU which usually has access to more memory compared to a GPU.
We apply this approach in our own experiments, with overheads of around 30 seconds being observed for the pruning step.
Alternatively, when the number of training steps needs to be large we can use the following ﬁrstorder approximation. Using Equation (10), the meta-gradient is: [Equation]
[Equation]
as it is differentiable w.r t. the 1: Inputs: a training dataset D train , number of initial training steps M , number of main training steps N ( M (cid:28) N ), learning rate α 2: Initialise: network weights w init 3: c init =(cid:46) Initialise mask with ones 4: w 0 = c init (cid:12) w init (cid:46)
Apply mask to initial weights 5: for k = 0 , . .
, M − 1 do 6:
D k ∼ D train (cid:46) Sample batch of data 7: w i +1 =
w i − α ∇ w L ( w i , D k ) (cid:46) Update network weights 8: end for 9: g j ( w , D ) =
∂ L ( w M , D ) /∂c
j (cid:46) Compute meta-gradient 10: s j = |
g j ( w , D ) | (cid:80) mk =1 | g k ( w , D ) | (cid:46) Compute saliency scores 11: Determine the k-th largest element in s , s k .
12: c prune = (cid:26) 1 , if c j ≥ s k 0 , otherwise (cid:46) Set pruning mask 13: ˆ w 0 =
c prune (cid:12) w init (cid:46)
Apply mask to initial weights w init 14: for i = 1 , . .
. , N do (cid:46) Train pruned model 15:
{+ˆ w i +
 =+}
ˆ w i − α ∇ w L ( ˆ w i , D ) 16: end for writing w m in terms of w m − 1 following SGD,
[Equation]
(cid:34)
[Equation]
m [Equation] [Equation]
[Equation]
D m )) (cid:35)
[Equation]
(14) carrying through the partial derivative, =
[Equation]
) (cid:34) M (cid:89)
[Equation]
and ﬁnally dropping small terms for sufﬁciently small learning rates,
[Equation]
In the second-to-last step, we drop the higher-order terms, which gives us a ﬁrst-order approximation of the meta-gradient 4 .
With this approximation, we only need to save the initial weight vector w init in memory and multiply it with the ﬁnal gradient.
This approximation can be crude when the Laplacian terms are large, but with a sufﬁciently small learning rate it becomes precise.
The approximation allows us to take many more intermediate gradient-steps which can be beneﬁcial for performance when the training dataset has many classes, as we will see in Section 4.2.  E XPERIMENTS
We empirically evaluate the performance of our method, ProsPr, compared to various vision classiﬁcation baselines across different architectures and datasets. In supplementary sections we show different. [Figure] [Figure]
T e s t
A cc u r a c y
ResNet-20 (CIFAR-10) 20 . 0 48 .
Sparsity (%)
T e s t
A cc u r a c y
VGG-16 (CIFAR-10) [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
T e s t
A cc u r a c y
[Figure]
effectiveness of our method on image segmentation tasks (Appendix D) and when using selfsupervised initialization (Appendix E).
We provide details of our hyper-parameters, experiment setup, and implementation details in Appendix
4. R ESULTS ON CIFAR AND T INY -I MAGENET
In recent work, Frankle et al.
(2021) extensively study and evaluate different pruning-at-initialization methods under various effects such as weight re-initialization, weight shufﬂing, and score inversion.
They report the best achievable results by these methods and highlight the gap between their performance and two pruning-at-convergence methods, weight rewinding and magnitude pruning (Renda et al., 2020; Frankle et al., 2020).
In Figure 2 we evaluate ProsPr on this benchmark using ResNet-20 and VGG-16 on CIFAR-10, and ResNet-18 on Tiny-ImageNet.
It can be seen that ProsPr reduces the performance gap, especially at higher sparsity levels, and in some cases exceeds the accuracy of pruning-after-convergence methods.
Full results are also summarised in Appendix B.
This is a remarkable achievement: ProsPr is the ﬁrst work to close the gap to methods that prune after training.
Previous works that prune at the start have not been able to outperform methods that prune after training on any settings, including smaller datasets such as CIFAR-10 or Tiny-ImageNet.
It is also important to note that other baselines that have comparable accuracies are all iterative methods.
ProsPr is the only method that can do this in a single shot after using only 3 steps batch-sizes ofin the inner-loop before computing the meta-gradients.
In total, we only use 4 batches of data.
We also do not do any averaging of scores by repeating the method multiple times.
The performance in these small datasets comes from the fact that ProsPr computes higher-order gradients.
While there are other iterative methods that can work without any data, their effect is mostly a more graceful degradation at extreme pruning ratios as opposed to best accuracy at more practical sparsity levels.
One example is SynFlow which is similar to FORCE but uses an all-one input tensor instead of samples from the training set (Tanaka et al., 2020).
[Table]
[Table]
R ESULTS ON I MAGE N ET DATASET
To evaluate the performance of ProsPr on more difﬁcult tasks we run experiments on the larger ImageNet dataset.
Extending gradient-based pruning methods to this dataset poses several challenges.
Number of classes. In synaptic-saliency methods, the mini batches must have enough examples from all classes in the dataset.
Wang et al.
(2020) recommend using class-balanced mini-batches sized ten times the number of classes.
In datasets with few classes this is not an issue and even a single batch includes multiple examples per class.
This is one reason why methods like SNIP work with a single batch, and why we kept the number of steps in ProsPr’s inner loop ﬁxed to only 3.
ImageNet however has 1,000 classes, and using a single or a handful of small batches is inadequate.
Previous methods such as FORCE, GraSP, or SynFlow avoid this problem by repeating the algorithm with new data batches and averaging the saliency scores.
In ProsPr we instead increase the number of updates before computing the meta-gradients, ensuring they ﬂow through enough data.
Computing meta-gradients through many steps however poses new challenges.
Gradient degradation. We start to see gradient stability issues when computing gradients over deep loops.
Gradient degradation problems, i.e., vanishing and exploding gradients, have also been observed in other ﬁelds that use meta-gradients such as Meta-Learning.
Many solutions have been proposed to stabilize gradients when the length of loop increases beyond 4 or 5 steps, although this remains an open area of research (Antoniou et al., 2019).
Computation Complexity. For ImageNet we must make the inner loop hundreds of steps deep to achieve balanced data representation.
In addition to stability issues, backpropagating through hundreds of steps is very compute intensive.
Therefore for our experiments on ImageNet we use the ﬁrst-order approximation of ProsPr (Sec 3.2).
We evaluate ProsPr using ResNet-50 and VGG-19 architectures and compare against state-of-the-art methods FORCE and Iter-SNIP introduced by de Jorge et al.
We include multi-batch versions of SNIP and GraSP (SNIP-MB and GraSP-MB) to provide a fair comparison to iterative methods, which partially prune several times during training, in terms of the amount of data presented to the method.
We use 1024 steps with a batch size of 256 (i.e. 262,144 samples) for ResNet-50. For
VGG-19, a much larger model, and which requires more GPU memory we do 256 steps with batch size of 128.
This is still far fewer samples than other methods.
Force, for example, gradually prunes in 60 steps, where each step involves computing and averaging scores over 40 batches of size 256, i.e. performing backpropagation 2400 times and showing 614,400 samples to the algorithm.
Table 1 shows our results compared to the baselines reported by de Jorge et al.
First-order ProsPr exceeds previous results in all conﬁgurations except one, where it is outperformed by GraSP.
Note the surprisingly good performance of random pruning of ResNets, compared to other methods. This was also observed by de Jorge et al.
This could be explained by the fact that VGGis a much larger architecture with 143.6 million parameters, compared to 15.5 million in ResNet50s.
More speciﬁcally the ﬁnal three dense layers of VGG-19 constitute 86% of its total prunable parameters.
The convolution layers of VGG constitute only 14% of the prunable weights.
Pruning methods are therefore able to keep more of the convolution weights and instead prune extensively from the over-parametrized dense layers.
ResNet architectures on the hand have a single dense classiﬁer at the end.
4. S TRUCTURED P RUNING
We also evaluate ProsPr in the structured pruning setup where instead of pruning individual weights, entire convolutional channels (or columns of linear layers) are removed.
This is a signiﬁcantly more restricted setup but in addition to memory savings also reduces the computational cost of training and inference.
Adopting ProsPr for structured pruning is as simple as changing the shape of the pruning mask c in Eq 3 to have one entry per channel (or column of the weight matrix).
We evaluate our method against 3SP, a method that extends SNIP to structured pruning (van Amersfoort et al., 2020).
Our results are summarized in Table 2 which show accuracy improvements in all scenarios. In Appendix C we also evaluate wall-clock improvements in training time as a result of structured pruning at initialization. [Table] 4. N UMBER OF M ETA S TEPS
Finally
[Table]
we evaluate ProsPr when using a varying number of meta steps
[Table]
which gives insight into whether using meta-gradients is beneﬁcial
We repeated experiments from Section 4.3 but this time we vary the depth of training steps between 0 and 3.
The results in Table 3 show that the ﬁnal accuracy consistently increases as we increase the depth of the training, showing the effectiveness of meta-gradients.
We used the same data batch in all M training steps to isolate the effect of M, while in other experiments we use a new batch in every step.
In theory increasing the number of training steps should always help and in the limit of large M, the match to the ultimate objective (estimating the loss after many epochs of training) would be exact.
However
[Table]
in practice increasing the number of steps beyond 3 poses a lot of gradient stability issues (and is computationally expensive).
These issues have been also identiﬁed in the meta-learning literature and various solutions have been proposed to ﬁx it (Antoniou et al., 2019)
[Table]
[Table] [Table]  R ELATED W ORK Pruning at initialization Several works extend the approach proposed by Lee et al.
de Jorge et al.
(2021) evaluate SNIP objective in a loop in which pruned parameters still receive gradients and therefore have a chance to get un -pruned.
The gradual pruning helps avoid the layercollapse issue, and their method, known as FORCE, achieves better performance at extreme sparsity levels.
Tanaka et al.
(2020) provide theoretical justiﬁcation for why iteratively pruning helps with the layer-collapse issue and propose a data-free version of the method where an all-one input tensor is used instead of real training data.
Wang et al.
(2020) propose an alternative criterion to minimizing changes in the loss and instead argue for preserving the gradient ﬂow.
Their method, GraSP, keeps weights that contribute most to the norm of the gradients.
van Amersfoort et al.
(2020) extends SNIP and GraSP to structured pruning to make training and inference faster .
They further augment the scores by their compute cost to push the pruning decision towards more FLOPS reduction.
Gradual pruning As discussed in Section 1, in existing methods the training step has been absent from the saliency computation step.
As a workaround, many methods make their approaches training-aware by applying pruning gradually and interleaving it with training: Zhu & Gupta (2018) proposed an exponential schedule for pruning-during-training and Gale et al.
(2019) showed its effectiveness in a broader range of tasks.
Frankle & Carbin (2019) show that weight rewinding achieves better results when done in multiple prune-retrain steps.
Lym et al.
(2019) continuously apply structured pruning via group-lasso regularization while at the same time increasing batch sizes.
You et al.
(2020) a ﬁnd pruned architectures after a few epochs of training-and-pruning and monitoring a distance metric.
Meta-Gradients Backpropagation through gradients, and its ﬁrst-order approximation, is also used in model-agnostic meta-learning literature (Finn et al., 2017; Zintgraf et al., 2019) where the objective is to ﬁnd a model that can be adapted to new data in a few training steps.
Similar to our setup, the meta-loss captures the trainability of a model, but additionally, the meta-gradients are used to update the network’s weights in a second loop.
In self-supervised learning setting, Xiao et al.
(2021) use meta-gradients to explicitly optimize a learn-to-generalize regularization term in nested meta-learning loops. Computing gradients-of-gradients is also used to regularize loss with a penalty on the gradients, for instance, to enforce Lipschitz continuity on the network (Gulrajani et al., 2017) or to control different norms of the gradients (Alizadeh et al., 2020).
D ISCUSSION
Although pruning at initialization has the potential to greatly reduce the cost of training neural networks, existing methods have not lived up to their promise.
We argue that this is, in part, because they do not account for the fact that the pruned network is going to be trained after it is pruned.
We take this into account, using a saliency score that captures the effect of a pruning mask on the training procedure.
As a result, our method is competitive not just with methods that prune before training, but also with methods that prune iteratively during training and those that prune after training.
In principle, compressing neural networks at initialization has the potential to reduce energy and environmental costs of machine learning.
Beyond our context, taking into account that methods which prune-at-convergence generally have to be ﬁne-tuned, it is possible that our work could have further implications for these pruning methods as well (Molchanov et al., 2016; Wang et al., 2019).
A E XPERIMENTAL SETUP A. A RCHITECTURE D ETAILS
We use standard VGG and ResNet models provided by torchvision throughout this work where possible.
The ResNet-20 model, which is not commonly evaluated, was implemented to match the version used by Frankle et al.
(2021) so that we could compare using the benchmark supplied by this paper.
For smaller datasets, it is common to patch models deﬁned for ImageNet.
Speciﬁcally, for ResNets, we replace the ﬁrst convolution with one 3 × 3 ﬁlter size, and stride 1; the ﬁrst max-pooling layer is replaced with an identity operation.
For VGG, we follow the convention used by works such as FORCE (de Jorge et al., 2021).
We do not change any convolutional layers, but we change the classiﬁer to use a single global average pooling layer, followed by a single fully-connected layer.
A. T RAINING D ETAILS
For CIFAR-10, CIFAR-100 and TinyImageNet we perform 3 meta-steps to calculate our saliency criteria.
We train the resulting models for 200 epochs, with initial learning rate 0.1; we divide the learning rate by 10 at epochs 100 and 150.
Weight decay was set to 5 × 10 − 4 .
Batch size for CIFAR10, CIFAR-100, and TinyImageNet was 256.
For CIFAR-10 and CIFAR-100 we augment training data by applying random cropping ( 32 × 32 , padding 4), and horizontal ﬂipping.
For TinyImageNet we use the same procedure, with random cropping parameters set to 64 × 64 , padding 4.
For ImageNet we train models for 100 epochs, with an initial learning rate of 0.1; we divide the learning rate by 10 at epochs 30, 60 and 90.
Weight decay was set to 1 × 10 − 4 .
Batch size was 256.
We use the ﬁrst order approximation to do pruning, and use 1024 steps for ResNet-50.
For VGGwe use 2048 steps, but with batch size set to 128 (due to memory limitations, as our implementation only utilized a single GPU for meta-training).
We apply random resizing, then crop the image to 224 × 224 , with horizontal ﬂipping [Table]
A. I MPLEMENTATIONS
In addition to our code [Table] the reader may ﬁnd it useful to reference the following repos from related work.
Our experiments were performed using code derived from these implementations:
• https://github.com/naver/force • https://github.com/alecwangcq/GraSP • https://github.com/facebookresearch/open_lth • https://github.com/ganguli-lab/Synaptic-Flow • https://github.com/mil-ad/snip
B N UMBERS FROM F IGURE
[Table]
[Figure]
[-R ES N ET -
-][Table]
[Figure]
[Table]
[Figure]
[Table] [Table] [Table]
[Figure]
[Table]
When pruning is done at convergence, the beneﬁts of having a compressed model (in terms of memory saving and speed-up ) can only be utilized at inference/deployment time. However, with pruning-at-initialization these beneﬁts can be reaped during training as well. This is especially true in the case of structured pruning, where pruning results in weight and convolutional kernels with smaller dimensions (as opposed to unstructured pruning, where we end up with sparse weights with the original dimensions).
This means that in addition to memory savings, training take fewer operations which speeds up training. To evaluate the beneﬁts of training at initialization in terms of speed improvements we measured the wall-time training time on an NVIDIA RTX 2080 ti GPU for the architectures used in Section 4.3 (an additionally on ImageNet dataset). The results in Tableshow that structured pruning with ProsPr can signiﬁcantly reduce the overall training time [Table]
[Table]
D R ESULTS ON S EGMENTATION T ASK
An interesting, albeit less common, application for pruning models is within the context of segmentation.
In a recent paper Jeong et al.
(2021) train and prune the U-Net (Ronneberger et al., 2015) architecture on two image datasets from the Cell Tracking Challenge (PhC-C2DH-U373 and DICC2DH-HeLa).
They use the classic multi-step approach of gradually applying magnitude-pruning interleaved with ﬁne-tuning stages.
To evaluate the ﬂexibility of our method we used meta-gradients at the beginning of training (on a randomly initialized U-Net), prune in a single shot, and train the network once for the same number of epochs (50).
We kept the training set-up the same as the baseline by Jeong et al.
(2021) (i.e., resizing images and segmentation maps to (256,256), setting aside 30% of training data for validation) and similarly aim to ﬁnd the highest prune ratio that does not result in IOU degradation.
We report intersection-over-union (IOU) metric for the two datasets in
Tables 8 and 9: [Table] [Table] These results show that our method works as well (or better) compared to this compute-expensive
E S ELF - SUPERVISED I NITIALIZATION
To evaluate the robustness and consistency of our method against non-random initialization we ran experiments using BYOL to learn representations from unlabeled samples (Grill et al., 2020).
We used ResNet-18 as a backbone and trained for 1000 epochs with an embedding size of 64.
Unlike the vanilla ResNet-18 architecture used in Section 4.3 we used the commonly-used modiﬁed version of
ResNet-18 for smaller inputs (removing the ﬁrst pooling layer and modifying the ﬁrst convolutional layer to have kernel kernel size of 3, stride of 1, and padding size of 1).
We then used this trained
ResNet18 as the initialization for our meta-gradient pruning method.
After the pruning step, all layers were trained as before until convergence.
All training hyper-parameters were kept as before.
The results (ﬁnal test accuracies for 95% pruning) are summarized in Table 10.
[Table]
These results show the robustness of our method for this particular self-supervised initialization.
Starting from a learned representation can be challenging because these representations are much closer to weight values at convergence, and therefore the magnitude of their gradients is signiﬁcantly smaller than randomly initialized weights.
However, this is less of a problem for meta-gradients as their magnitude is still signiﬁcant due to back-propagation through training steps.
This can be seen in Figure 3 which shows the L2 norm of gradients of each layer of a BYOL-initialized ResNet-18 for meta-gradients compared to normal gradients.
It can be seen that meta-gradients provide a stronger signal compared to normal gradients.