P ROGRESSIVE S KELETONIZATION : T RIMMING MORE FAT FROM A NETWORK AT INITIALIZATION
A BSTRACT
Recent studies have shown that skeletonization (pruning parameters) of networks at initialization provides all the practical beneﬁts of sparsity both at inference and training time, while only marginally degrading their performance.
However, we observe that beyond a certain level of sparsity (approx 95% ), these approaches fail to preserve the network performance, and to our surprise, in many cases perform even worse than trivial random pruning.
To this end, we propose an objective to ﬁnd a skeletonized network with maximum foresight connection sensitivity (FORCE) whereby the trainability, in terms of connection sensitivity, of a pruned network is taken into consideration.
We then propose two approximate procedures to maximize our objective (1) Iterative SNIP: allows parameters that were unimportant at earlier stages of skeletonization to become important at later stages; and (2) FORCE: iterative process that allows exploration by allowing already pruned parameters to resurrect at later stages of skeletonization.
Empirical analysis on a large suite of experiments show that our approach, while providing at least as good a performance as other recent approaches on moderate pruning levels, provide remarkably improved performance on higher pruning levels (could remove up to 99 .
5% parameters while keeping the networks trainable)
I NTRODUCTION
The majority of pruning algorithms for Deep Neural Networks require training dense models and often ﬁne-tuning sparse sub-networks in order to obtain their pruned counterparts.
In Frankle & Carbin (2019), the authors provide empirical evidence to support the hypothesis that there exist sparse sub-networks that can be trained from scratch to achieve similar performance as the dense ones.
However, their method to ﬁnd such sub-networks requires training the full-sized model and intermediate sub-networks, making the process much more expensive.
Recently, Lee et al.
(2019) presented SNIP.
Building upon almost a three decades old saliency criterion for pruning trained models (Mozer & Smolensky, 1989), they are able to predict, at initialization, the importance each weight will have later in training.
Pruning at initialization methods are much cheaper than conventional pruning methods.
Moreover, while traditional pruning methods can help accelerate inference tasks, pruning at initialization may go one step further and provide the same beneﬁts at train time Elsen et al.
Wang et al.
(2020) (GRASP) noted that after applying the pruning mask, gradients are modiﬁed due to non-trivial interactions between weights.
Thus, maximizing SNIP criterion before pruning might be sub-optimal.
They present an approximation to maximize the gradient norm after pruning, where they treat pruning as a perturbation on the weight matrix and use the ﬁrst order Taylor’s approximation.
While they show improved performance, their approximation involves computing a Hessian-vector product which is expensive both in terms of memory and computation.
[Figure] We argue that both SNIP and GRASP approximations of the gradients after pruning do not hold for high pruning levels, where a large portion of the weights are removed at once.
In this work, while we rely on the saliency criteria introduced by Mozer & Smolensky (1989), we optimize what this saliency would be after pruning, rather than before .
Hence, we name our criteria Foresight
Connection sEnsitivity (FORCE).
We introduce two approximate procedures to progressively optimize our objective.
The ﬁrst, which turns out to be equivalent to applying SNIP iteratively, removes a small fraction of weights at each step and re-computes the gradients after each pruning round.
This allows to take into account the intricate interactions between weights, re-adjusting the importance of connections at each step.
The second procedure, which we name FORCE, is also iterative in nature, but contrary to the ﬁrst, it allows pruned parameters to resurrect.
Hence, it supports exploration, which otherwise is not possible in the case of iterative SNIP.
Moreover, one-shot SNIP can be viewed as a particular case of using only one iteration.
Empirically, we ﬁnd that both SNIP and GRASP have a sharp drop in performance when targeting higher pruning levels.
Surprisingly, they perform even worse than random pruning as can be seen in Fig 1.
In contrast, our proposed pruning procedures prove to be signiﬁcantly more robust on a wide range of pruning levels.
R ELATED WORK
Pruning trained models Most of the pruning works follow the train – prune – ﬁne-tune cycle (Mozer & Smolensky, 1989; LeCun et al., 1990; Hassibi et al., 1993; Han et al., 2015; Molchanov et al., 2017; Guo et al., 2016), which requires training the dense network until convergence, followed by multiple iterations of pruning and ﬁne-tuning until a target sparsity is reached.
Particularly,
Molchanov et al.
(2017) present a criterion very similar to Mozer & Smolensky (1989) and therefore similar to Lee et al.
(2019) and our FORCE,
but they focus on pruning whole neurons, and involve training rounds while pruning.
Frankle & Carbin (2019) and Frankle et al.
(2020) showed that it was possible to ﬁnd sparse sub-networks that, when trained from scratch or an early training iteration, were able to match or even surpass the performance of their dense counterparts.
Nevertheless, to ﬁnd them they use a costly procedure based on Han et al.
All these methods rely on having a trained network, thus, they are not applicable before training.
In contrast, our algorithm is able to ﬁnd a trainable sub-network with randomly initialized weights.
Making the overall pruning cost much cheaper and presenting an opportunity to leverage the sparsity during training as well.
Induce sparsity during training Another popular approach has been to induce sparsity during training.
This can be achieved by modifying the loss function to consider sparsity as part of the optimization (Chauvin, 1989; Carreira-Perpiñán & Idelbayev, 2018;
Louizos et al., 2018) or by dynamically pruning during training (Bellec et al., 2018; Mocanu et al., 2018; Mostafa & Wang, 2019; Dai et al., 2019; Dettmers & Zettlemoyer, 2020; Lin et al., 2020; Kusupati et al., 2020; Evci et al., 2019).
These methods are usually cheaper than pruning after training, but they still need to train the network to select the ﬁnal sparse sub-network.
We focus on ﬁnding sparse sub-networks before any weight update, which is not directly comparable.
Pruning at initialization These methods present a signiﬁcant leap with respect to other pruning methods.
While traditional pruning mechanisms focused on bringing speed-up and memory reduction at inference time, pruning at initialization methods bring the same gains both at training and inference time.
Moreover, they can be seen as a form of Neural Architecture Search (Zoph & Le, 2016) to ﬁnd more efﬁcient network topologies.
Thus, they have both a theoretical and practical interest.
Lee et al.
(2019) presented SNIP, a method to estimate, at initialization, the importance that each weight could have later during training.
SNIP analyses the effect of each weight on the loss function when perturbed at initialization.
In Lee et al.
(2020), the authors studied pruning at initialization from a signal propagation perspective, focusing on the initialization scheme.
Recently, Wang et al.
(2020) proposed GRASP, a different method based on the gradient norm after pruning and showed a signiﬁcant improvement for higher levels of sparsity.
However, neither SNIP nor GRASP perform sufﬁciently well when larger compressions and speed-ups are required and a larger fraction of the weights need to be pruned.
In this paper, we analyse the approximations made by SNIP and GRASP, and present a more suitable solution to maximize the saliency after pruning.
P ROBLEM F ORMULATION : P RUNING AT I NITIALIZATION
Given a dataset D = { ( x i , y i ) } ni =1 , the training of a neural network f parameterized by θ ∈ R m can be written as minimizing the following empirical risk:
[Equation]
where L and C denote the loss function and the constraint set, respectively.
Unconstrained (standard) training corresponds to C
= R m .
Assuming we have access to the gradients (batch-wise) of the empirical risk, an optimization algorithm ( e.g . SGD) is generally used to optimize the above objective, that, during the optimization process, produces a sequence of iterates { θ i } Ti =0 , where θ 0 and θ T denote the initial and the ﬁnal (optimal) parameters, respectively.
Given a target sparsity level of k < m , the general parameter pruning problem involves C with a constraint (cid:107) θ T (cid:107) 0 ≤ k , i.e ., the ﬁnal optimal iterate must have a maximum of k non-zero elements.
Note that there is no such constraint with the intermediate iterates.
Pruning at initialization, the main focus of this work, adds further restrictions to the above mentioned formulation by constraining all the iterates to lie in a ﬁxed subspace of C .
Precisely, the constraints are to ﬁnd an initialization θ 0 such that (cid:107) θ 0 (cid:107) 0 ≤ k 1 , and the intermediate iterates are θ i ∈ ¯ C ⊂ C , ∀ i ∈ { 1 , . . .
, T } , where ¯ C is the subspace of R m spanned by the natural basis vectors { e j } j ∈ supp ( θ 0 ) .
Here, supp ( θ 0 ) denotes the support of θ 0 , i.e ., the set of indices with non-zero entries.
The ﬁrst condition deﬁnes the sub-network at initialization with k parameters, and the second ﬁxes its topology throughout the training process.
Since there are (cid:0) mk (cid:1) such possible sub-spaces, exhaustive search to ﬁnd the optimal sub-space to optimize (1) is impractical as it would require training (cid:0) mk (cid:1) neural networks.
Below we discuss two recent approaches that circumvent this problem by maximizing a hand-designed data-dependent objective function.
These objectives are tailored to preserve some relationships between the parameters, the loss, and the dataset, that might be sufﬁcient to obtain a reliable θ 0 .
For the ease of notation, we will use θ to denote the dense initialization.
SNIP Lee et al.
(2019) present a method based on the saliency criterion from Mozer & Smolensky (1989).
They add a key insight and show this criteria works surprisingly well to predict, at initialization, the importance each connection will have during training.
The idea is to preserve the parameters that will have maximum impact on the loss when perturbed.
Let c ∈ { 0 , 1 } m be a binary vector, and (cid:12) the Hadamard product.
Then, the connection sensitivity in SNIP is computed as:
[Equation]
Once g ( θ ) is obtained, the parameters corresponding to the top- k values of | g ( θ ) i | are then kept.
Intuitively, SNIP favors those weights that are far from the origin and provide high gradients (irrespective of the direction).
We note that SNIP objective can be written as the following problem:
[Equation]
It is trivial to note that the optimal solution to the above problem can be obtained by selecting the indices corresponding to the top- k values of |
θ i ∇L ( θ )
i | .
GRASP Wang et al.
(2020) note that the SNIP saliency is measuring the connection sensitivity of the weights before pruning, however, it is likely to change after pruning.
Moreover, they argue that, at initialization, it is more important to preserve the gradient signal than the loss itself.
They propose to use as saliency the gradient norm of the loss ∆ L ( θ ) =
∇L ( θ ) T ∇L ( θ ) , but measured after pruning.
To maximize it, Wang et al.
(2020) adopt the same approximation introduced in LeCun et al.
and treat pruning as a perturbation on the initial weights.
Their method is equivalent to solving:
[Equation]
Where H and g denote the Hessian and the gradient of the loss respectively.
F ORESIGHT C ONNECTION S ENSITIVITY
Since removing connections of a neural network will have signiﬁcant impact on its forward and backward signals, we are interested in obtaining a pruned network that is easy to train.
We use connection sensitivity of the loss function as a proxy for the so-called trainability of a network.
To this end, we ﬁrst deﬁne connection sensitivity after pruning which we name Foresight Connection sEnsitivity (FORCE), and then propose two procedures to optimize it in order to obtain the desired pruned network.
Let ¯ θ = θ (cid:12) c denotes the pruned parameters once a binary mask c with (cid:107) c (cid:107)
= k ≤ m is applied.
The FORCE at ¯ θ for a given mask ˆ c is then obtained as:
[Equation]
The last equality is obtained by rewriting ¯ θ as diag ( θ ) c , where diag ( θ ) is a diagonal matrix with θ as its elements, and then differentiating w.r.t.
c .
Note, when k < m , the sub-network ¯ θ is obtained by removing connections corresponding to all the weights for which the binary variable is zero.
Therefore, only the weights corresponding to the indices for which c ( i ) = 1 contribute in equation (5), all other weights do not participate in forward and backward propagation and are to be ignored.
We now discuss the crucial differences between our formulation (5), SNIP (2) and GRASP (4).
• When ˆ c = 1 , the formulation is exactly the same as the connection sensitivity used in SNIP.
However, ˆ c = 1 is too restrictive in the sense that it assumes that all the parameters are active in the network and they are removed one by one with replacement , therefore, it fails to capture the impact of removing a group of parameters.
• Our formulation uses weights and gradients corresponding to ¯ θ thus, compared to SNIP, provides a better indication of the training dynamics of the pruned network.
However, GRASP formulation is based on the assumption that pruning is a small perturbation on the Gradient Norm which, also shown experimentally, is not always a reliable assumption.
• When (cid:107) ˆ c (cid:107)
(cid:28) (cid:107) 1 (cid:107) 0 ,
i
e ., extreme pruning, the gradients before and after pruning will have very different values as (cid:107) θ (cid:12) ˆ c (cid:107) 2 (cid:28) (cid:107) θ (cid:107) 2 , making SNIP and GRASP unreliable (empirically we ﬁnd SNIP and GRASP fail in the case of high sparsity).
FORCE saliency Note FORCE (5) is deﬁned for a given sub-network which is unknown a priori , as our objective itself is to ﬁnd the sub-network with maximum connection sensitivity.
Similar to the reformulation of SNIP in (3), the objective to ﬁnd such sub-network corresponding to the foresight connection sensitivity can be written as:
[Equation]
Here ∇L ( θ (cid:12) c ) i represents the i -th index of ∂ L (¯ θ ) ∂ ¯ θ (cid:12)(cid:12)(cid:12) c .
As opposed to (3), ﬁnding the optimal solution of (6) is non trivial as it requires computing the gradients of all possible (cid:0) mk (cid:1) sub-networks in order to ﬁnd the one with maximum sensitivity.
To this end, we present two approximate solutions to the above problem that primarily involve (i) progressively increasing the degree of pruning, and (ii) solving an approximation of (6) at each stage of pruning.
Progressive Pruning (Iterative SNIP)
Let k be the number of parameters to be kept after pruning.
Let us assume that we know a schedule (will be discussed later) to divide k into a set of natural numbers { k t }
Tt =1 such that k t > k t +1 and k T
= k .
Now, given the mask c t corresponding to k t , pruning from k t to k t +1 can be formulated using the connection sensitivity (5) as:
[Equation]
where ¯ θ = θ (cid:12) c t .
The additional constraint c (cid:12) c t =
c ensures that no parameter that had been pruned earlier is activated again.
Assuming that the pruning schedule ensures a smooth transition from one topology to another ( (cid:107) c t (cid:107)
≈ (cid:107) c
[Equation] +1 (cid:107) 0 ) such that the gradient approximation
[Equation]
[Table]
[Equation] [Table]
is valid, (7) can be approximated as solving (3) at ¯ θ .
Thus, for a given schedule over k , our ﬁrst approximate solution to (6) involves solving (3) iteratively.
This allows re-assessing the importance of connections after changing the sub-network.
For a schedule with T = 1 , we recover SNIP where a crude gradient approximation between the dense network c
= 1 and the ﬁnal mask c is being used.
This approach of ours turns out to be algorithmically similar to a concurrent work (Verdenius et al., 2020).
However, our motivation comes from a novel objective function (5) which also gives place to our second approach (FORCE).
Tanaka et al.
(2020) also concurrently study the effect of iterative pruning and report, similar to our ﬁndings, pruning progressively is needed for high sparsities.
Progressive Sparsiﬁcation (FORCE)
The constraint c (cid:12) c
t
=
c in (7) (Iterative SNIP) might be restrictive in the sense that while re-assessing the importance of unpruned parameters, it does not allow previously pruned parameters to resurrect (even if they could become important).
This hinders exploration which can be unfavourable in ﬁnding a suitable sub-network.
Here we remove this constraint, meaning, the weights for which c ( i ) = 0 are not removed from the network, rather they are assigned a value of zero.
Therefore, while not contributing to the forward signal, they might have a non-zero gradient.
This relaxation modiﬁes the saliency in (5) whereby the gradient is now computed at a sparsiﬁed network instead of a pruned network.
Similar to the above approach, we sparsify the network progressively and once the desired sparsity is reached, all connections with c ( i ) = 0 are pruned .
Note, the step of removing zero weights is valid if removing such connections does not adversely impact the gradient ﬂow of the unpruned parameters.
We, in fact, found it to be true in our experiments shown in Fig 7 (Appendix). However, this assumption might not hold always.
An overview of Iterative SNIP and FORCE is presented in Algorithm 1.
Sparsity schedule Both the above discussed iterative procedures approximately optimize (5), however, they depend on a sparsity/pruning schedule favouring small steps to be able to reliably apply the mentioned gradient approximation .
One such valid schedule would be where the portion of newly removed weights with respect to the remaining weights is small.
We ﬁnd a simple exponential decay schedule, deﬁned below, to work very well on all our experiments:
[Equation]
In section 5.3 we empirically show that these methods are very robust to the hyperparameter T .
Some theoretical insights When pruning weights gradually, we are looking for the best possible sub-network in a neighbourhood deﬁned by the previous mask and the amount of weights removed at that step.
The problem being non-convex and non-smooth makes it challenging to prove if the mask Algorithm 1 FORCE/ Iter SNIP obtained by our method is globally optimal.
However, in Appendix D we prove that each intermediate mask obtained with Iterative SNIP is indeed an approximate local minima, where the degree of sub-optimality increases with the pruning step size.
This gives some validation on why SNIP fails on higher sparsity.
We can not provide the same guarantees for FORCE (there is no obvious link between the step size and the distance between masks), nevertheless, we empirically observe that
FORCE is quite robust and more often than not improves over the performance of Iterative SNIP, which is not able to recover weights once pruned.
We present further analysis in Appendix C.4.
E XPERIMENTS
In the following we evaluate the efﬁcacy of our approaches accross different architectures and datasets.
Training settings, architecture descriptions, and implementation details are provided in Appendix A.
5. R ESULTS ON CIFAR-
Fig 2 compares the accuracy of the described iterative approaches with both SNIP and GRASP.
We also report the performance of a dense and a random pruning baseline.
Both SNIP and GRASP consider a single batch to approximate the saliencies, while we employ a different batch of data at each stage of our gradual skeletonization process.
For a fair comparison, and to understand how the number of batches impacts performance, we also run these methods averaging the saliencies over T batches, where T is the number of iterations.
SNIP-MB and GRASP-MB respectively refer to these multi-batch (MB) counterparts.
In these experiments, we use T = 300 .
We study the hyper-parameter robustness regarding T later in section 5.3.
[Figure]
We observe that for moderate sparsity levels, one batch is sufﬁcient for both SNIP and GRASP as reported in Lee et al.
; Wang et al.
However, as we increase the level of sparsity, the performance of SNIP and GRASP degrades dramatically.
For example, at 99 . 0% sparsity, SNIP drops down to 10% accuracy for both ResNet50 and VGG19, which is equivalent to random guessing as there are 10 classes.
Note, in the case of randomly pruned networks, accuracy is nearly 75% and 82% for ResNet50 and VGG19, respectively, which is signiﬁcantly better than the performance of
SNIP.
However, to our surprise, just using multiple batches to compute the connection sensitivity used in SNIP improves it from 10% to almost 90% .
This clearly indicates that a better approximation of the connection sensitivity is necessary for good performance in the case of high sparsity regime.
Similar trends, although not this extreme, can be observed in the case of GRASP as well.
On the other hand, gradual pruning approaches are much more robust in terms of sparsity for example, in the case of 99 . 9% pruning, while one-shot approaches perform as good as a random classiﬁer (nearly 10% accuracy), both FORCE and Iterative SNIP obtain more than 80% accuracy.
While the accuracies obtained at higher sparsities might have degraded too much for some use cases, we argue this is an encouraging result, as no approach before has pruned a network at initialization to such extremes while keeping the network trainable and these results might encourage the community to improve the performance further.
Finally, gradual pruning methods consistently improve over other methods even at moderate sparsity levels (refer to Fig 5), this motivates the use of FORCE or Iterative SNIP instead of other methods by default at any sparsity regime.
Moreover, the additional cost of using iterative pruning instead of SNIP-MB is negligible compared to the cost of training and is signiﬁcantly cheaper than GRASP-MB, further discussed in section 5.3. [Figure]
5. R ESULTS ON LARGER DATASETS
We now present experiments on large datasets.
Wang et al.
(2020) and Lee et al.
(2019) suggest using a batch of size ∼ 10 times the number of classes, which is very large in these experiments.
Instead, for memory efﬁciency, we average the saliencies over several mini-batches.
For CIFAR100 and Tiny-ImageNet, we average 10 and 20 batches per iteration respectively, with 128 examples per batch.
As we increase the number of batches per iteration, computing the pruning mask becomes more expensive.
From Fig 4, we observe that the accuracy converges after just a few iterations.
Thus, for the following experiments we used 60 iterations.
For a fair comparison, we run SNIP and GRASP with T × B batches, where T is the number of iterations and B the number of batches per iteration in our method.
We ﬁnd the results, presented in Fig 3, consistent with trends in CIFAR-10.
In the case of Imagenet, we use a batch size of 256 examples and 40 batches per iteration.
We use the ofﬁcial implementation of VGG19 with batch norm and Resnet50 from Paszke et al.
As presented in Table 1, gradual pruning methods are consistently better than SNIP, with a larger gap as we increase sparsity.
We would like to emphasize that FORCE is able to prune 90% of the weights of VGG while losing less than 3% of the accuracy, we ﬁnd this remarkable for a method that prunes before any weight update.
Interestingly, GRASP performs better than other methods at 95% sparsity (VGG), moreover, it also slightly surpasses FORCE for Resnet50 at 90%, however, it under-performs random pruning at 95%.
In fact, we ﬁnd all other methods to perform worse than random pruning for Resnet50.
We hypothesize that, for a much more challenging task (Imagenet with 1000 classes), Resnet50 architecture might not be extremely overparametrized.
For instance, [Table] [Figure] VGG19 has 143.68M parameters while Resnet50 uses 25.56M (refer to Table 2).
On the other hand the fact that random pruning can yield relatively trainable architectures for these sparsity levels is somewhat surprising and might indicate that there still is room for improvement in this direction.
Results seem to indicate that the FORCE saliency is a step in the right direction and we hypothesize further improvements on its optimization might lead to even better performance.
In Appendix C.6, we show superior performance of our approach on the Mobilenet-v2 architecture (Sandler et al., 2018) as well, which is much more "slim" than Resnet and VGG 2 .
5. A NALYSIS
Saliency optimization To experimentally validate our approach (7), we conduct an ablation study where we compute the FORCE saliency after pruning (5) while varying the number of iterations T for different sparsity levels.
In Fig 4 (left) we present the relative change in saliency as we vary the number of iterations T , note when T = 1 we recover one-shot SNIP.
As expected, for moderate levels of sparsity, using multiple iterations does not have a signiﬁcant impact on the saliency.
Nevertheless, as we target higher sparsity levels, we can see that the saliency can be better optimized when pruning iteratively.
In Appendix C.1 we include results for FORCE where we observe similar trends.
Hyperparameter robustness As shown in Figures 2 and 3, for low sparsity levels, all methods are comparable, but as we move to higher sparsity levels, the gap becomes larger.
In Fig 4 (middle) we ﬁx sparsity at 99.5% and study the accuracy as we vary the number of iterations T .
Each point is averaged over 3 trials.
SNIP ( T = 1 ) yields sub-networks unable to train (10% acc), but as we move to iterative pruning ( T > 1 ) accuracy increases up to 90% for FORCE and 89% for Iter SNIP.
Moreover, accuracy is remarkably robust to the choice of T , the best performance for both FORCE and Iter SNIP is with more iterations, however a small number of iterations already brings a huge boost.
This suggests these methods might be used by default by a user without worrying too much about hyper-parameter tuning, easily adapting the amount of iterations to their budget.
Pruning cost As shown in Fig 2, SNIP performance quickly degrades beyond 95% sparsity.
Wang et al.
(2020) suggested GRASP as a more robust alternative, however, it needs to compute a Hessian vector product which is signiﬁcantly more expensive in terms of memory and time.
In Fig(right), we compare the time cost of different methods to obtain the pruning masks along with the corresponding accuracy.
We observe that both SNIP and GRASP are fragile when using only one batch (red accuracy indicates performance below random baseline).
When using multiple batches their robustness increases, but so does the pruning cost.
Moreover, we ﬁnd that gradual pruning based on the FORCE saliency is much cheaper than GRASP-MB when using equal amount of batches, this is because GRASP involves an expensive Hessian vector product.
Thus, FORCE (or Iterative SNIP) would be preferable over GRASP-MB even when they have comparable accuracies.
FORCE vs Iterative SNIP Empirically, we ﬁnd that FORCE tends to outperform Iter SNIP more often than not, suggesting that allowing weights to recover is indeed beneﬁcial despite having less theoretical guarantees (see gradient approximation in Section 4).
Thus, we would make FORCE algorithm our default choice, especially for Resnet architectures.
In Appendix C.4 we empirically observe two distinct phases when pruning with FORCE.
The ﬁrst one involves exploration (early phase) when the amount of pruned and recovered weights seem to increase, indicating exploration of masks that are quite different from each other.
The second, however, shows rapid decrease in weight recovery, indicating a phase where the algorithm converges to a more constrained topology.
As opposed to Iter SNIP, the possibility of the exploration of many possible sub-networks before converging to a ﬁnal topology might be the reason behind the slightly improved performance of
FORCE.
But this exploration comes at a price, in Fig 4 (middle and left) we observe how, despite
FORCE reaching a higher accuracy when using enough steps, if we are under a highly constrained computational budget and can only afford a few pruning iterations, Iter SNIP is more likely to obtain a better pruning mask.
This is indeed expected as FORCE might need more iterations to converge to a good sub-space, while Iter SNIP will be forced to converge by construction.
A combination of
FORCE and Iter SNIP might lead to an even better approach, we leave this for future work.
Early pruning as an additional baseline Our gradual pruning approaches (SNIP-MB and GRASPMB as well) use multiple batches to obtain a pruned mask, considering that pruning can be regarded as a form of training (Mallya et al., 2018), we create another baseline for the sake of completeness.
We train a network for one epoch, a similar number of iterations as used by our approach, and then use magnitude pruning to obtain the ﬁnal mask, we call this approach early pruning (more details in Appendix C.5).
Interestingly, we ﬁnd that early pruning tends to perform worse than SNIP-MB (and gradual pruning) for Resnet, and shows competitive performance at low sparsity level for VGG but with a sharp drop in the performance as the sparsity level increases.
Even though these experiments support the superiority of our approach, we would like to emphasize that they do not conclude that any early pruning strategy would be suboptimal compared to pruning at initialization as an effective approach in this direction might require devising a well thought objective function.
Iterative pruning to maximize the Gradient Norm
In Sec 4, we have seen Iterative SNIP can be used to optimize the FORCE saliency.
We also tried to use GRASP iteratively, however, after a few iterations the resulting networks were not trainable.
Interestingly, if we apply the gradient approximation to GRASP saliency (instead of Taylor), we can come up with a different iterative approximation to maximize the gradient norm after pruning.
We empirically observe this method is more robust than GRASP to high sparsity levels.
This suggests that 1) Iterative pruning, although beneﬁcial, can not be trivially applied to any method.
The gradient approximation is more general than in the context of FORCE/SNIP sensitivity.
We present further details and results in Appendix E.
D ISCUSSION
Pruning at initialization has become an active area of research both for its practical and theoretical interests.
In this work, we discovered that existing methods mostly perform below random pruning at extreme sparsity regime.
We presented FORCE, a new saliency to compute the connection sensitivity after pruning, and two approximations to progressively optimize FORCE in order to prune networks at initialization.
We showed that our methods are signiﬁcantly better than the existing approaches for pruning at extreme sparsity levels, and are at least as good as the existing ones for pruning at moderate sparsity levels. We also provided theoretical insights on why progressive skeletonization is beneﬁcial at initialization, and showed that the cost of iterative methods is reasonable compared to the existing ones.
Although pruning iteratively has been ubiquitous in the pruning community, it was not evident that pruning at initialization might beneﬁt from this scheme.
Particularly, not every approximation could be used for gradual pruning as we have shown with GRASP.
However, the gradient approximation allowed us to gradually prune while maximizing either the gradient norm or FORCE.
We consider our results might encourage future work to further investigate the exploration/exploitation trade-off in pruning and ﬁnd more efﬁcient pruning schedules, not limited to the pruning at initialization. A CKNOWLEDGMENTS
This work was supported by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme, EPSRC/MURI grant EP/N019474/1 and Five AI Limited. Pau de Jorge was fully funded by NAVER LABS Europe.
Amartya Sanyal acknowledges support from The Alan Turing Institute under the Turing Doctoral Studentship grant TU/C/000023. Harkirat was supported using a Tencent studentship through the University of Oxford. A P RUNING IMPLEMENTATION DETAILS
We present experiments on CIFAR-10/100 (Krizhevsky et al., 2009), which consists of 60k 32 ×colour images divided into 10 / 100 classes, and also on Imagenet challenge ILSVRC-2012 (Russakovsky et al., 2015) and its smaller version Tiny-ImageNet, which respectively consist of 1.2M / 1k and 100k / 200 images / classes.
Networks are initialized using the Kaiming normal initialization (He et al., 2015).
For CIFAR datasets, we train Resnet50 3 and VGG19 4 architectures during 350 epochs with a batch size of 128.
We start with a learning rate of 0 . 1 and divide it by 10 at 150 and 250 epochs.
As optimizer we use SGD with momentum 0 .
9 and weight decay 5 × 10 − 4 .
We separate 10% of the training data for validation and report results on the test set.
We perform mean and std normalization and augment the data with random crops and horizontal ﬂips.
For Tiny-Imagenet, we use the same architectures.
We train during 300 epochs and divide the learning rate by 10 at 1/2 and 3/4 of the training.
Other hyper-parameters remain the same.
For ImageNet training, we adapt the ofﬁcial codeof Paszke et al.
and we use the default settings.
In this case, we use the Resnet50 and VGG with batch normalization architectures as implemented in Paszke et al.
In the case of FORCE and Iter SNIP, we adapt the same public implementation 6 of SNIP as Wang et al.
Instead of deﬁning an auxiliary mask to compute the saliencies, we compute the product of the weight times the gradient, which was shown to be equivalent in Lee et al.
As for
GRASP, we use their public code.
After pruning, we implement pruned connections by setting the corresponding weight to 0 and forcing the gradient to be 0.
This way, a pruned weight will remainduring training.
An important difference between SNIP and GRASP implementations is in the way they select the mini-batch to compute the saliency.
SNIP implementation simply loads a batch from the dataloader.
In contrast, in GRASP implementation they keep loading batches of data until they obtain exactly 10 examples of each class, discarding redundant samples.
In order to compare the methods in equal conditions, we decided to use the way SNIP collects the data since it is simpler to implement and does not require extra memory.
This might cause small discrepancies between our results and the ones reported in Wang et al.
[Table]
A meaningful design choice regarding SNIP and GRASP implementations is that they only prune convolutional and fully connected layers.
These layers constitute the vast majority of parameters in most networks, however, as we move to high sparsity regimes, batch norm layers constitute a non-negligible amount.
For CIFAR10, batch norm plus biases constitute 0.2% and 0.05% of the parameters of Resnet50 and VGG19 networks respectively.
For consistency, we have as well restricted pruning to convolutional and fully connected layers and reported percentage sparsity with respect to the prunable parameters, as is also done in Lee et al.
(2019) and Wang et al.
(2020) to the best of our knowledge.
In Table 2 we show the percentage of prunable weights for each network and
[Figure]
dataset we use.
In future experiments we will explore the performance of pruning at initialization when including batch norm layers and biases as well.
B A DDITIONAL ACCURACY - SPARSITY PLOTS
In the main text we show the complete range of the accuracy-sparsity curves for the different methods so it is clear why more robust methods are needed.
However, it makes it more difﬁcult to appreciate the smaller differences at lower sparsities.
In Fig 5 we show the accuracy-sparsity curves where we cut the y axis to show only the higher accuracies
[Figure]
C F URTHER ANALYSIS OF PRUNING AT INITIALIZATION C. S ALIENCY VS T
In Fig 4 (left) we have seen that for higher sparsity levels, FORCE obtains a higher saliency when we increase the number of iterations.
In Fig 6 we compare the relative saliencies as we increase the number of iterations for FORCE and Iterative SNIP.
As can be seen, both have a similar behaviour.
C. P RUNING VS S PARSIFICATION
FORCE algorithm is able to recover pruned weights in later iterations of pruning.
In order to do that, we do not consider the intermediate masks as pruning masks but rather as sparsiﬁcation masks , where connections are set to 0 but not their gradients.
In order to understand how does computing the FORCE (5) on a sparsiﬁed vs pruned network affect the saliency, we prune several masks with
FORCE algorithm at varying sparsity levels.
For each mask, we then compute their FORCE saliency either considering the pruned network (gradients of pruned connections will be set to 0 during the backward pass) or the sparsiﬁed network (we only set to 0 the connections, but let the gradient signal
[Figure] [Figure]
ﬂow through all the connections).
Results are presented in Fig 7.
We observe that the two methods to compute the saliency are strongly correlated, thus, we can assume that when we use the FORCE algorithm that maximizes the saliency of sparsiﬁed networks we will also maximize the saliency of the corresponding pruned networks
[Figure]
C. N ETWORK STRUCTURE AFTER PRUNING
In Fig 8 we visualize the structure of the networks after pruning 99 .
9% of the parameters.
We show the fraction of remaining weights and the total number of remaining weights per layer after pruning.
As seen in (a) and (d), all analysed methods show a tendency to preserve the initial and ﬁnal layers and to prune more heavily the deep convolutional layers, this is consistent with results reported in Wang et al.
In (b) and (e), we note that FORCE has a structure that stands out compared to other methods that are more similar.
This is reasonable since, it is the only method that allows pruned weights to recover.
In the zoomed plots (c) and (f) we would like to point out that FORCE and Iterative SNIP preserve more weights on the deeper layers than GRASP and SNIP for VGG while we observe the opposite behaviour for Resnet50.
In Fig 2, we observe that gradual pruning is able to prune the Resnet50 network up to 99.99% sparsity without falling to random accuracy.
In contrast, with VGG19 we observe Iterative SNIP is not able to prune more than 99.9%.
In Fig 8 we observe that for Resnet50, all methods prune some layers completely.
However, in the case of ResNets, even if a convolutional layer is entirely pruned, skip connections still allow the ﬂow of forward and backward signal.
On the other hand, architectures without skip connections, such as VGG, require non-empty layers to keep the ﬂow of information.
Interestingly, in (c) we observe how FORCE and Iter SNIP have a larger amount of completely pruned layers than GRASP, however, there are a few deep layers with a signiﬁcantly larger amount of unpruned weights.
This seems to indicate that when a high sparsity is required, it is more efﬁcient to have fewer layers with more weights than several extremely sparse layers.
[Figure]
[Figure]
C. E VOLUTION OF PRUNING MASKS
As discussed in the main text
FORCE allows weights that have been pruned at earlier iterations to become non-zero again, we argue this might be beneﬁcial compared to Iterative SNIP which will not be able to correct any possible "mistakes" made in earlier iterations.
In particular, it seems to give certain advantage to prune VGG to high sparsities without breaking the ﬂow of information (pruning a layer entirely) as can be seen in Fig 2 (b).
In order to gain a better intuition of how does the amount of pruned/recovered weights ratio evolve during pruning, in Fig 9 we plot the normalized amount of pruned and recovered weights (globally on the whole network) at each iteration of FORCE and also for Iter SNIP as a sanity check.
Note that Iterative SNIP does not recover weights and the amount of weights pruned at each step decays exponentially (this is expected since we derived Iterative
SNIP as a constrained optimization of FORCE where each network needs to be a sub-network of the previous iteration.
On the other hand, FORCE does recover weights.
Moreover, the amount of pruned/recovered weights does not decay monotonically but has a clear peak, indicating there are two phases during pruning: While the amount of pruned weights increases, the algorithm explores masks which are quite far away from each other, although this might be harmful for the gradient approximation (refer to section 4), we argue that during the initial pruning iterations the network is still quite over-parametrized.
After reaching a peak, both the pruning and recovery rapidly decay, thus the masks converge to a more constrained subset.
C. C OMPARISON WITH EARLY PRUNING
For fair comparison, we provided the same amount of data to SNIP and GRASP as was used by our approach and call this variant SNIP-MB and GRASP-MB.
Similarly, under this new baseline which we call early pruning , we train the network on 1 epoch of data of CIFAR-10, that is slightly more examples than our pruning at initialization methods which use 128 · 300 = 38400 examples (see section 5).
After training for 1 epoch we perform magnitude pruning which requires no extra cost
[Figure]
(results presented in Fig 10).
Although early pruning yields competitive results for VGG at moderate sparsity levels, it soon degrades its performance as we prune more weights.
On the other hand, for
Resnet architecture it is sub-optimal at all evaluated sparsity levels.
Note, this result does not mean that any early pruning strategy would be sub-optimal compared to pruning at initialization, however exploring this further is out of the scope of this work.
C. M OBILENET EXPERIMENTS
All our experiments were on overparameterized architectures such as Resnet and VGG.
To test the wider usability of our methods, in this section we prune the Mobilenet-v2 architecture 8 (Sandler et al., 2018) which is much more "slim" than Resnet and VGG (Mobilenet has 2.3M params compared to 20.03M and 23.5M of VGG and Resnet respectively).
Results are provided in Fig 11.
Similarly to Resnet and VGG architectures, we see that gradual pruning tends to better preserve accuracy at higher sparsity levels than one-shot methods.
Moreover, both FORCE and Iter SNIP improve over
SNIP at moderate sparsity levels as well.
FORCE and Iter SNIP have comparable accuracies except for high sparsity where Iter SNIP surpasses FORCE.
We hypothesize for such a slim architecture ( ≈ 10 × fewer parameters than Resnet and VGG)
the gradient approximation becomes even more sensitive to the distance between iterative masks and perhaps the exploration of FORCE is harmful in this case.
As discussed in the main paper, we believe that further research to understand the exploration/exploitation trade-off when pruning might yield to even more efﬁcient pruning schemes, especially for very high sparsity levels.
We train using the same settings as described in Appendix A except for the weight decay which is set to 4 × 10 − 5 , following the settings of the original Mobilenet paper.
[Figure]
tends to improve slightly does not an drop it seems to be sub-
D L OCAL OPTIMAL MASKS
Deﬁnition 1 ( ( p, (cid:15) ) -local optimal mask) .
Consider any two sets 9 c t ⊆ { 1 , · · · , m } and c t +1 ⊂ c t .
For any (cid:15) > 0 and 0 ≤ p ≤ | c t \ c t +1 | , c t +1 is a ( p, (cid:15) )
local optimal with respect to c t if the following holds
[Equation]
(9) for all S − ⊂ c t +1 , | S − | = p and S + ⊂ ( c t \ c t +1 ) , | S + | = p .
Deﬁnition 2 (CRS; Coordinate-Restricted-Smoothness) .
Given a function L : R m → R (which encodes both the network architecture and the dataset), L is said to be λ c -Coordinated Restricted
Smooth with respect to c ⊆ { 1 , · · · , m } if there exists a real number λ c such that
[Equation]
[Equation] for all w ∈ R m and (cid:98) c ⊂ c .
When s = | c \ (cid:98) c | , an application of Holder’s inequality shows
[Equation]
We deﬁne L to be Λ -total CRS if there exists a function Λ : { 0 , 1 } m → R such that for all c ∈ { 0 , 1 } m
L is Λ ( c ) -Coordinate-Restricted-Smooth with respect to c (for ease of notation we use Λ ( c )
=
λ c ).
Theorem 1 (Informal) .
The mask c t +1 produced from c t by FORCE is (cid:16) p, 2 λp (cid:107) θ (cid:107) 2 ∞ | c t | (cid:17) -local optimal if the L is Λ -CRS..
Proof.
Consider the masks c t and c t +1 where the latter is obtained by one step of FORCE on the former.
Let S − and S + be any set of size p such that S − ⊂ c t +1 and S + ⊂ ( c t \ c t +1 ) .
Finally, for ease of notation we deﬁne ζ = ( c t +1 \ S − ) ∪ S +
S ( θ, ζ ) − S ( θ, c t + [Equation] ) = (cid:88)
[Equation]
Let us look at the three terms individually.
We assume that L is Λ -CRS.
[Equation]
By Triangle Inequality
[Equation]
By Holder’s Inequality
[Equation]
[Equation]
Adding eqs.
(13) and (14), we get Γ
[Equation]
≤ (cid:88)
[Equation]
γp (15) where γ = min i ∈ c t +1 ,j [Equation]
[Equation]
t +
[Equation]
θ i
[Equation] θ [Equation] c t )
[Equation] | − [Equation]
[Table]
Substituting eqs.
(12) and (15) into (11) we get
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
E I TERATIVE PRUNING TO MAXIMIZE THE GRADIENT NORM
[Figure]
E. M AXIMIZING THE GRADIENT NORM USING THE GRADIENT APPROXIMATION
In order to maximize the gradient norm after pruning, the authors in Wang et al.
(2020) use the ﬁrst order Taylor’s approximation.
While this seems to be better suited than SNIP for higher levels of sparsity, it assumes that pruning is a small perturbation on the weight matrix.
We argue that this approximation will not be valid as we push towards extreme sparsity values.
Our gradient approximation (refer to section 4) can also be applied to maximize the gradient norm after pruning.
In this case, we have
[Equation]
where we assume pruned connections have null gradients (this is equivalent to the restriction used for Iterative SNIP)
and we assume gradients remain unchanged for unpruned weights (gradient approximation).
Combining this approximation with Eq.
(7), we obtain a new pruning method we name Iterative GRASP, although it is not the same as applying GRASP iteratively .
Unlike FORCE,
Iterative GRASP does not recover GRASP when T = 1 .
In Fig 12 we compare Iterative GRASP to other pruning methods.
We use the same settings as described in section 5.
We observe that Iterative GRASP outperforms GRASP in the high sparsity region.
Moreover, for VGG19 architecture Iterative GRASP achieves comparable performance to
Iterative SNIP.
Nevertheless, for Resnet50 we see that Iterative GRASP performance falls below that of FORCE and Iterative SNIP as we prune more weights.
FORCE saliency takes into account both the gradient and the magnitude of the weights when computing the saliency, on the other hand, the Gradient Norm only takes gradients into account, therefore it is using less information.
We hypothesize this might the reason why Iterative GRASP does not match Iterative SNIP.
E. I TERATIVE GRASP (P RUNING CONSISTENCY )
[Figure]
As explained in the main text, we tried to apply GRASP iteratively with the Taylor approximation described in Wang et al.
Unfortunately, we found that all resulting masks yield networks unable to train.
In light of this result, we performed some analysis of the behaviour of GRASP compared to that of SNIP and, in the following, we provide some insights as to why we can not use
GRASP’s approximation iteratively.
In Liu et al.
, the authors show that applying a pruning method to the same architecture with different random initializations would yield consistent pruning masks.
Speciﬁcally, they ﬁnd that the percentage of pruned weights in each layer had very low variance.
We reproduce the same experiment and additionally explore another dimension, the (global) sparsity level.
Given an architecture, we prune it at varying levels of sparsity and extract the percentage of remaining weights at each layer.
For each level of sparsity, we average the results over 3 trials of initialize-prune.
As shown in Fig 2, both SNIP and GRASP have very low variance across initializations, on the other hand, as we vary the global sparsity with GRASP, the percentage of remaining weights for each layer is inconsistent.
The layers that are most preserved at high levels of sparsity, such as the initial and last layers, are the most heavily pruned at low sparsity levels.
The authors in Liu et al.
(2018), reason that manually designed networks have layers which are more redundant than others.
Therefore, pruning methods even this redundancies by pruning layers with different percentages.
We extend this reasoning, and hypothesize that pruning algorithms should always have preference for pruning the same (redundant) layers across all levels of sparsity.
We denote this as pruning consistency .
We observe that when applying iterative pruning to GRASP, the resulting masks tend to prune almost all of the weights at the initial and ﬁnal layers, producing networks that are unable to converge.
When using iterative pruning, we prune a small portion of remaining weights at each step.
Thus, we are always in the low sparsity regime, where the GRASP behaviour is reversed.
Conversely, when we use SNIP the behaviour changes completely.
In this case, the preserved layers are consistent across sparsity levels, and when we use iterative pruning we obtain networks that reach high accuracy values.