R ETHINKING A GAIN THE V ALUE OF N ETWORK P RUNING - A D YNAMICAL I SOMETRY P ERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
Several recent works questioned the value of inheriting weights in structured neural network pruning because they empirically found training from scratch can match or even outperform ﬁnetuning a pruned model.
In this paper, we present evidence that this argument is actually inaccurate because of using improperly small ﬁnetuning learning rates.
With larger learning rates, our results consistently suggest pruning outperforms training from scratch on multiple networks (ResNets,
VGG11) and datasets (MNIST, CIFAR10, ImageNet) over most pruning ratios.
To deeply understand why ﬁnetuning learning rate holds such a critical role, we examine the theoretical reason behind through the lens of dynamical isometry , a nice property of networks that can make the gradient signals preserve norm during propagation.
Our results suggest that weight removal in pruning breaks dynamical isometry, which fundamentally answers for the performance gap between a large ﬁnetuning LR and a small one .
Therefore, it is necessary to recover the dynamical isometry before ﬁnetuning.
In this regard, we also present a regularization-based technique to do so, which is rather simple-to-implement yet effective in dynamical isometry recovery on modern residual convolutional neural networks.
I NTRODUCTION
Pruning is a time-honored methodology to reduce parameters in a neural network without seriously compromising its performance (Reed, 1993; Sze et al., 2017).
The prevailing pipeline of pruning comprises three steps: 1) pretraining : train a dense model; 2) pruning : prune the dense model based on certain rules; 3) ﬁnetuning : retrain the pruned model to regain performance.
Most existing research focuses on the second step, seeking the best criterion to select unimportant weights so as to incur as less performance degradation as possible.
This 3-step pipeline has been practiced for more than 30 years (Mozer & Smolensky, 1989; LeCun et al., 1990) and is still extensively adopted in today’s pruning methods (Sze et al., 2017).
These said, several recent works (Crowley et al., 2018; Liu et al., 2019) questioned the necessity of inheriting weights from a pretrained model because they empirically found the small model trained from scratch can match (or sometimes outperform) the counterpart pruned from the pre-trained large model.
This acutely challenges the past wisdom as well as our common belief about pruning.
As far as we know, there is no formal response to this critical conﬂict.
A theoretical-level understanding of this problem is even more elusive.
Meanwhile, the pruning community has been observing even more open questions.
Speciﬁcally, (Renda et al., 2020; Le & Hua, 2021) found that the learning rate (LR) in ﬁnetuning holds a critical role in the ﬁnal performance.
A proper learning rate schedule ( e.g. , a larger initial LR 10 − 2 vs. 10 −with step-decay schedule) can improve the top-1 accuracy of a pruned ResNet-34 model (He et al., 2016) by more than 1% on ImageNet (Deng et al., 2009).
This discovery calls for more attention being paid to the ﬁnetuning step when comparing different pruning methods.
Unfortunately, they did not present more theoretical insights to explain its occurrence.
This also remains an open question in the community up to date.
In this paper, we will show these two open questions actually point to the same one.
Speciﬁcally, we rerun the experiments of (Crowley et al., 2018; Liu et al., 2019) and ﬁnd simply using a larger ﬁnetuning LR ( 10 − 2 vs. 10 − 3 and decay it) can signiﬁcantly improve the ﬁnal performance.
Compared to the improved pruning performance, training from scratch does not compete or surpass pruning anymore (see Tab. 1 and Tab. 2 on ImageNet).
This observation invites many questions immediately: (1) Theoretical understanding: Why does this happen?
What is the theoretical reason behind it?
Practical solution: If this is a problem, how to ﬁx it?
Can the understanding of this problem lead us to better pruning algorithms?
This paper will present answers to all these questions.
The key tool we employ to unveil the mysteries is dynamical isometry (Saxe et al., 2014), which describes a kind of nice property in neural networks that are easy to optimize.
We carefully design an explanatory experiment using a linear
MLP (multi-layer perceptron) network to demonstrate how ﬁnetuning LR affects the ﬁnal performance by affecting dynamical isometry.
In brief, we observe the ﬁnetuning process can recover dynamical isometry; a larger LR can help recover it faster (or better), hence the better ﬁnal performance.
The proposed explanation is validated by our empirical results and resonates with many empirical observations.
Furthermore, by the explanation, we learn dynamical isometry recovery is rather imperative.
To achieve so, we present a very simple regularization-based method for pruning and show its effectiveness in recovering dynamical isometry on modern residual convolutional neural networks (CNNs).
Contributions .
(1) We empirically demonstrate the questioning about the value of inheriting weights in structured pruning in previous works is inaccurate and point out that the direct cause is improperly using a small ﬁnetuning LR.
Our ﬁnding justiﬁes the value of inheriting weights in structured pruning.
On top of the empirical ﬁnding, more importantly, we present a theoretical explanation through examining the dynamical isometry of networks in pruning.
This explanation is empirically validated by our carefully designed control experiments.
In addition to the theoretical understanding, we also propose a regularization-based method for dynamical isometry recovery.
Despite its brutal simplicity, it is shown effective to recover the broken dynamical isometry on modern residual convolutional neural networks.
R ELATED W ORK
Conventional pruning .
Pruning aims to remove as many parameters as possible in a neural network meanwhile maintaining its performance.
There are many ways to categorize pruning methods.
The most popular two are grouping by pruning structure and methodology.
(1) In terms of pruning structure, pruning can be speciﬁed into unstructured pruning (Han et al., 2015; 2016) and structured pruning (Wen et al., 2016;
Li et al., 2017; He et al., 2017).
For the former, a single weight is the basic pruning element.
Unstructured pruning can deliver a high compression ratio; whereas, without regularization, the pruned locations usually spread randomly in the network, which is hard to exploit for acceleration.
On the opposite, structured pruning introduces certain patterns in the pruned locations, which beneﬁt subsequent acceleration while cannot achieve as much compression.
Choices between unstructured and structured pruning depend on speciﬁc application needs.
For structured pruning, there are still many sub-groups (Mao et al., 2017).
In the literature, without speciﬁc mention, structured pruning means ﬁlter pruning or channel pruning.
This paper focuses on structured (ﬁlter) pruning because the “no value of inheriting weights” argument is mainly discussed in this context (Liu et al., 2019).
In terms of pruning methodology ( i.e. , how to select unimportant weights to prune), pruning falls into two paradigms in general: importance-based and penalty-based.
The former prunes weights based on some established importance criteria, such as magnitude (for unstructured pruning) (Han et al., 2015; 2016) or L 1 -norm (for ﬁlter pruning) (Li et al., 2017), saliency based on 2nd-order gradients ( e.g. , Hessian or Fisher) (LeCun et al., 1990; Hassibi & Stork, 1993; Theis et al., 2018;
Wang et al., 2019a; Singh & Alistarh, 2020).
The latter adds a penalty term to the objective function, drives unimportant weights towards zero, then removes those with the smallest magnitude.
Note, the two groups are not starkly separated.
Many methods take wisdom from both sides.
For example, (Ding et al., 2018; Wang et al., 2019b; 2021b) select unimportant weights by magnitude (akin to the ﬁrst group) while also employing the regularization to penalize weights (akin to the second group).
There is no conclusion about which paradigm is better, yet empirically, the state-of-the-art pruning methods are closer to the second paradigm, i.e. , deciding weights via training instead of some derived formulas.
Although no theories have formally discussed the reason, we can take a rough guess with the knowledge from this paper: Training can recover dynamical isometry (which is our ﬁnding in Fig. ?? ), which is beneﬁcial to subsequent ﬁnetuning.
For more comprehensive literature, we refer interested readers to several surveys: an outdated one (Reed, 1993), some recent surveys of pruning alone (Gale et al., 2019; Blalock et al., 2020) or pruning as a sub-topic under the general umbrella of model compression and acceleration (Sze et al., 2017; Cheng et al., 2018a;b; Deng et al., 2020).
Pruning at initialization (PaI) .
Recent years have seen several new pruning paradigms.
The most prominent one is pruning at initialization.
Different from the conventional pruning, which prunes a pretrained model, PaI methods prune a randomly initialized model.
Existing PaI approaches mainly include (Lee et al., 2019; 2020;
Wang et al., 2020; Frankle et al., 2021; Ramanujan et al., 2020) and the series of lottery ticket hypothesis (Frankle & Carbin, 2019; Frankle et al., 2020).
Interested readers may refer to (Wang et al., 2021a) for a comprehensive summary about PaI.
This topic is relevant to this work mainly because one PaI paper (Lee et al., 2020) also examines pruning using the tool of dynamical isometry.
The similarity between our paper and theirs is that we both employ dynamical isometry as a tool to examine the property of network pruning.
However, our paper is signiﬁcantly different from theirs in many axes: (1) Basic setting.
The most obvious difference is that we focus on pruning a pretrained model while (Lee et al., 2020) focuses on pruning at initialization (PaI).
They are two different tracks in pruning (as such, PaI methods typically do not compare with the methods of pruning pretrained models) and the latter was shown to consistently underperform the former (Frankle et al., 2021;
Wang et al., 2021a).
(2) Motivation.
Despite the same tool (mean JSV), (Lee et al., 2020) uses it to select unimportant weights to prune ( i.e. , for a new pruning criterion), while we use it to analyze why ﬁnetuning LR has a signiﬁcant impact on ﬁnal performance.
The role of ﬁnetuning LR in pruning is not mentioned at all in their paper.
Proposed technical method.
(Lee et al., 2020) focuses on unstructured pruning , while we focuses on structured pruning .
This further leads to fundamental difference when designing the dynamical isometry recovery (DIR) methods – In (Lee et al., 2020), their proposed method is to use iterative optimization for approximated isometry (due to the irregular sparsity); while in our case, since the pruned ﬁlers can be completely removed from the network, one of our DIR method (OrthP) has closed-form solution and can achieve exact isometry.
(4) Finally, in terms of empirical results, (Lee et al., 2020) only conducts experiments on MNIST (LeCun et al., 1998) and CIFAR (Krizhevsky, 2009), while we have extensive results on the large-scale ImageNet dataset (Deng et al., 2009).
2. E MPIRICAL S TUDY : L ARGER FINETUNING LR IS C RITICAL
As far as we know, mainly two papers question the value of inheriting weights from a pretrained model: (Crowley et al., 2018;
Liu et al., 2019).
Both papers draw two similar conclusions.
Inheriting weights from a pretrained model in pruning has no value, i.e. , training from scratch the small model can match (or outperform sometimes)
the counterpart pruned from a big pretrained model.
Given the fact of (1), what really matters in pruning may lie in the pruned architecture instead of the inherited weight values.
As such, both papers propose to view pruning as a form of neural architecture search (Zoph & Le, 2017; Elsken et al., 2019).
In this section, we ﬁrst reexamine the empirical studies in (Crowley et al., 2018;
Liu et al., 2019) to show that the “no value of inheriting weights” argument is actually inaccurate owing to improper ﬁnetuning LR schedules.
Reexamination of (Liu et al., 2019) .
Before presenting results, here are some important comparison setting changes worth particular attention:
(1) In (Liu et al., 2019), they compare training from scratch with six pruning methods (ﬁve structured pruning methods (Li et al., 2017; Luo et al., 2017;
Liu et al., 2017; He et al., 2017; Huang & Wang, 2018) and one unstructured pruning method (Han et al., 2015)).
Here, we only focus on the L 1 -norm pruning (Li et al., 2017) on ImageNet.
The main reason is that, L 1 -norm pruning is well-known a very basic ﬁlter pruning method.
If we can show it outperforms training from scratch already, it will be no surprise to see other more advanced pruning methods also outperform training from scratch.
In this sense, L 1 -norm pruning is the most representative method here for our investigation.
In (Liu et al., 2019), they have two variants for the number of epochs in scratch training, “Scratch-E” and “Scratch-B”.
For the former, different small models are trained for a ﬁxed number of epochs; for the latter, smaller models are trained for more epochs to maintain the same computation budget (Scratch-B was shown to be better than
Scratch-E in (Liu et al., 2019)).
Also, they decay LR only to 10 − 3 following the ofﬁcial PyTorch
ImageNet example 1 .
Here, we simply train all the networks for the same number of epochs but
[Table]
pruning (Li et al., 2017) on
We adopt the ofﬁcial torchvision models as
[Table]
ensure the epochs are abundant ( 120 epochs) and decay LR to a very small amount ( 10 − 5 ).
These two changes are to make sure the networks are trained to full convergence .
As we will show [Table] one primary cause possibly leading (Liu et al., 2019) to an inaccurate conclusion is exactly that the pruned networks are not fully converged (see Tab. 1)
[Table]
With the LR schedule changes
[Table]
we rerun the experiments using the released code of (Liu et al., 2019) .
Results are presented in Tab. 1.
In the implementations of (Liu et al., 2019), the ﬁnetuned model is outperformed by the scratch training one, hence their “no value of inheriting weights” argument.
We also reproduce their settings (the two rows of “ 20 epochs, initial 10 − 3 , ﬁxed” in “Our rerun”) for conﬁrming their argument.
However, the ﬁnetuning LR schedule “ 20 epochs, initial 10 − 3 , ﬁxed” is actually sub-optimal; the network is not fully converged.
Using the proper ones (“epochs, initial 10 − 3 , decay” or “ 90 epochs, initial 10 − 2 , decay”), pruning outperforms training from scratch for both ResNet-34-A and ResNet-34-B. (We note the pruned models even outperform the original models.
This is probably because pruning reduces the network redundancy, thus curbing overﬁtting.
This phenomenon is also widely observed in past pruning works (Han et al., 2016; Wen et al., 2016;
He et al., 2017) especially under small pruning ratios as in Tab. 1.)
Tab. 1 only presents two ResNet models and their speedups are actually quite small.
To see if the ﬁnetuning LR effect still holds across the full spectrum of pruning ratios and on other types of networks, we vary the pruning ratios from 0 .
1 to 0 .
95 and include experiments on VGG11 BN (Simonyan & Zisserman, 2015).
Results are presented in Tab. 2.
With a more proper ﬁnetune LR scheme (column “Pruned-Fintuned2” vs. “Pruned-Fintuned-1”), the performance can be improved signiﬁcantly .
A clear pattern is, the larger the pruning ratio, the more of the improvement.
Now, comparing the results of “PrunedFintuned-2” to those of “Scratch”, we can see pruning outperforms scratch-training in most cases.
Exceptions appear on ResNet-34/18 under extreme pruning ratios ( 90% and 95% ).
Despite them, we believe it is fair to say inheriting weights has value given the fact that 17 / 20 experiments in Tabs.
1 and 2 show pruning is better than training from scratch , especially under the pruning ratios of practical interest ( i.e. , non-extreme pruning ratios).
Retrospectively, (Liu et al., 2019) concluded oppositely because they faithfully re-implemented the L 1 -norm pruning method just according to the description in the original paper (Li et al., 2017): ﬁxed LR 10 − 3 , 20 epochs, which turns out far from optimal as we know now.
Reexamination of (Crowley et al., 2018) .
Coincidentally, (Crowley et al., 2018) adopted a very similar ﬁnetuning LR scheme to (Liu et al., 2019):
They ﬁnetuned the pruned network with the lowest LR ( 8 ∗ 10 − 4 , close to 10 − 3 in (Liu et al., 2019)) during scratch training and also ﬁxed .
Like the empirical study above, we reproduce the experiments of (Crowley et al., 2018) and rerun them with a larger initial LR ( 10 − 2 ) and decay it during ﬁnetuning.
Detailed results are deferred to the Appendix (Tab. 10) due to the limited length here.
We summarize the observation here – Exactly the same as the case in (Liu et al., 2019), when the proper ﬁnetuning
LR is used, pruning actually outperforms the best scratch training scheme consistently.
Up to now, the results above have shown that the “no value of inheriting weights” argument in previous works is largely attributed to sup-optimal ﬁnetuning settings.
A larger LR ( e.g. , 10 − 2 ) can signiﬁcantly improve the ﬁnetuning performance than a small one ( e.g. , 10 − 3 ).
In fact, we are not the only one to discover this.
Previous works (Renda et al., 2020; Le & Hua, 2021) also reported similar observation.
Nevertheless, they do not link the phenomenon with the “value of inheriting weights” argument and do not conduct systematical empirical studies as we do here.
More importantly, neither of them presented theoretical explanations about its occurrence – next, we are about to bridge this gap.
We present a faithful theoretical explanation through the lens of dynamical isometry.
D YNAMICAL I SOMETRY IS THE K EY 3. P REREQUISITE : D YNAMICAL I SOMETRY
Dynamical isometry (DI) is studied under the topic of trainability of deep neural networks.
It was ﬁrst brought up in (Saxe et al., 2014).
Speciﬁcally, the dynamical isometry is deﬁned as the singular values of the Jacobian matrix being around 1 (Saxe et al., 2014).
It is easy to see that the networks with dynamical isometry is easy to train, because JSVs around 1 imply the gradient signals will not be ampliﬁed or attenuated seriously during propagation, preventing the network from gradient exploding or vanishing, which are well-known the main difﬁculties in deep network training (Glorot & Bengio, 2010; Sutskever et al., 2013).
For linear networks, dynamical isometry can be achieved exactly by the orthogonal initialization proposed in (Saxe et al., 2014); while for neural networks with non-linearity (like ReLU (Nair & Hinton, 2010)) and convolution, it can only be approximated up to date (see Tab. 3).
Mean Jacobian singular values .
Since dynamical isometry is measured by the Jacobian singular values (JSV’s), we adopt the mean of Jacobian singular values (denoted by ¯ S ) as a scalar metric for analysis.
Speciﬁcally, for a Jacobian J ∈ R C × D in ( C stands for the output dimension, i.e. , the number of classes, D in for the input dimension), apply singular value decomposition (Trefethen &
[Table]
Bau III, 1997) to it,
[Equation]
where Σ is the singular value matrix and K = min( C, D in ) .
Pruning as poor initialization .
We investigate how the pruning affects ¯ S .
The results are shown in
Fig.
As seen, the mean JSV is consistently damaged by pruning; and a larger pruning ratio, more decrease of the mean JSV.
This means, pruning actually servers as a very poor initialization scheme for the subsequent ﬁnetuning.
In stark contrast to the broad awareness that initialization is rather critical to neural network training (Glorot & Bengio, 2010; Sutskever et al., 2013; Mishkin & Matas, 2016;
Kr ¨ ahenb ¨ uhl et al., 2015; He et al., 2015), the initialization role of pruning has received negligible research attention, however.
As far as we know, no prior works have noted this issue when pruning a pretrained network or tried to recover the broken dynamical isometry before ﬁnetuning.
Simply put, our goal next is to show it is this broken dynamical isometry that answers for the performance gap between LR 10 − 2 and 10 − 3 .
If dynamical isometry is fully recovered before ﬁnetuning, LR 10 − 2 vs. 10 − 3 should not cause signiﬁcant performance difference anymore.
To validate this explanation, we need a method to fully recover dynamical isometry in ﬁlter pruning, as is introduced next.
3. D YNAMICAL I SOMETRY R ECOVERY IN F ILTER P RUNING
In (Saxe et al., 2014), they propose a weight orthogonalization scheme to achieve dynamical isometry for neural network initialization , namely, the initial weights are randomly sampled .
Different from their case, here the initial weights are inherited from a pretrained model by pruning.
Therefore, we need to adapt it to our application.
For a fully-connected layer parameterized by a matrix W 0 ∈ R J × K (for a convolutional layer parameterized by a 4-d tensor of shape R N × C × H × W , it can be reshaped to a matrix of shape R N × CHW ), it reduces to matrix W of size R J 1 × M 1 ( J 1 ≤ J, K 1 ≤ K ) after structured pruning.
Then, we apply the weight orthogonalization technique (Mezzadri, 2006) based on QR decomposition (Trefethen &
Bau III, 1997) to W ,
[Equation]
where qrd( · ) means QR decomposition; Q is an orthogonal matrix of the same size as W ( R J 1 × K 1 );
R is an an upper triangular matrix of size R K 1 × K 1 ; sign( · ) is the sign function which returns the positive or negative sign of its argument; (cid:12) represents the Hadamard (element-wise) product aligned to the last axis (since Q and sign(diag(R)) share the same dimension at the last axis).
As an orthogonalized version of W , W ∗ recovers the dynamical isometry damaged by pruning.
Therefore, we propose to employ W ∗ instead of the original W as the initialization weights for later ﬁnetuning.
We dub this weight orthogonalization method for pruned models as OrthP .
With this method, we can fully recover broken dynamical isometry and continue our analysis as follows.
[Table]
[Table] [Table] A NALYSIS WITH MLP-7-L INEAR ON MNIST
Evaluated network .
The network for analysis is a 7-layer linear MLP.
We are aware that this toy network has little practical meaning, but it is very appropriate here for two reasons.
First, as mentioned above, in our analysis we need a method to recover DI exactly .
Up to date, this can only be achieved on linear networks (see Tab. 3).
Second, the linear MLP network is free from the intervention of modern CNN features ( e.g. , BN (Ioffe & Szegedy, 2015), residual (He et al., 2016)).
By our observation, these features will make the problem complex and prevent us from seeing consistent results at the early analysis stage.
LR schedule setup .
When we set different LR schedules with different initial LRs, we will (1) keep the total number of epochs the same, (2) keep the last LR the same, (3) intentionally use prolonged training epochs (like 90 or even 900 epochs on MNIST dataset.
Typically, it only needs 30 epochs to reach convergence on MNIST).
All of these are to ensure the networks are fully converged, helping us render a faithful conclusion.
Step decay LR schedule is employed given its broad use.
Proposed explanation and its deducted hypotheses .
We list the ﬁrst 10-epoch mean JSV and test accuracy of pruning MLP-7-Linear (at pruning ratio 0 . 9 ) under different ﬁnetuning setups in Tab.and Tab. 5, respectively.
The following observations are worth our attention.
• In Tab. 4, one important fact is that, the mean JSV can recover itself without any extra help during ﬁnetuning, regardless of different setups.
• In Tab. 4, without OrthP, the mean JSV of LR 10 − 2 arises much faster than that of LR 10 − 3 . By our analysis in the Appendix (Sec. D), a larger mean JSV implies better dynamical isometry, which further implies easier optimization. Easier optimization ﬁnally leads to the better test accuracy of LR 10 − 2 against 10 − 3 when the training epoch is insufﬁcient, as shown in Tab. 5.
• In Tab. 4, with OrthP, the broken dynamical isometry is exactly recovered (note at epoch 0, the mean JSV is 1).
Then the accuracy advantage of LR 10 − 2 over 10 − 3 is not signiﬁcant anymore, e.g. , in Tab. 5, at epoch 10, LR 10 − 2 is better than LR 10 − 3 by 16 . 05% without OrthP; while it is not better (actually worse) than LR 10 − 3 with OrthP. • Particularly note how the mean JSV trend in Tab. 4 correlates well with the test accuracy trend in Tab.
This promotes the plausible idea that it is dynamical isometry that answers for the accuracy gap fundamentally.
These observations inspire us to the following plausible explanation about how a larger ﬁnetuning LR can improve performance signiﬁcantly: A larger ﬁnetuning LR helps the network update faster, thus the dynamical isometry (measured by mean JSV) recovers faster (and possibly better), which further leads to faster (and possibly better) optimization.
For deep networks nowadays, better optimization typically implies better generalization, thus the larger ﬁnetuning LR eventually leads to the better test accuracy as we see. [Table] [Table]
[Table] [Table] [Table] That is, a larger ﬁnetuning LR shows performance advantage only if dynamical isometry is broken by pruning ﬁrst .
If dynamical isometry is fully recovered before ﬁnetuning, a larger LR should not pose performance advantage anymore.
The validation of this explanation can be speciﬁed into the following 4 deducted hypotheses:
• Hypothesis 1 :
Given a small number of epochs, mean JSV cannot be fully recovered by training, then the larger LR should show a signiﬁcant advantage over the smaller LR. • Hypothesis 2 : With sufﬁcient training epochs, mean JSV can be fully recovered by training.
Then the larger LR should have less advantage over the smaller LR. • Hypothesis 3 : If we employ OrthP to exactly recover the mean JSV, given the small number of epochs again, the larger LR should have much less advantage now.
• Hypothesis 4 : If we combine abundant epochs with OrthP, mean JSV will be recovered even completely, then the performance advantage of the larger LR over the smaller one should be even weaker.
Corresponding to these four hypotheses, the eight ﬁnetuning LR settings are summarized in Tab. 15.
The unpruned MLP model is trained with LR schedule “ 90 epochs, initial 10 − 2 , decay 30 / 60 ”.
For ﬁlter pruning, we employ L 1 -norm pruning (Li et al., 2017) throughout this paper.
Speciﬁcally, it sorts the neurons (or ﬁlters) by their L 1 norms in ascending order and prunes those with the least norms by a predeﬁned pruning ratio r .
The ﬁnal accuracy results are shown in Tab.
We ﬁrst analyze the results of pruning ratio 0 . 8 in Tab. 7.
As seen, when ﬁnetuned for 90 epochs, LR 10 − 2 shows an advantage over LR 10 − 3 by 0 . 82% accuracy.
It is tempting to draw a conclusion based on this comparison that LR 10 − 2 is much better than LR 10 − 3 .
However, this is not the whole story:
• With 900 epochs, LR 10 − 2 is greatly surpassed by LR 10 − 3 (91.64 vs. 92.54).
The reason by our proposed explanation is that, with abundant epochs, the dynamical isometry can be recovered more completely, hence LR 10 − 2 does not show advantages anymore over 10 − 3 .
• When OrthP applied, LR 10 − 2 does not show signiﬁcant advantages either, similar to the effect of increasing the number of training epochs.
This is because that ﬁnetuning shares the same role of recovering dynamical isometry with OrthP. Just OrthP is more effective since it is analytically targeting exact dynamical isometry.
• When the best setting used (OrthP + 900 epochs), LR 10 − 3 is slightly better than 10 − 2 .
Comparing “OrthP, 900 epochs” with “OrthP, 90 epochs”, the gains are only marginal.
This is because the dynamical isometry has already been fully recovered by OrthP, thus more training epochs do not show much value anymore.
A different pruning ratio 0 .
9 is also explored.
Its results are in line with those of pruning ratio 0 .as shown in Tab.
In short, these empirical observations are fully in line with our expectations, justifying the validity of the proposed explanation.
Explaining LR effect on ImageNet .
In Tab. 2, there is an apparent trend that the larger pruning ratio, the more performance advantage of LR 10 − 2 over 10 − 3 (this also appears on CIFAR10 dataset, see Tab. 8).
Using our explanation, this phenomenon can also be explained now – When the pruning ratio is greater, more dynamical isometry is damaged.
LR 10 − 2 can ﬁnd more use in these cases since it is faster/better to recover dynamical isometry, hence the more pronounced advantage.
A closer look and more lessons .
In the ResNet34 results above (Tab. 1) reported by (Li et al., 2017) and (Liu et al., 2019)), it is still easy to notice their ﬁnetuning LR schedule may be sub-optimal given [Table] their ﬁnetuning process ( 20 epochs) is obviously shorter than training from scratch ( 90 epochs).
For
Tab. 7, quite differently, the “ 90 epochs” ﬁnetuning setting appears nothing wrong (considering the unpruned network is trained for 90 epochs, ﬁnetuning for 90 epochs is not short; besides, the LR is also decayed to a very small amount).
However, the results of this setting actually lead us to a partial conclusion that the larger LR is better than the smaller LR.
This particular example shows how misleading the comparison results can be, even though they look perfectly fair, if we are not aware of the effect of dynamical isometry in structured pruning .
D YNAMICAL I SOMETRY R ECOVERY (DIR)
By our analysis above, dynamical isometry recovery before ﬁnetuning is rather important.
An effective dynamical isometry recovery method is supposed to (1) close up the performance gap between ﬁnetuning LR 10 − 2 and 10 − 3 , (2) improve the ﬁnal performance (otherwise, there is no point using it at all).
As shown in Tab. 7, the proposed OrthP method for linear MLP networks can meet these two requirements; and since it recovers DI exactly, it actually completely closes the performance gap between ﬁnetuning LR 10 − 2 and 10 − 3 .
Unfortunately, this method does not generalize to more practical non-linear convolutional neural networks, as shown in Tab. 8, where using OrthP degrades the ﬁnal performance.
As a remedy, in this section we introduce a very simple regularization-based method that can work on practical CNNs.
Strong L 2 regularization as a drop-in remedy .
In a typical structured pruning algorithm, unimportant ﬁlters are selected by some criterion.
Then they are removed (zeroed out), followed by a ﬁnetuning process.
We propose to apply a super strong L 2 regularization ( e.g. , regularization factor equals to 1 ) to push the unimportant ﬁlters to rather close to zero ﬁrst, before permanently removing them.
This simple technique can readily work as a drop-in step into any structured pruning algorithm and it is very easy to implement on any deep learning framework.
In Appendix (Sec. E), we provide more explanation regarding how StrongReg is related to dynamical isometry.
Results in Tab.
8 demonstrate the effectiveness of this simple technique on ResNet56.
It improves the ﬁnal pruning performance and also shrinks the performance gap between ﬁnetuning LR 10 −and 10 − 3 (especially when the pruning ratio is large), meeting the two requirements above.
We are aware that this technique is preceded by several variants in the past literature ( e.g. , GReg-1 in (Wang et al., 2021b)), but they do not link their method with dynamical isometry recovery as we do here.
Also, our technique is even simpler than theirs – in (Wang et al., 2021b), L 2 regularization is increased gradually to a very strong level, while here we use a ﬁxed strong regularization, which probably is the simplest form of utilizing a strong regularization.  R ECTIFIED A RGUMENT ON V ALUE OF P RUNING
When addressing concerns from the reviewers, we ﬁnd a phenomenon against our “pruning has value” argument.
Speciﬁcally, in Tab. 7, we provide the scratch training results with two random
[Table] [Table] [Table] initialization schemes, kaiming uniform and orthogonal initialization.
The former is the default
PyTorch initialization scheme for convolutional and linear layers 2 ; the latter is proposed by Saxe et al.
(2014) which can achieve exact isometry.
The problem is, the best pruning results in Tab.using OrthP are generally on par with the scratch training results using orthogonal initialization.
Namely, pruning is shown no value therein.
How should we respond to this observation against our prior claim and how does it affect our conclusions?
We present another set of results – We use MLP-7-ReLU as evaluation network instead of MLP-7Linear above.
Clearly, here we want to see if the non-linearity can make any difference.
Results are shown in Tab. 9, where the dynamical isometry recovery method is changed from OrthP to
StrongReg because OrthP is proposed for linear MLP networks while here the MLP is non-linear.
[Table]
As seen, now, either using kaiming uniform or orthogonal initialization, pruning is consistently better than scratch training by a fair margin.
The advantage is ampliﬁed when using StrongReg.
Then, a worthy question here is: Why on the MLP-7-Linear, pruning is not better than orthogonal initialization, while on MLP-
-ReLU
it is
This is actually straightforward to see if we see pruning as a kind of initialization .
As mentioned above, on linear MLP networks, orthogonal initialization is proven to be the optimal in the sense of exact isometry.
That is, no other initialization can be better.
Pruning is essentially also a kind of initialization for the subsequent ﬁne-tuning, so not surprisingly, it cannot beat the (optimal) orthogonal initialization, hence no value.
However, when it comes to the MLP-7-ReLU network, orthogonal initialization is no longer optimal (as mentioned, there has been no method up to date that can achieve exact isometry for non-linear networks).
Then, it is likely that pruning provides better initialization weights than orthogonal initialization or kaiming uniform.
It just turns out that pruning really achieves this, and archives more with the help of StrongReg.
There are at least 3 key takeaways from the above toy but inspiring comparisons:
• First, the comparison above shows that whether pruning has value seriously hinges on the network type, initialization scheme, and pruning ratios (and maybe more) .
Looking at pruning as a kind of initialization as we do in this paper (and examine it through the lens of dynamical isometry) is actually a pretty good perspective, from which many results that appear inconsistent or random start to be explainable and logically consistent.
• Second, pruning shows the value in the non-linear case, but no value in the linear case.
On the whole, we believe it is fair to say pruning has value considering the non-linear case is more practical.
• Third, more profoundly, the above results actually suggest, if dynamical isometry is fully recovered, pruning will (probably) have no value indeed because it cannot beat the initialization scheme that can achieve exact isometry .
Acquired with this knowledge, we update our prior “pruning has value” argument to a more rigorous one: “ pruning has the potential to be valuable if the random initialization of scratch training cannot achieve exact isometry ”.
Practically, up to date, for non-linear networks (not to mention BN, residuals, convolution), there has been no such method that can achieve exact isometry.
Thus, it is still likely for pruning to be valuable at present.  C ONCLUSION In this work, we present extensive empirical evidences to show the “no value of inheriting weights” argument in prior works is inaccurate because of improper ﬁnetuning LR schedules.
We further tap into dynamical isometry to explain why the ﬁnetuning LR has such a great impact on the ﬁnal performance, through carefully designed control experiments with 4 hypotheses. We show whether pruning has value seriously depends on the context ( e.g. , network type, random initialization scheme, pruning ratio).
Looking at pruning as a kind of initialization is a favorable perspective that can make seemingly inconsistent and random results become predictable and coherent.
For practically amending the broken dynamical isometry, we also present a rather simple regularization-based technique that works effectively on residual convolutional networks.
The ﬁnding of dynamical isometry in structured pruning in this paper justiﬁes the value of inheriting weights, in line with the past research wisdom and our common beliefs.
It also helps us towards a better understanding of pruning and possibly can inspire more advanced pruning algorithms as dynamical isometry recovery has been shown a worthy direction in the paper.
In addition, the awareness of dynamical isometry in structured pruning can help us render a more faithful conclusion when comparing different pruning methods.
A R ESULTS OF REEXAMINATION OF (C ROWLEY ET AL ., 2018)
Dataset: CIFAR-10 . Results above the dashline are directly cited from (Crowley et al., 2018); results below the dashline are from our reproducing (with the ofﬁcial code of (Crowley et al., 2018) at https://github.com/BayesWatch/pytorch-prunes for fair comparison). “Rerun” means we rerun the code of (Crowley et al., 2018) as it is.
“LR 10 − 2 ” means we redo the ﬁnetuing for the pruned models in “Rerun” using our ﬁnetuning LR schedule ( 120 epochs, initial 10 − 2 , decay 60 / 90 ). Finetuning is randomly repeated for 3 times, mean (stddev) accuracies reported. The main point here is that (Crowley et al., 2018) draws the conclusion that scratch training is better than pruning because of an improper ﬁnetuning LR scheme . With the proper ﬁnetuning LR scheme, pruning is actually better than scratch training
[Table]
B T RAINING S ETTING S UMMARY
There are three datasets in our experiments in the paper: MNIST, CIFAR [Table] and ImageNet.
Apart from some key settings stated in the paper, a more detailed training setting summary is shown as
Tab. 11.
We use 8 NVIDIA V100 GPUs for all our experiments. [Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
A typical ﬁnetuning LR schedule looks like this in our paper: “ 90 epochs, initial 10 − 2 , decay 30 / 60 ”.
Its meaning is: the total number of training epochs is 90 ; initial LR is 10 − 2 ; at epoch 30 and 60 , LR is multiplied by a default factor 0 .
The others can be inferred likewise.
C W HY OUR REPRODUCED RESULTS ARE BETTER THAN (L IU ET AL ., 2019)
In Tab. 1, we present the results of ResNet34 on ImageNet, pruned by the L 1 -norm pruning method (Li et al., 2017).
Readers may be curious that why our scratch-training results are much higher than those of Rethinking (Liu et al., 2019) (by 0 .
This has a historical reason.
(Liu et al., 2019) 3 refers to the ofﬁcial PyTorch ImageNet example 4 , where the LR is only decayed twice (from 1e-1 to 1e-2, then to 1e-3).
However, by our empirical observation, 1e-3 is still not the lowest
LR that the network converges.
If the LR is decayed another time (to 1e-4), the top-1 accuracy can still bump by round 0.5-0.8% point, which is a signiﬁcant improvement for ImageNet thus not negligible.
Therefore, we choose to decay the LR further to 1e-4 and ﬁnally to 1e-5 to ensure the network is fully converged .
This reiterates the comparison rule of our paper: comparing different methods in their best shape .
Comparing scratch training to L 1 -norm pruning before the ﬁnetuned network ﬁnally converges may appear fair (given the same number of epochs) but is of less practical meaning and may hide the true picture. D H OW TO Properly LOOK AT THE M EAN JSV M ETRIC FOR D YNAMICAL I SOMETRY
In the main paper, we use mean JSV as the metric to measure dynamical isometry, for the following 3 speciﬁc reasons: (1) It was used by Lee et al.
(2020), which analyzes the dynamical isometry for randomly initialized network.
Given its success there, it is very natural for us to also employ this metric for analyzing pretrained networks here.
(2) We currently do not have a better alternative, either.
In practice, we ﬁnd mean JSV is informative ( e.g. , in Tab. 7, the mean JSV trend is well-correlated with the test accuracy trend), as long as we see it properly .
This section is meant to provide more background regarding how to look at the mean JSV for dynamical isometry properly .
It is not a new invention of our paper but a general practical guideline about the relationship between mean JSV and dynamical isometry in order to help readers better understand our paper.
DI (dynamical isometry) is deﬁned by mean JSV close to 1 in Saxe et al.
Rigorously, in Saxe et al.
, DI describes the distribution of all JSVs.
Mean JSV is only an average sketch of the distribution.
Nevertheless, this average approximation is accurate enough for analysis.
In other words, if a network has mean JSV close to 1, we can say this network has dynamical isometry .
Then, a non-trivial technical question is: When we deal with practical DNNs in the real world, how close is the so-called “close to 1”?
To our best knowledge, there is no outstanding theory to quantify this, so we resort to empirical analysis, speciﬁcally on the MLP-7-Linear network used in the main paper.
[Table]
As seen, there is a clear trend in Tab.
12: larger pruning ratio, smaller mean JSV, and lower test accuracy (either before or after ﬁnetuning).
Particularly note the mean JSV range where the pruned network can be ﬁnetuned back to the original accuracy ( 92 . 77% ) , which is > = 0 .
This means, for networks with mean JSV greater than 0 .
0151 , in spite that their immediate accuracies (without ﬁnetuning) can be distinct ( e.g. , 91 .
35% at
PR 0.1 vs. 9 .
74% at PR 0.7), intrinsically, they are equivalently potential after ﬁnetuning.
DI theory suggests that mean JSV equal to 1 is the best case.
Then we would ask, how about mean
JSV equal to 2 ? 1 . 5 ? 0 .
Are they really worse than the ideal value 1 ?
Tab.
12 above shows not necessarily , because a network with mean JSV 0 .
2711 can reach comparable accuracy to the network with mean JSV 1 .
Only when the mean JSV is smaller than some threshold (in this case,
0 . 0151 ), it leads to an irrecoverable damage to the network optimization, which eventually results in lower generalization ability ( i.e. , test accuracy).
By this “trainable with equivalent potential” rule, if a mean JSV lies in the range of > = 0 .
0151 , we can regard it as “close to 1” because they can do just as well as 1 .
Here, we only mention the lower bound, how about the upper bound?
Can mean JSV 10 , 000 also be regarded as “close to 1”?
This is a good question worth further dedicated investigation.
{+Yet, for now, we do not need to worry much about it, because in practice, a normally trained network rarely presents a very large mean JSV by our empirical observation, but is quite likely to have a very small mean JSV ( 0 . 
+}
at pruning ratio 0 .
9 in Tab.
12 is a concrete example).
E H OW S TRONG R EG IS R ELATED TO D YNAMICAL I SOMETRY
The intuition behind StrongReg .
Dynamical isometry describes a nice state of the network that signals can propagate through it without serious magnitude explosion or attenuation.
Since the weights in a network are dependent on each other, removing some of them will deﬁnitely hurt the isometry because it is based upon all the weights.
When we use a strong regularization to push these unimportant parameters to zero, it explicitly makes the other parameters learn to not rely on them, that is, encouraging the gradients not to pass through these weights/neurons because they are going to be cleaned out.
This way, when the unimportant parameters are physically removed, it will incur much less damage to the left parameters, thus maintaining the network dynamical isometry well.
StrongReg works by improving dynamical isometry .
Here we apply StrongReg to pruning the
MLP-7-Linear network at pruning ratio 0.8 and 0.9.
randomly
5 times, mean accuracy and stddev reported.
“Acc. gain” refers to the mean accuracy improvement of LR 10 − 2 over 10 − 3 .
Hyper-parameters: batch size 100 , SGD optimizer, weight decay 0 . 0001 , LR schedule: initial LR 0 .
01 , decayed at epoch 30 and 60 by factor 0 .
1 , total epochs: 90 .
The LR schedule is used for both scratch training and ﬁnetuning.
[Table]
As seen in Tab. 13, (1) similar to OrthP, StrongReg can rectify the test accuracy from the underrated ones to 92 .
48 ∼ 92 .
80 , close to the best possible performance (around 92 . 77 ); (2) StrongReg can close the performance gap between LR 10 − 3 and 10 − 2 , just like OrthP.
In short, in terms of performance, StrongReg behaves very similarly to OrthP.
To further show StrongReg really works by maintaining dynamical isometry , like Tab. 4 and Tab.above, we list the ﬁrst 10-epoch mean JSV and test accuracies (at PR 0.9) and get the following summarized results: mean JSV isbecause OrthP can achieve exact isometry
[Table]
[Table]
(1) First, in Tab. 14, at epoch 0, the mean JSV is around 3 (which can be considered close to the exact isometry 1, by our analysis in Sec.
D) vs. 0 .
when the dynamical isometry is not recovered.
Second, the test accuracy also has a high starting point: at epoch 1, using StrongReg achieves 91 .(LR 10 − 2 ) and 92 .
03 (LR 10 − 3 ) vs. 63 .
86 (PR 10 − 2 ) and 9 .
74 (PR 10 − 3 ) when the dynamical isometry is not recovered.
Particularly note StrongReg can achieve similar mean JSV and test accuracy to OrthP, showing that
StrongReg works by maintaining dynamical isometry, as OrthP does.