Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm
Abstract
Pruning techniques have been successfully used in neural networks to trade accuracy for sparsity.
However, the impact of network pruning is not uniform: prior work has shown that the recall for underrepresented classes in a dataset may be more negatively affected.
In this work, we study such relative distortions in recall by hypothesizing an intensiﬁcation effect that is inherent to the model.
Namely, that pruning makes recall relatively worse for a class with recall below accuracy and, conversely, that it makes recall relatively better for a class with recall above accuracy.
In addition, we propose a new pruning algorithm aimed at attenuating such effect.
Through statistical analysis, we have observed that intensiﬁcation is less severe with our algorithm but nevertheless more pronounced with relatively more difﬁcult tasks, less complex models, and higher pruning ratios.
More surprisingly, we conversely observe a de-intensiﬁcation effect with lower pruning ratios, which indicates that moderate pruning may have a corrective effect to such distortions.
Introduction
Back in the old days, network pruning was important to reduce the size of neural networks [23, 53, 33, 41, 24, 25].
With new advances being increasingly reliant on overwhelmingly large and costly models [3, 64, 10], network pruning is back to the game despite the gains in computing power.
Network pruning reduces the complexity of large models by setting a substantial number of parameters to zero and then using gradient descent to ﬁne-tune the sparser model.
There is typically a trade-off between making the model sparser—i e., having fewer parameters, since those set to zero can simply be discarded—and keeping the model as accurate as possible [9], although moderate gains in sparsity may be obtained with little impact to accuracy [28].
However, recent studies have found that the side effects of network pruning on model performance are not evenly distributed [30, 55, 31, 1, 63, 69].
Those studies have considered the impact of network pruning on recall—i.e., the number of correct predictions per class—based on dataset representability.
Namely, they have found that the degradation in performance is inﬂuenced by the uneven representation of classes [30, 55, 69] and features [31] as well as class complexity [55].
Hence, they corroborate long-standing concerns that unbalanced datasets yield models that are less accurate in and potentially harmful to minoritized groups [11, 7].
In this work, we complement those prior studies by posing recall distortion as inherent to network pruning even when the networks are trained and ﬁne-tuned on datasets that are seemingly balanced; hence investigating when and how such distortions manifest, as well as how to reduce their effect.
More speciﬁcally, this paper presents the following contributions:
(i)
We conduct a statistical study of model-level distortion to investigate how recall is affected by network pruning due to pruning ratios, dataset and model complexity, and pruning algorithms.
(ii)
We observe an intensiﬁcation effect, meaning that for classes with recall below accuracy we observe the relative difference between recall and accuracy negatively widening if the network is sufﬁciently pruned. For classes with recall above accuracy, we conversely observe that gap positively widening.
The intensiﬁcation correlates with excessive pruning ratios as well as more complex data and models, and is more pronounced with some pruning algorithms.
(iii) More surprisingly, we observe that otherwise network pruning has a corrective effect on recall differences, hence implying a de-intensiﬁcation effect under moderate pruning.
(iv) We introduce a new gradient-based pruning method for networks trained with weight decay, Undecayed Pruning, which attenuates the intensiﬁcation effect observed with other methods.
Related work
The use of large models has been justiﬁed and further encouraged by ﬁndings that the overparameterized regime may avoid the classic bias–variance trade-off [83, 6] and lead to better convergence during training [44, 66].
However, these models come with steep environmental footprint and hardware needs [64].
Not surprisingly, they represent a relevant application of network pruning [20, 78].
Network pruning has been motivated by parameter redundancy in models [13] and found to improve generalization [4] and robustness against adversarial manipulation [74, 35].
The amount of pruning that is tolerable by an architecture may depend on the task in which it is trained [46].
As observed on a recent survey [9], most approaches to network pruning are guided by either (i) removing the parameters with smallest absolute value [23, 53, 33, 22, 21, 43, 17, 15, 20, 68, 48]; or (ii) removing the parameters with smallest impact on the model [41, 24, 25, 39, 52, 14, 79, 82, 5, 42, 71, 47, 72, 76, 61, 80].
We may regard exact pruning as a special case of (ii) in which the model is not affected [59, 62, 60, 18].
Alternative approaches to network pruning include quantization [32, 50, 2], knowledge distillation [37, 45, 84, 73, 70, 67, 34, 77], and the use of regularization during training [81, 16, 56].
In this work, we contribute to the study of recall distortion due to model compression [30, 55, 31, 1, 63], which a particular focus on network pruning.
Concurrent to our work, Tran et al.
[69] framed these recall distortions in unbalanced datasets as a Matthew effect that can be attributed to model (Hessian loss and gradient ﬂows) as well as data characteristics (input norms and distance to decision boundary).
Pruning distortion has also been studied under other metrics, such as by focusing on the samples for which the classiﬁcation is affected [30, 31, 36, 78] and the relationship between false positive and negative rates [8]. The issue is also studied in the context of other compression techniques such as knowledge distillation [27], in which some approaches to remedy this issue are focused on adjusting the loss function used in the compressed model [36, 78].
Such disproportional impact on recall across classes relates to fairness in machine learning.
The lack of representativeness and the biased context in which data is collected as well as the lack of transparency may lead to models making life-altering decisions that negatively affect minoritized groups, such as in criminal justice [54, 11, 57].
These concerns have motivated an extensive discussion on fairness in machine learning [12] and on the proper assessment of datasets [19], models [51], and the circumstances in which they are applied [58].
In this work, we focus on studying the algorithmic bias [49, 29] of network pruning, that is, the bias that is either (i) not present in the input data nor the original model; or (ii) deteriorated or relieved by pruning.
By subscribing to this line of study, our work aims at reducing the potential negative societal impacts associated with network pruning.
Pruning algorithms
We propose Undecayed Pruning (UP) by considering the interplay between the classic representatives of pruning methods described in Section 2: Magnitude Pruning (MP) and Gradient Pruning (GP).
Magnitude Pruning MP is a simple but rather (mysteriously) effective technique of selecting among the parameters θ = ¯ θ the one with smallest absolute value to be pruned next from the network:
[Equation]
A parameter having a smaller value may not necessarily imply lesser importance to the model [24] (hence the mystery part).
However, it is worth noticing that this technique is usually applied to networks that were trained with regularization.
Since we cannot add L 0 regularization to the loss function for directly minimizing the number of nonzero parameters as this would make the loss function not differentiable for gradient descent, we can use L 1 or L 2 as a proxy, which induces the values of the parameters to be as small as possible.
Although no parameter ends up being zero, some parameters become so small that making them equal to zero implies an almost negligible change.
Gradient Pruning GP approximates the impact of modifying the parameters of a neural network by estimating the ﬁrst-order variation of the loss function L on the training set between the trained parameters θ = ¯ θ to a new set of parameters θ through the Taylor series around θ = ¯ θ :
[Equation]
By assuming that O ( (cid:107) θ − ¯ θ (cid:107) 2 ) {+≈ 
+} for small changes, we estimate the change to the loss function by pruning a single parameter i —i.e., with ˜ θ such that ˜ θ i = 0 and ˜ [-✓ j = ¯ ✓ j for j 
-]θ
j = ¯ θ j for j (cid:54) =
i —as follows:
[Equation]
Given the approximate nature of estimating the impact on the training set, which may not necessarily reﬂect on the test set, small absolute perturbations are often preferred to negative but larger variations.
Hence, the choice of the parameter i to be pruned next is effectively framed as follows:
[Equation]
Undecayed Pruning We aim to address what we believe is a double-edged approach to reducing the number of parameters. Namely, that it is common to use a term in the loss function to make the weights as small as possible, such as weight decay, and then that we use the gradient of the loss function for pruning weights while ignoring that the gradient is also affected by the weight decay term.
We understand that simultaneously using regularization and GP is conﬂicting because sparsity is induced in two different ways.
However, whereas GP has a more direct proxy to model impact, using some amount of regularization tends to be beneﬁcial during training.
Hence, we approach this by isolating the effect of regularization from the variation of the loss function.
Due to its greater popularity, we consider L 2 regularization–i.e„ weight decay–in what follows.
Let us denote by T the loss function of a model after deducting weight decay with hyperparameter ε :
[Equation]
With ∇ i T ( θ ) = ∇ i L ( θ ) − εθ i , we estimate the change to the alternative loss function T by pruning a single parameter i —i.e., with ˜ θ such that ˜ θ i = 0 and ˜ θ j = ¯ θ j for j [-
-](cid:54) =
i —as follows:
[Equation]
In other words, discounting weight decay is equivalent to a speciﬁc balance of the criteria for MP and GP: the ﬁrst term is equivalent to GP; the second term is curiously equivalent to MP; and the latter is prioritized in proportion to the weight decay hyperparameter ε .
Moreover, if the neural network training converges to a local optimum, in which case ∇L (¯ θ ) = 0 , then UP is equivalent to
MP.
Similar to the case of GP, we choose the parameter i to prune based on the absolute impact on T :
[Equation]
The way that weights are now ranked reﬂects a weighted combination of the criteria used for MP and
GP.
This is particularly interesting because UP coincides with MP when the gradient is sufﬁciently close to zero, hence providing a principled argument for the effectiveness of MP under weight decay.
Model properties
Let A ( m ) be the accuracy of an unpruned model m as measured on the test data.
In other words,
A ( m ) is the number of correct predictions divided by the number of samples.
For a pruning ratio t , meaning that t is the number of parameters before pruning divided by the number of parameters after pruning, let A t ( m ) denote the accuracy of model m on the test data after pruning. For simplicity, we may assume that t = 1 if t is absent from the notation and omit m if always referring to the same model, and thus we may assume that A = A ( m ) = A 1 ( m ) .
The same applies to other metrics. Similarly, let R c ( m ) be the recall for class c of an unpruned model m on the test data.
In other words,
R c is the number of correct predictions for class c divided by the number of samples for class c .
For a pruning ratio t , let R ct ( m ) denote the recall for class c of model m on the test data after pruning.
We are particularly interested in the how these metrics differ for each class.
Namely, let
[Equation]
denote the recall balance 3 .
When B ct ( m ) > 0 , we say that model m at pruning ratio t overperforms for class c ; and when B ct ( m ) < 0 we may say that m underperforms for class c .
Finally, let
[Equation]
denote the normalized recall balance .
The further away this value is from 1, the more pronounced is the difference in performance between class c and the other classes in model m at the pruning ratio t .
Proposition 4.1.
If a dataset with set of classes C is balanced , meaning that each class c ∈ C has the same number of samples, then for any model m and pruning ratio t it holds that
[Equation]
Proof.
If the dataset is balanced, then it follows that A t = (cid:80) c
∈ C R ct | C | and hence (cid:80) c ∈ C R ct =
| C |
A t .
Therefore, 0 = (cid:80) c ∈ C ( R ct ) − | C |
A t = (cid:80) c ∈ C ( R ct − A t ) = (cid:80) c ∈ C B ct .
Corollary 4.1.1.
If a dataset is balanced, then for any model m and pruning ratio t it holds that (cid:88) c ∈ [Equation] ct
[Equation]
Proof.
Immediate from dividing (cid:80) c
∈ C B ct ( m ) by A t ( m ) due to Proposition 4.1.
Example Consider a model in which A = 80% with two classes X and Y such that R X = 90% and R Y =
70% , and after pruning we have A t = 60% , R X t = 70% , and R Y t
=
Note that the recall balance remains the same for each class before and after pruning: B X = B Xt
= 10% and
B Y = B
Yc = − 10% .
However, the predictions of the pruned network for class Y are as good as a guess.
In turn, the normalized recall balance is the same in absolute value when the two classes are compared, but that value increases due to pruning: ¯ B X = − ¯ B Y = 18 and ¯ B Xt
= − ¯ B Yt = 16 .
Our choice of normalizing through dividing by A t ( m ) rather than by R ct ( m ) aims at avoiding outliers from small recall values as well as undeﬁned results if a high pruning ratio leads to zero recall.
Moreover, an extension of Proposition 4.1 such as Corollary 4.1.1 would not be possible otherwise.
We may obtain similar properties by replacing recall R c ( m ) with precision P c ( m ) .
In other words,
P c is the number of correct predictions for class c divided by the total number of predictions for class c .
By extension we may also consider the effect of pruning on the F-score , in which case we would consider replacing R c with the harmonic mean of recall and precision, i.e., 2 R c P c R c + P c .
However, we have preliminarily found the effect of pruning more expressive on recall than on precision or F-score.
Measuring intensiﬁcation
For a given pruning ratio t and class c of a model m , we consider the following intensiﬁcation ratio :
[Equation]
This metric can be used to evaluate if pruning widens the performance gap between classes, which happens when I ct > 1 .
For a class c in which the original model overperforms ( ¯ B c > 0 ), a ratio greater than 1 after pruning implies that | ¯ B ct | > | ¯ B c | since ¯ B c > [-
 ^-]0 ∧ I ct > 1 → ¯ B ct > ¯ B c > 0 .
Similarly, for a class c in which the original model underperforms ( ¯ B c > 0 ), a ratio greater thanafter pruning also implies that | ¯ B ct | > | ¯ B c | since ¯ B c <
∧
I ct > 1 → ¯ B ct < ¯ B c < 0 .
In other words, we can use the same metric to determine if pruning comparatively improves and worsens normalized recall balance for classes in which the model respectively overperforms and underperforms.
Example (cont.)
For both classes X and Y , we obtain the same intensiﬁcation ratio I
Xt
= I Yt = 43 .
Hence, pruning the model leads to a greater disparity in relative performance for those classes.
Since I ct = B ct B c
A A t
and we may assume A t <
A for sufﬁciently large t , one could argue that A A t >alone leads to I ct > 1 .
However, our experiments show quite the opposite in the case of random pruning, which can be regarded as a less careful selection of weights to prune.
Moreover, we consider a ratio above 1 from another perspective.
Namely, I ct > 1 implies B ct B c > A t
A indicating the relative drop in recall balance (if indeed it drops at all) is less severe than the relative drop in accuracy.
We summarize intensiﬁcation across classes for a model m at pruning ratio t by estimating the slope α t ( m ) of a simple linear regression between normalized recall balance before and after pruning:
[Equation]
The slope α t ( m ) is ﬁt using all classes in C .
We omit the intercept term from this regression as it would always be zero in the balanced datasets that we study due to Corollary 4.1.1.
Consequently, the ordinary least squares estimate of the slope becomes a weighted mean of the intensiﬁcation ratios I ct :
[Equation]
In this weighted mean, less weight is given to classes in which normalized recall balance before pruning is closer to 0.
We believe this is an appropriate model-level summary, since it de-emphasizes classes for which recall and accuracy are nearly the same before pruning.
Hence, we conclude that pruning induces an overall intensiﬁcation on m if α t ( m ) > 1 and de-intensiﬁcation if α t ( m ) < 1 .
In addition, within each model there is a statistical dependency between the set of per-class ratios { I c t | c ∈ C } obtained after pruning.
However, the model-level summaries α t ( m ) are independent across random initializations associated with training each model m (conditional on the dataset), which allows us to use simpler statistical inference methods as described in Section 6.2.
In our statistical analysis, we evaluate E [ α t ] as the hypothetical expectation across inﬁnitely many m .
Comparatively, the statistical analysis in Hooker et al.
[30] would be equivalent to evaluating E [ I ct ] , as it is based on R ct A t
= I ct − 1 .
A multiple linear regression with R ct A t as dependent variable and R c
A as one of the independent variables is ﬁt by Paganini [55] over a large number of models and classes.
Experimental setting
We seek to understand when E [ α t ] > 1 , or alternatively E [ α t ] < 1 , by evaluating the impact of network pruning algorithms on models trained on seemingly balanced datasets and through varying but well-known architectures.
Hence, we consider the dependence of the intensiﬁcation ratio at the model level by denoting it as α D,Mt,P for a dataset D ∈ D , an architecture M ∈ M , a pruning ratio t ∈ T , and a pruning algorithm P ∈ P .
In what follows, we may omit the indices if they are constant.
6.1 Computational details
We use some combinations of (i) models based on ResNet- { 20 , 32 , 44 , 56 , 110 } [26] and an adaptation of LeNet5 [40] (ii) trained in MNIST [40], Fashion-MNIST [75], CIFAR-10, and CIFAR-100 [38], and then (iii) pruned using MP, GP, and Random Pruning (RP) from ShrinkBench [9] and our implementation of UP over ShrinkBench, all of which tested with (iv) pruning ratios 2, 4, 10, 20, and 50.
For each combination of model and dataset evaluated, we have trained 30 models, which are the same used for pruning at each pruning ratio.
The datasets were chosen due to their popularity and equal representation across classes.
The architectures and pruning ratios were chosen based on preliminary experiments aiming for good accuracy and also to prune to up to a ratio with evidence of distortion.
MNIST and Fashion-MNIST are only trained in LeNet5.
When the dataset or model does not vary, we use CIFAR-10 and ResNet-56 due to their intermediary complexity.
When the pruning algorithm does not vary, we use MP due to its popularity and seemingly better performance than GP [9].
The LeNet5 models are trained with SGD optimizer for 30 epochs, with batch size of 128 and learning rate of 0 . 01 , and then ﬁne-tuned for another 15 epochs after pruning.
The ResNet models are trained with SGD optimizer for 60 epochs, with batch size of 128, a decreasing learning rate schedule, and weight decay of 0 . 0005 .
These values were selected based on preliminary testing.
For all models, the weights for the epoch with the lowest validation loss are saved during training and ﬁne-tuning for testing.
All experiments are run on GPUs of Nividia GeForce 3060ti and 3090.
6.2 Statistical methods
We are interested in comparing the distributions of α D,Mt,P across different scenarios.
Each α D,Mt,P is calculated using the test set recall metrics from just one run of training and a pair of models, corresponding to before and after pruning at ratio t .
Thus we frame our statistical analyses as inferences about E [ α D,Mt,P ] , where the expectation is taken over neural network models trained with different random seeds.
We have tested varying M , D , or P along with t .
Hence, we evaluate the impact of the pruning ratio along every other dimension.
For model complexity through α Mt , we use CIFAR-10 trained on ResNet- { 20 , 32 , 44 , 56 , 110 } and pruned with MP.
For dataset complexity through α Dt , we use all the datasets with their default model and pruned with MP.
For pruning algorithm through α t,P , we use CIFAR-10 trained on ResNet-56 and pruned with all pruning methods.
Due to the encouraging results with UP, we have repeated the model complexity with UP for comparison.
Figures Each scatterplot matrix shows the normalized recall balances before vs after pruning, at several t (rows) and several of either M , D , or P (columns). The scatterplots for varying P are in Figure 5; the remaining scatterplots can be found in Appendix B, including Figure 8 for varying
M with UP instead of MP as the pruning algorithm.
Each point corresponds to one class c for one model m .
One regression line and numeric summaries are overlaid in each subplot: ˆ α is an average slope using all m together, r 2 is the corresponding coefﬁcient of determination, and ¯ A is the average accuracy across all m .
Boxplots summarize how the corresponding model-level slopes α D,Mt,P ( m ) vary across m , and how their distribution changes with t and either M , D , or P . These boxplots are in Figures 1 to 3, in addition to Figure 4 for varying M with UP instead of MP.
Conﬁdence intervals At each boxplot, we calculate a t-based 99% conﬁdence interval (CI) for its E [ α D,Mt,P ] .
We chose 99% conﬁdence to achieve 95% family-wide conﬁdence (within each family of 5 pruning ratios at a given M , D , or P ) after a Bonferroni correction for simultaneously reportingdependent CIs.
The labels for pruning ratios in the boxplots include < , > , or ? to denote whether the corresponding CI is below 1 (de-intensiﬁcation), above 1 (intensiﬁcation), or overlaps 1.
Appendix C has plots for each CI.
Figures 1, 2, and 3 indicate that within each M , D , or P , low ratios tend to have CIs entirely below 1, some moderate ratios have CIs that overlap with 1, and high ratios tend to have CIs entirely above 1.
The exception is P =RP, where the CIs are below 1 at all ratios.
Hypothesis tests by t , M , D , or P
We carry out t-tests for each set of hypotheses in Section 7.
Tests comparing pairs of M or of D are independent-samples tests, but tests comparing pairs of t or of P are paired-samples tests: for each m there is a natural pairing between two α s using the same uncompressed m . Each p-value reported in Tables 1 through 9 (Appendix A) has been multiplied [Figure] [Figure]
[Figure] , at each t within each D . Superscripts < , > , or ? denote where 99% CIs were below 1, above 1, or overlapped 1. by the number of rows in its column of the table, as a Bonferroni multiple-testing correction for simultaneously evaluating all the rows in that column.
When a reported p-value is below the usual signiﬁcance level 0.05, we have evidence that intensiﬁcation differs between the pairs in that setting.
Accuracy before pruning In order to assess the impact of pruning on accuracy, the mean accuracy before pruning for each dataset and model is reported in Table 10 (Appendix D).
Analysis 7.1 The inﬂuence of the pruning ratio
We carry out one-sided paired-samples t-tests between the following pairs of hypotheses:
[Equation]
In all of those, t i < t i +1 are consecutive pruning ratios in T .
We based our a priori on the observation that a higher pruning ratio typically leads to models that have a lower performance. As such we believe more pruned models would have less capacity to identify as many classes with similar recall when compared with less pruned models, and thus be more susceptible to intensiﬁcation after pruning. Figures 2 and 6 suggest that at each M , intensiﬁcation tends to remain similar or become stronger at higher pruning ratios.
Tables 1 and 2 conﬁrm strong evidence of this trend between most consecutive [Figure] pairs of ratios, for both MP and UP, at most model sizes, except for ratios 2 vs 4 on some larger architectures.
Figure 3 and Figure 7 suggest that at each D , intensiﬁcation tends to remain similar or become stronger at higher pruning ratios.
Table 3 conﬁrms that we have strong evidence of increasing intensiﬁcation for MNIST when going from pruning ratio 20 to 50; for Fashion, when going fromto 20 and from 20 to 50; and for both CIFAR-10 and CIFAR-100, for every consecutive pair of ratios.
Figure 1 and Figure 5 suggest that at each P except RP, intensiﬁcation tends to remain similar or become stronger at higher pruning ratios.
Table 4 conﬁrms strong evidence of this trend between most consecutive pairs of ratios at most P .
However, for RP, the average slope actually appears to decrease instead during several ratio increases.
7.2 The inﬂuence of model complexity
For each ratio t ∈ T , we carry out a one-sided independent-samples t-test of H
[Equation] [Equation]
] for M i and M i +1 as consecutive models in ResNet- { 20 , 32 , 44 , 56 , 110 } on CIFAR-10, using MP and UP.
We based our a priori on the observation that larger (deeper) networks are able to learn more complex nonlinear functions than smaller (shallower) networks. As such we believe smaller pruned models would have less capacity to identify as many classes with similar recall when compared to larger pruned models, and thus be more susceptible to intensiﬁcation after pruning.
Figures 2, 4, 6, and 8 suggest that at most ratios, smaller model sizes tend to have more intensiﬁcation than larger model sizes.
Tables 5 and 6 conﬁrm strong evidence of this trend between most consecutive pairs of model sizes (except 56 vs 110 with MP) at high t , but rarely at low t .
7.3 The inﬂuence of dataset complexity
For each ratio t ∈ T , we carry out a one-sided independent-samples t-test of
[Equation]
for the following ( D i , D j ) pairs: (MNIST, CIFAR-10); (Fashion, CIFAR-10); and (CIFAR-10,
CIFAR-100), using MP.
We based our a priori on how “more complex” datasets often require larger (deeper) models to achieve acceptable accuracy. As such we believe by increasing dataset complexity the pruned model would not be able to identify as many classes with similar recall when compared to less complex datasets, and thus be more susceptible to intensiﬁcation.
We judged that MNIST and
Fashion have comparable complexity (10 black-and-white classes), but both are less complex than
CIFAR-10 (10 RGB classes), which is less complex than CIFAR-100 (100 RGB classes).
Figure 3 and Figure 7 suggest that at most ratios, MNIST and Fashion tend to have shallower slopes (less intensiﬁcation) than CIFAR-10, which has shallower slopes than CIFAR-100.
Table 7 conﬁrms strong evidence of this trend between MNIST and CIFAR-10 at the smaller ratios; between Fashion and CIFAR-10 only at moderate ratios; and between CIFAR-10 and CIFAR-100 only at the higher ratios.
7.4 The inﬂuence of the pruning algorithm
For each ratio t ∈ T , we carry out a two-sided paired-samples t-test of
[Equation]
for pruning algorithms
P i , P j ∈ { MP , GP , UP , RP } on CIFAR-10 with ResNet-56.
Without clear a priori expectations for which algorithms would experience more intensiﬁcation than others, we compare all pairs and use two-sided tests.
We also compared just MP and UP at all models in
ResNet- { 20 , 32 , 44 , 56 , 110 } .
Figure 1 and Figure 5 suggest a trend in which, at most ratios, UP tends to have shallower but still positive slopes (less intensiﬁcation) than MP or GP, while RP always has negative slopes.
Tableconﬁrms we have evidence that the average slopes for MP, GP, and UP differ from each other at higher pruning ratios, but not necessarily at the lowest ratios.
We also have strong evidence that all three methods differ from RP, at each ratio.
Moreover, comparing Figure 8 to Figure 6 suggests that UP tends to have less intensiﬁcation than MP at each t and M .
Table 9 conﬁrms this for the larger ratios and models.
We can also observe by comparing Figure 5 and Figure 8 that ˆ α is always smaller for UP, whereas ¯ A is only slightly greater for MP in 3 out of 20 cases: ResNet-20 with t = 4 , ResNet-110 with t = 2 , and ResNet-110 with t = 4 .
Supplemental discussion and experiments
Following the advice from the anonymous reviewers, we have augmented our discussion and the experiments carried out in the paper.
First, we consider the tradeoff between accuracy and intensiﬁcation at lower pruning ratios to discuss the operationalization of our results in Appendix E. Second, we report the variance of recall balance in Appendix F. Third, we discuss at greater length the use of α as our metric of interest in Appendix G.
Finally, we included additional results comparing accuracy and intensiﬁcation for two recent approaches, LTH [17] and CHIP [65], in Appendix H.
Conclusion
In this work, we have found evidence that network pruning may cause recall distortion even in models trained on seemingly balanced datasets, thereby complementing prior studies with underrepresented classes and features [30, 55, 31, 1, 63, 69].
In our experiments, we have observed a statistically signiﬁcant effect of higher pruning ratios, increased task difﬁculty, and greater model complexity intensifying the effect of normalized recall balance — the difference between recall and accuracy divided by accuracy. This happens for both underperforming classes (recall below accuracy) and overperforming classes (recall above accuracy) for any pruning method other than Random Pruning (RP), hence showing that accuracy reduction alone does not lead to intensiﬁcation.
More surprisingly, however, is that we have observed the opposite phenomenon at lower pruning ratios. Namely, we have observed a de-intensiﬁcation effect, by which model accuracy improves at the same time that the differences between class recall and model accuracy goes down.
To the best of our knowledge, no other work has identiﬁed this corrective effect as a positive externality of moderate pruning. We have also introduced Undecayed Pruning (UP) as a variant of Gradient Pruning (GP) that discounts the effect of weight decay from the loss function, leading to an algorithm that is more closely aligned with the better acclaimed Magnitude Pruning (MP). Since UP consists of a ﬁne adjustment between the MP and GP approaches, we are not surprised that this leads only to a minor improvement.
Nevertheless, it shows that a better alignment of the pruning method with the loss function alleviates recall balance. We may explore in future how to deduct other forms of regularization before pruning. One possible limitation of our study is that it does not include a vast list of pruning algorithms, which would nevertheless be prohibitive for a comprehensive statistical evaluation.
However, it emphasizes
[Table] [Figure]
one m . See Section 6.2.
the classic methods conventionally used and sheds light on how they can be combined through our new algorithm. Nevertheless, we believe that the insight on how to develop better variations of simple algorithms such as MP and GP helps us understand how to design more complex network pruning algorithms as well.
In fact, additional experiments carried out by suggestion of the anonymous reviewers show that the intensiﬁcation effect can also be observed in modern pruning methods. Acknowledgments and Disclosure of Funding We would like to thank the anonymous reviewers for the constructive discussion, including their suggestions to expand the scope of our work as well as to consider future directions of work.
Aidan Good, Jiaqi Lin, Hannah Sieg, and Thiago Serra were supported by the National Science Foundation (NSF) grant IIS 2104583. Hannah Sieg was supported by the H. Royer Undergraduate Research Fund.
Mikey Ferguson was supported by Bucknell University’s Presidential Fellowship.
Xin
Yu was partially supported by the National Science Foundation (NSF) grant IIS 1764071.
Shandian
Zhe was supported by the National Science Foundation (NSF) grant IIS 1910983.
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Instruction • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 6.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 6.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section6.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
<0.
44 vs0.
<
[Table]
<0.
[Table]
[Table]
[Table]
-56, Bonferroni-corrected by column.
Methods
[Table]
[Table]
B Additional scatterplots
[Figure]
[Table]
[Figure]
See Section 6.2.
[Table]
[Figure]
m .
See
C Conﬁdence intervals for each boxplot
[Figure]
[Figure]
[Figure]
[Figure]
D Mean accuracy before training
[Table]
E Tradeoff between recall distortion and accuracy
The supplemental results in this section illustrate the tradeoff between recall distortion and accuracy, which are illustrated in Figure 13.
We reﬁne our study in lower pruning ratios by evaluatingmodels at ratios 2, 4, 6,
[Table]
and 10.
In those plots, the blue curve associated with the left y-axis represents the mean accuracy at each compression ratio, with the initial observation at compression ratio 1 corresponding to the model accuracy before pruning.
The orange curve associated with the right y-axis represents the mean intensiﬁcation at each compression ratio.
We have aligned both y-axes so that the center of the left axis represents the mean accuracy before pruning and the right axis represents an intensiﬁcation ratio of 1, and then a dashed horizontal line is drawn at the center of the plot.
Whenever we see the blue plot above that line and the orange plot below that line, which is typical for the lower pruning ratios, we are observing accuracy going up while intensiﬁcation is going down.
We observe some agreement between recall distortion and accuracy for the pruning ratios at which both are more beneﬁcial.
More speciﬁcally, we observe model accuracy improving at the same time that intensiﬁcation is reduced if a small pruning ratios are used.
These plots suggest that intensiﬁcation could be another axis along which pruning methods should be evaluated.
When the pruning ratio increases, model accuracy and intensiﬁcation no longer move in the same direction.
Whereas heavy pruning makes the accuracy worse and intensiﬁcation stronger, lighter pruning makes the accuracy better while not pushing intensiﬁcation above 1.
We believe that these results are actionable to the extent that they encourage the use of a moderate amount of pruning both for the sake of improving generalization as well as to reduce the performance balance across classes.
Moreover, these results provide a qualitative understanding of how to adjust for different cases.
Namely, if we work with a comparatively more complex task we should compensate with a larger model or a lower pruning ratio if we would like to obtain a pruned model with similar improvements for both metrics.
[Figure]
F Variance of recall balance
Table 11 reports mean recall variance on each model before and after pruning for each type of model and pruning ratio used in our study.
All the cases in which the recall variance after pruning is greater than before pruning are in bold.
That probably conﬁrms the intuition of one of the anonymous reviewers that recall variance increases with pruning.
We would emphasize, however, that variance alone is not a clear indicator if recall differences increase or decrease.
[Table]
CIFAR-10 @
ResNet-56 → UP
[Table]
[Table]
CIFAR-10 @
ResNet-56 → RP
[Table]
[Table]
G On the use of α instead of I to measure intensiﬁcation
One concern with using α is that we do not attribute the same weight to large variations around the origin.
However, we note that across all scatterplots in the experimental data that we collected, we did not observe any behavior around the origin that would differ substantially from the linear trend.
Hence, the slope (as measured by α ) seems to be an appropriate summary of trends seen in our plots.
On the other hand, points near the origin correspond to intensiﬁcation ratios whose denominators are near zero, and hence on the scale of y/x ratios they are often volatile outliers—even though on our scatterplots their (x,y) pairs are not outliers.
Since the equally-weighted mean of intensiﬁcation ratios is not robust to outliers, it is not an appropriate summary of trends seen in our data.
To illustrate that, consider the results for CIFAR-10 @
ResNet-32 with MP at rate 20 in Figure 6.
We picked this plot because it has many points concentrated around the origin while presenting a clearly linear behavior.
The value of α reported in the plot (1.455) corresponds to a linear regression using data from all models and classes.
If we calculate for each of the 30 models separately, we obtain 1 .
5 ± 0 .
1 with a minimum of 1.3 and a maximum of 1.7.
In turn, if we calculate the mean of the intensiﬁcations averaging all classes for each model, we obtain 1 .
4 ± 1 . 4 with a minimum of -5.
and a maximum of 4.5.
For the model that yields the minimum of -5.0, there is a single outlier intensiﬁcation ratio, corresponding to the pair of (x,y) values (0.0004, -0.0285).
For the class associated with these values, the recall before pruning is 90.3% and the model accuracy is 90.26%, hence implying that the model overperforms for this class.
Since the test set has 1,000 samples for each class, it would take only one more sample being incorrect for the model to underperform for this class.
However, this class alone contributes with an intensiﬁcation of -64, which would have been positive but similarly large in absolute value if one more test sample were incorrect before pruning.
For the model that yields the maximum of 4.5, there is a single outlier intensiﬁcation ratio, corresponding to the pair of (x,y) values (-0.0003, -0.0099).
The recall for this class before pruning is again 90.3% and the model accuracy is 90.33%, hence implying that the model slightly underperforms for this class.
Although the intensiﬁcation in this case is 30, we note that the normalized recall balance after pruning remains the smallest across all classes.
Furthermore, it would take only one more sample being incorrect for the model to overperform for this class, in which case the intensiﬁcation would be negative but again similarly large in absolute value.
In other words, we believe that the value of α represents a more consistent and representative characterization of the intensiﬁcation effect of pruning on recall balance, since it reﬂects the consistent trends across models shown in our scatterplots.
It is true that α gives less weight to outlier cases corresponding to classes that had recall very close to the model accuracy before pruning, but we believe this is reasonable because those classes have unstable ratios due to their denominators being near zero.
Moreover, since their recalls are so close to the accuracy, it does not seem as appropriate to attribute such changes to intensiﬁcation.
Another way to think about this is that for the classes in such a situation, the model is only narrowly over- or underperforming, which means that the intensiﬁcation ratio for the class is not as informative for our purposes.
We agree that α is not the only way to aggregate information across classes, but we wish to emphasize that and intensiﬁcation ratios are meant to help us think about questions such as “If a recall-balance is already non-negligible, when does pruning push it even farther away in the same direction?”
This is distinct from asking general questions about variability, such as “When does pruning make small recall-balances more variable?”
H Experiments on modern pruning methods
We studied in more detail the effect of the intensiﬁcation in the recent pruning methods
LTH (Lottery
Ticket Hypothesis) [17] and CHIP (CHannel Independence-based Pruning)
[65] on ResNet-56.
In both cases, we observe that the intensiﬁcation ratio ultimately increases with the pruning ratio and that an intensiﬁcation above 1 consistently occurs if the pruning ratio exceeds a certain threshold, which depends on the method.
In order to reach that threshold, we have experimented with higher pruning ratios than those reported in the papers describing those methods.
We also note that it is not straightforward to adapt multiple methods to successfully operate on exactly the same pruning ratios, since there is a lot of engineering in making sophisticated methods work well.
For example, to decide the amount pruned on each iteration with LTH or to decide the amount pruned from each layer with CHIP.
For that reason, we emphasize once more our belief that studying classic methods makes it easier to isolate different factors that may inﬂuence intensiﬁcation, as we did in our study.
Nevertheless, we appreciate the recommendations by the reviewers to consider other methods, and we believe that the results that we obtained for those methods endorse the main message of our work about how intensiﬁcation operates at lower and higher pruning ratios.
Table 12 summarizes the outcome of 10 runs of LTH on models trained on the same setting as those of our other experiments using 30,000 steps for training at each level of pruning, with the corresponding pruning ratio next to it.
The level 0 (pruning ratio 1) corresponds to the original model without pruning.
The LTH paper [17] only goes as far as step 15.
We extend the number of steps using the same pruning ratio between steps used up to step 15.
Even if restricted to the ﬁrst 15 steps, we already observe intensiﬁcation by step 12.
The increase in intensiﬁcation is consistent since step 6, which is in line with our ﬁndings using classic methods.
Table 13 summarize the outcome of 15 runs of CHIP on models trained on the same setting as those of our other experiments, with the corresponding pruning ratio next to it.
Pruning ratio 1 corresponds to the original model without pruning.
The CHIP paper [65] only goes as far as pruning ratio 3.33.
We extend the number of pruning ratios with ratios 8.27 and 19.11 by preserving the proportion of unpruned weights used for pruning ratio 3.33.
If restricted to pruning ratios 1.75 and 3.33, we observe that intensiﬁcation starts to increase from one pruning ratio to another, which is in line with our ﬁndings.
[Table]
[Table]