Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm
Abstract
Pruning techniques have been successfully used in neural networks to trade accuracy for sparsity.
However, the impact of network pruning is not uniform: prior work has shown that the recall for underrepresented classes in a dataset may be more negatively affected.
In this work, we study such relative distortions in recall by hypothesizing an intensiﬁcation effect that is inherent to the model.
Namely, that pruning makes recall relatively worse for a class with recall below accuracy and, conversely, that it makes recall relatively better for a class with recall above accuracy.
In addition, we propose a new pruning algorithm aimed at attenuating such effect.
Through statistical analysis, we have observed that intensiﬁcation is less severe with our algorithm but nevertheless more pronounced with relatively more difﬁcult tasks, less complex models, and higher pruning ratios.
More surprisingly, we conversely observe a de-intensiﬁcation effect with lower pruning ratios, which indicates that moderate pruning may have a corrective effect to such distortions.
Introduction
Back in the old days, network pruning was important to reduce the size of neural networks [23, 53, 33, 41, 24, 25].
With new advances being increasingly reliant on overwhelmingly large and costly models [3, 64, 10], network pruning is back to the game despite the gains in computing power.
Network pruning reduces the complexity of large models by setting a substantial number of parameters to zero and then using gradient descent to ﬁne-tune the sparser model.
There is typically a trade-off between making the model sparser—i e., having fewer parameters, since those set to zero can simply be discarded—and keeping the model as accurate as possible [9], although moderate gains in sparsity may be obtained with little impact to accuracy [28].
However, recent studies have found that the side effects of network pruning on model performance are not evenly distributed [30, 55, 31, 1, 63, 69].
Those studies have considered the impact of network pruning on recall—i.e., the number of correct predictions per class—based on dataset representability.
Namely, they have found that the degradation in performance is inﬂuenced by the uneven representation of classes [30, 55, 69] and features [31] as well as class complexity [55].
Hence, they corroborate long-standing concerns that unbalanced datasets yield models that are less accurate in and potentially harmful to minoritized groups [11, 7].
In this work, we complement those prior studies by posing recall distortion as inherent to network pruning even when the networks are trained and ﬁne-tuned on datasets that are seemingly balanced; hence investigating when and how such distortions manifest, as well as how to reduce their effect.
More speciﬁcally, this paper presents the following contributions:
(i)
We conduct a statistical study of model-level distortion to investigate how recall is affected by network pruning due to pruning ratios, dataset and model complexity, and pruning algorithms.
(ii)
We observe an intensiﬁcation effect, meaning that for classes with recall below accuracy we observe the relative difference between recall and accuracy negatively widening if the network is sufﬁciently pruned.
For classes with recall above accuracy, we conversely observe that gap positively widening.
The intensiﬁcation correlates with excessive pruning ratios as well as more complex data and models, and is more pronounced with some pruning algorithms.
(iii) More surprisingly, we observe that otherwise network pruning has a corrective effect on recall differences, hence implying a de-intensiﬁcation effect under moderate pruning.
(iv) We introduce a new gradient-based pruning method for networks trained with weight decay, Undecayed Pruning, which attenuates the intensiﬁcation effect observed with other methods.
Related work
The use of large models has been justiﬁed and further encouraged by ﬁndings that the overparameterized regime may avoid the classic bias–variance trade-off [83, 6] and lead to better convergence during training [44, 66].
However, these models come with steep environmental footprint and hardware needs [64].
Not surprisingly, they represent a relevant application of network pruning [20, 78].
Network pruning has been motivated by parameter redundancy in models [13] and found to improve generalization [4] and robustness against adversarial manipulation [74, 35].
The amount of pruning that is tolerable by an architecture may depend on the task in which it is trained [46].
As observed on a recent survey [9], most approaches to network pruning are guided by either (i) removing the parameters with smallest absolute value [23, 53, 33, 22, 21, 43, 17, 15, 20, 68, 48]; or (ii) removing the parameters with smallest impact on the model [41, 24, 25, 39, 52, 14, 79, 82, 5, 42, 71, 47, 72, 76, 61, 80].
We may regard exact pruning as a special case of (ii) in which the model is not affected [59, 62, 60, 18].
Alternative approaches to network pruning include quantization [32, 50, 2], knowledge distillation [37, 45, 84, 73, 70, 67, 34, 77], and the use of regularization during training [81, 16, 56].
In this work, we contribute to the study of recall distortion due to model compression [30, 55, 31, 1, 63], which a particular focus on network pruning.
Concurrent to our work, Tran et al.
[69] framed these recall distortions in unbalanced datasets as a Matthew effect that can be attributed to model (Hessian loss and gradient ﬂows) as well as data characteristics (input norms and distance to decision boundary).
Pruning distortion has also been studied under other metrics, such as by focusing on the samples for which the classiﬁcation is affected [30, 31, 36, 78] and the relationship between false positive and negative rates [8].
The issue is also studied in the context of other compression techniques such as knowledge distillation [27], in which some approaches to remedy this issue are focused on adjusting the loss function used in the compressed model [36, 78].
Such disproportional impact on recall across classes relates to fairness in machine learning.
The lack of representativeness and the biased context in which data is collected as well as the lack of transparency may lead to models making life-altering decisions that negatively affect minoritized groups, such as in criminal justice [54, 11, 57].
These concerns have motivated an extensive discussion on fairness in machine learning [12] and on the proper assessment of datasets [19], models [51], and the circumstances in which they are applied [58].
In this work, we focus on studying the algorithmic bias [49, 29] of network pruning, that is, the bias that is either (i) not present in the input data nor the original model; or (ii) deteriorated or relieved by pruning.
By subscribing to this line of study, our work aims at reducing the potential negative societal impacts associated with network pruning.
Pruning algorithms
We propose Undecayed Pruning (UP) by considering the interplay between the classic representatives of pruning methods described in Section 2: Magnitude Pruning (MP) and Gradient Pruning (GP).
Magnitude Pruning MP is a simple but rather (mysteriously) effective technique of selecting among the parameters θ
= ¯ θ the one with smallest absolute value to be pruned next from the network:
[Equation]
A parameter having a smaller value may not necessarily imply lesser importance to the model [24] (hence the mystery part).
However, it is worth noticing that this technique is usually applied to networks that were trained with regularization.
Since we cannot add L 0 regularization to the loss function for directly minimizing the number of nonzero parameters as this would make the loss function not differentiable for gradient descent, we can use L 1 or L 2 as a proxy, which induces the values of the parameters to be as small as possible.
Although no parameter ends up being zero, some parameters become so small that making them equal to zero implies an almost negligible change.
Gradient Pruning GP approximates the impact of modifying the parameters of a neural network by estimating the ﬁrst-order variation of the loss function L on the training set between the trained parameters θ = ¯ θ to a new set of parameters θ through the Taylor series around θ =
¯ θ :
[Equation]
By assuming that O ( (cid:107) θ − ¯ θ (cid:107) 2 )
≈
for small changes, we estimate the change to the loss function by pruning a single parameter i —i.e., with ˜ θ such that ˜ θ i
= 0 and ˜ θ
j = ¯ θ j for j (cid:54) =
i —as follows:
[Equation]
Given the approximate nature of estimating the impact on the training set, which may not necessarily reﬂect on the test set, small absolute perturbations are often preferred to negative but larger variations.
Hence, the choice of the parameter i to be pruned next is effectively framed as follows:
[Equation]
Undecayed Pruning We aim to address what we believe is a double-edged approach to reducing the number of parameters.
Namely, that it is common to use a term in the loss function to make the weights as small as possible, such as weight decay, and then that we use the gradient of the loss function for pruning weights while ignoring that the gradient is also affected by the weight decay term.
We understand that simultaneously using regularization and GP is conﬂicting because sparsity is induced in two different ways.
However, whereas GP has a more direct proxy to model impact, using some amount of regularization tends to be beneﬁcial during training.
Hence, we approach this by isolating the effect of regularization from the variation of the loss function.
Due to its greater popularity, we consider L 2 regularization–i.e„ weight decay–in what follows.
Let us denote by T the loss function of a model after deducting weight decay with hyperparameter ε :
[Equation]
With ∇ i T ( θ ) = ∇ i L ( θ ) − εθ i , we estimate the change to the alternative loss function T by pruning a single parameter i —i.e., with ˜ θ such that ˜ θ i
= 0 and ˜ θ
j = ¯ θ j for j (cid:54) =
i —as follows:
[Equation]
In other words, discounting weight decay is equivalent to a speciﬁc balance of the criteria for MP and GP: the ﬁrst term is equivalent to GP; the second term is curiously equivalent to MP; and the latter is prioritized in proportion to the weight decay hyperparameter ε .
Moreover, if the neural network training converges to a local optimum, in which case ∇L (¯ θ ) = 0 , then UP is equivalent to
MP.
Similar to the case of GP, we choose the parameter i to prune based on the absolute impact on T :
[Equation]
The way that weights are now ranked reﬂects a weighted combination of the criteria used for MP and
GP.
This is particularly interesting because UP coincides with MP when the gradient is sufﬁciently close to zero, hence providing a principled argument for the effectiveness of MP under weight decay.
Model properties
Let A ( m ) be the accuracy of an unpruned model m as measured on the test data.
In other words,
A ( m ) is the number of correct predictions divided by the number of samples.
For a pruning ratio t , meaning that t is the number of parameters before pruning divided by the number of parameters after pruning, let A t ( m ) denote the accuracy of model m on the test data after pruning.
For simplicity, we may assume that t = 1 if t is absent from the notation and omit m if always referring to the same model, and thus we may assume that A =
A ( m ) =
A 1 ( m ) .
The same applies to other metrics.
Similarly, let R c ( m ) be the recall for class c of an unpruned model m on the test data.
In other words,
R c is the number of correct predictions for class c divided by the number of samples for class c .
For a pruning ratio t , let R ct ( m ) denote the recall for class c of model m on the test data after pruning.
We are particularly interested in the how these metrics differ for each class.
Namely, let
[Equation]
denote the recall balance 3 .
When B ct ( m ) > 0 , we say that model m at pruning ratio t overperforms for class c ; and when B ct ( m ) < 0 we may say that m underperforms for class c .
Finally, let
[Equation]
denote the normalized recall balance .
The further away this value is from 0, the more pronounced is the difference in performance between class c and the other classes in model m at the pruning ratio t .
Proposition 4.1.
If a dataset with set of classes C is balanced , meaning that each class c ∈ C has the same number of samples, then for any model m and pruning ratio t it holds that
[Equation]
Proof.
If the dataset is balanced, then it follows that A t = (cid:80) c
∈ C R ct | C | and hence (cid:80) c ∈ C R ct =
| C |
A t .
Therefore, 0 = (cid:80) c ∈ C ( R ct ) − | C |
A t = (cid:80) c
∈ C
( R ct − A t ) =
(cid:80) c ∈ C B ct .
Corollary 4.1.1.
If a dataset is balanced, then for any model m and pruning ratio t it holds that (cid:88) c
∈
[Equation]
ct
[Equation]
Proof.
Immediate from dividing (cid:80) c
∈ C B ct ( m ) by A t ( m ) due to Proposition 4.1.
Example Consider a model in which A = 80% with two classes X and Y such that R X = 90% and R Y =
70% , and after pruning we have A t = 60% , R X t = 70% , and R Y t
=
Note that the recall balance remains the same for each class before and after pruning: B X = B Xt
= 10% and
B Y = B
Yc = − 10% .
However, the predictions of the pruned network for class Y are as good as a guess.
In turn, the normalized recall balance is the same in absolute value when the two classes are compared, but that value increases due to pruning: ¯ B X = − ¯ B Y = 18 and ¯ B Xt
= − ¯ B Yt = 16 .
Our choice of normalizing through dividing by A t ( m ) rather than by R ct ( m ) aims at avoiding outliers from small recall values as well as undeﬁned results if a high pruning ratio leads to zero recall.
Moreover, an extension of Proposition 4.1 such as Corollary 4.1.1 would not be possible otherwise.
We may obtain similar properties by replacing recall R c ( m ) with precision P c ( m ) .
In other words,
P c is the number of correct predictions for class c divided by the total number of predictions for class c .
By extension we may also consider the effect of pruning on the F-score , in which case we would consider replacing R c with the harmonic mean of recall and precision, i.e., 2 R c P c R c + P c .
However, we have preliminarily found the effect of pruning more expressive on recall than on precision or F-score.
Measuring intensiﬁcation
For a given pruning ratio t and class c of a model m , we consider the following intensiﬁcation ratio :
[Equation]
This metric can be used to evaluate if pruning widens the performance gap between classes, which happens when I ct > 1 .
For a class c in which the original model overperforms ( ¯ B c > 0 ), a ratio greater than 1 after pruning implies that | ¯ B ct | > | ¯ B c | since ¯ B c > 0 ∧ I ct > 1 → ¯ B ct > ¯ B c > 0 .
Similarly, for a class c in which the original model underperforms ( ¯ B c < 0 ), a ratio greater thanafter pruning also implies that | ¯ B ct | > | ¯ B c | since ¯ B c <
∧
I ct > 1 → ¯ B ct < ¯ B c < 0 .
In other words, we can use the same metric to determine if pruning comparatively improves and worsens normalized recall balance for classes in which the model respectively overperforms and underperforms.
Example (cont.)
For both classes X and Y , we obtain the same intensiﬁcation ratio I
Xt
= I Yt = 43 .
Hence, pruning the model leads to a greater disparity in relative performance for those classes.
Since I ct = B ct B c
A A t
and we may assume A t <
A for sufﬁciently large t , one could argue that A A t >alone leads to I ct > 1 .
However, our experiments show quite the opposite in the case of random pruning, which can be regarded as a less careful selection of weights to prune.
Moreover, we consider a ratio above 1 from another perspective.
Namely, I ct > 1 implies B ct B c > A t
A indicating the relative drop in recall balance (if indeed it drops at all) is less severe than the relative drop in accuracy.
We summarize intensiﬁcation across classes for a model m at pruning ratio t by estimating the slope α t ( m ) of a simple linear regression between normalized recall balance before and after pruning:
[Equation]
The slope α t ( m ) is ﬁt using all classes in C .
We omit the intercept term from this regression as it would always be zero in the balanced datasets that we study due to Corollary 4.1.1.
Consequently, the ordinary least squares estimate of the slope becomes a weighted mean of the intensiﬁcation ratios I ct :
[Equation]
In this weighted mean, less weight is given to classes in which normalized recall balance before pruning is closer to 0.
We believe this is an appropriate model-level summary, since it de-emphasizes classes for which recall and accuracy are nearly the same before pruning.
Hence, we conclude that pruning induces an overall intensiﬁcation on m if
α t ( m ) > 1 and de-intensiﬁcation if α t ( m ) < 1 .
In addition, within each model there is a statistical dependency between the set of per-class ratios { I c t | c ∈ C } obtained after pruning.
However, the model-level summaries α t ( m ) are independent across random initializations associated with training each model m (conditional on the dataset), which allows us to use simpler statistical inference methods as described in Section 6.2.
In our statistical analysis, we evaluate E [ α t ] as the hypothetical expectation across inﬁnitely many m .
Comparatively, the statistical analysis in Hooker et al.
[30] would be equivalent to evaluating E [ I ct ] , as it is based on R ct A t
= I ct − 1 .
A multiple linear regression with R ct A t as dependent variable and R c
A as one of the independent variables is ﬁt by Paganini [55] over a large number of models and classes.
Experimental setting
We seek to understand when E [ α t ] > 1 , or alternatively E [ α t ] < 1 , by evaluating the impact of network pruning algorithms on models trained on seemingly balanced datasets and through varying but well-known architectures.
Hence, we consider the dependence of the intensiﬁcation ratio at the model level by denoting it as α D,Mt,P for a dataset D ∈ D , an architecture M ∈ M , a pruning ratio t ∈ T , and a pruning algorithm P ∈ P .
In what follows, we may omit the indices if they are constant.
6.1 Computational details
We use some combinations of (i) models based on ResNet- { 20 , 32 , 44 , 56 , 110 } [26] and an adaptation of LeNet5 [40] (ii) trained in MNIST [40], Fashion-MNIST [75], CIFAR-10, and CIFAR-100 [38], and then (iii) pruned using MP, GP, and Random Pruning (RP) from ShrinkBench [9] and our implementation of UP over ShrinkBench, all of which tested with (iv) pruning ratios 2, 4, 10, 20, and 50.
For each combination of model and dataset evaluated, we have trained 30 models, which are the same used for pruning at each pruning ratio.
The datasets were chosen due to their popularity and equal representation across classes.
The architectures and pruning ratios were chosen based on preliminary experiments aiming for good accuracy and also to prune to up to a ratio with evidence of distortion.
MNIST and Fashion-MNIST are only trained in LeNet5.
When the dataset or model does not vary, we use CIFAR-10 and ResNet-56 due to their intermediary complexity.
When the pruning algorithm does not vary, we use MP due to its popularity and seemingly better performance than
GP [9].
The LeNet5 models are trained with SGD optimizer for 30 epochs, with batch size of 128 and learning rate of 0 . 01 , and then ﬁne-tuned for another 15 epochs after pruning.
The ResNet models are trained with SGD optimizer for 60 epochs, with batch size of 128, a decreasing learning rate schedule, and weight decay of 0 . 0005 .
These values were selected based on preliminary testing.
For all models, the weights for the epoch with the lowest validation loss are saved during training and ﬁne-tuning for testing.
All experiments are run on GPUs of Nividia GeForce 3060ti and 3090.
6.2 Statistical methods
We are interested in comparing the distributions of α D,Mt,P across different scenarios.
Each α D,Mt,P is calculated using the test set recall metrics from just one run of training and a pair of models, corresponding to before and after pruning at ratio t .
Thus we frame our statistical analyses as inferences about E [ α D,Mt,P ] , where the expectation is taken over neural network models trained with different random seeds.
We have tested varying M , D , or P along with t .
Hence, we evaluate the impact of the pruning ratio along every other dimension.
For model complexity through α Mt , we use CIFAR-10 trained on ResNet- { 20 , 32 , 44 , 56 , 110 } and pruned with MP.
For dataset complexity through α Dt , we use all the datasets with their default model and pruned with MP.
For pruning algorithm through α t,P , we use CIFAR-10 trained on ResNet-56 and pruned with all pruning methods.
Due to the encouraging results with UP, we have repeated the model complexity with UP for comparison.
Figures Each scatterplot matrix shows the normalized recall balances before vs after pruning, at several t (rows) and several of either M , D , or P (columns).
The scatterplots for varying P are in
Figure 5; the remaining scatterplots can be found in Appendix B, including Figure 8 for varying
M with UP instead of MP as the pruning algorithm.
Each point corresponds to one class c for one model m .
One regression line and numeric summaries are overlaid in each subplot: ˆ α is an average slope using all m together, r 2 is the corresponding coefﬁcient of determination, and ¯ A is the average accuracy across all m .
Boxplots summarize how the corresponding model-level slopes α D,Mt,P ( m ) vary across m , and how their distribution changes with t and either M , D , or P .
These boxplots are in Figures 1 to 3, in addition to Figure 4 for varying M with UP instead of MP.
Conﬁdence intervals At each boxplot, we calculate a t-based 99% conﬁdence interval (CI) for its E [ α D,Mt,P ] .
We chose 99% conﬁdence to achieve 95% family-wide conﬁdence (within each family of 5 pruning ratios at a given M , D , or P ) after a Bonferroni correction for simultaneously reportingdependent CIs.
The labels for pruning ratios in the boxplots include < , > , or ? to denote whether the corresponding CI is below 1 (de-intensiﬁcation), above 1 (intensiﬁcation), or overlaps 1.
Appendix C has plots for each CI.
Figures 1, 2, and 3 indicate that within each M , D , or P , low ratios tend to have CIs entirely below 1, some moderate ratios have CIs that overlap with 1, and high ratios tend to have CIs entirely above 1.
The exception is P =RP, where the CIs are below 1 at all ratios.
Hypothesis tests by t , M , D , or P
We carry out t-tests for each set of hypotheses in Section 7.
Tests comparing pairs of M or of D are independent-samples tests, but tests comparing pairs of t or of P are paired-samples tests: for each m there is a natural pairing between two α s using the same uncompressed m .
Each p-value reported in Tables 1 through 9 (Appendix A) has been multiplied
[Figure] [Figure]
[Figure]
, at each t within each D .
Superscripts < , > , or ? denote where 99% CIs were below 1, above 1, or overlapped 1.
by the number of rows in its column of the table, as a Bonferroni multiple-testing correction for simultaneously evaluating all the rows in that column.
When a reported p-value is below the usual signiﬁcance level 0.05, we have evidence that intensiﬁcation differs between the pairs in that setting.
Accuracy before pruning In order to assess the impact of pruning on accuracy, the mean accuracy before pruning for each dataset and model is reported in Table 10 (Appendix D).
Analysis 7.1 The inﬂuence of the pruning ratio
We carry out one-sided paired-samples t-tests between the following pairs of hypotheses:
[Equation]
In all of those, t i < t i +1 are consecutive pruning ratios in T .
We based our a priori on the observation that a higher pruning ratio typically leads to models that have a lower performance.
As such we believe more pruned models would have less capacity to identify as many classes with similar recall when compared with less pruned models, and thus be more susceptible to intensiﬁcation after pruning.
Figures 2 and 6 suggest that at each M , intensiﬁcation tends to remain similar or become stronger at higher pruning ratios.
Tables 1 and 2 conﬁrm strong evidence of this trend between most consecutive
[Figure]
pairs of ratios, for both MP and UP, at most model sizes, except for ratios 2 vs 4 on some larger architectures.
Figure 3 and Figure 7 suggest that at each D , intensiﬁcation tends to remain similar or become stronger at higher pruning ratios.
Table 3 conﬁrms that we have strong evidence of increasing intensiﬁcation for MNIST when going from pruning ratio 20 to 50; for Fashion, when going fromto 20 and from 20 to 50; and for both CIFAR-10 and CIFAR-100, for every consecutive pair of ratios.
Figure 1 and Figure 5 suggest that at each P except RP, intensiﬁcation tends to remain similar or become stronger at higher pruning ratios.
Table 4 conﬁrms strong evidence of this trend between most consecutive pairs of ratios at most P .
However, for RP, the average slope actually appears to decrease instead during several ratio increases.
7.2 The inﬂuence of model complexity
For each ratio t ∈ T , we carry out a one-sided independent-samples t-test of
H
[Equation] [Equation]
a : E
[Equation]
M
[Equation]
>
[Equation]
1 t ] for M i and M i +1 as consecutive models in ResNet- { 20 , 32 , 44 , 56 , 110 } on CIFAR-10, using MP and UP.
We based our a priori on the observation that larger (deeper) networks are able to learn more complex nonlinear functions than smaller (shallower) networks.
As such we believe smaller pruned models would have less capacity to identify as many classes with similar recall when compared to larger pruned models, and thus be more susceptible to intensiﬁcation after pruning.
Figures 2, 4, 6, and 8 suggest that at most ratios, smaller model sizes tend to have more intensiﬁcation than larger model sizes.
Tables 5 and 6 conﬁrm strong evidence of this trend between most consecutive pairs of model sizes (except 56 vs 110 with MP) at high t , but rarely at low t .
7.3 The inﬂuence of dataset complexity
For each ratio t ∈ T , we carry out a one-sided independent-samples t-test of
[Equation]
for the following ( D i , D j ) pairs: (MNIST, CIFAR-10); (Fashion, CIFAR-10); and (CIFAR-10,
CIFAR-100), using MP.
We based our a priori on how “more complex” datasets often require larger (deeper) models to achieve acceptable accuracy.
As such we believe by increasing dataset complexity the pruned model would not be able to identify as many classes with similar recall when compared to less complex datasets, and thus be more susceptible to intensiﬁcation.
We judged that MNIST and
Fashion have comparable complexity (10 black-and-white classes), but both are less complex than
CIFAR-10 (10 RGB classes), which is less complex than CIFAR-100 (100 RGB classes).
Figure 3 and Figure 7 suggest that at most ratios, MNIST and Fashion tend to have shallower slopes (less intensiﬁcation) than CIFAR-10, which has shallower slopes than CIFAR-100.
Table 7 conﬁrms strong evidence of this trend between MNIST and CIFAR-10 at the smaller ratios; between Fashion and CIFAR-10 only at moderate ratios; and between CIFAR-10 and CIFAR-100 only at the higher ratios.
7.4 The inﬂuence of the pruning algorithm
For each ratio t ∈ T , we carry out a two-sided paired-samples t-test of
[Equation]
for pruning algorithms
P i , P j ∈ { MP , GP , UP , RP } on CIFAR-10 with ResNet-56.
Without clear a priori expectations for which algorithms would experience more intensiﬁcation than others, we compare all pairs and use two-sided tests.
We also compared just MP and UP at all models in
ResNet- { 20 , 32 , 44 , 56 , 110 } .
Figure 1 and Figure 5 suggest a trend in which, at most ratios, UP tends to have shallower but still positive slopes (less intensiﬁcation) than MP or GP, while RP always has negative slopes.
Tableconﬁrms we have evidence that the average slopes for MP, GP, and UP differ from each other at higher pruning ratios, but not necessarily at the lowest ratios.
We also have strong evidence that all three methods differ from RP, at each ratio.
Moreover, comparing Figure 8 to Figure 6 suggests that
UP tends to have less intensiﬁcation than MP at each t and M .
Table 9 conﬁrms this for the larger ratios and models.
We can also observe by comparing Figure 5 and Figure 8 that ˆ α is always smaller for UP, whereas ¯ A is only slightly greater for MP in 3 out of 20 cases: ResNet-20 with t = 4 , ResNet-110 with t = 2 , and ResNet-110 with t = 4 .
Supplemental discussion and experiments
Following the advice from the anonymous reviewers, we have augmented our discussion and the experiments carried out in the paper.
First, we consider the tradeoff between accuracy and intensiﬁcation at lower pruning ratios to discuss the operationalization of our results in Appendix E. Second, we report the variance of recall balance in Appendix F. Third, we discuss at greater length the use of α as our metric of interest in Appendix G.
Finally, we included additional results comparing accuracy and intensiﬁcation for two recent approaches, LTH [17] and CHIP [65], in Appendix H.
Conclusion
In this work, we have found evidence that network pruning may cause recall distortion even in models trained on seemingly balanced datasets, thereby complementing prior studies with underrepresented classes and features [30, 55, 31, 1, 63, 69].
In our experiments, we have observed a statistically signiﬁcant effect of higher pruning ratios, increased task difﬁculty, and greater model complexity intensifying the effect of normalized recall balance — the difference between recall and accuracy divided by accuracy.
This happens for both underperforming classes (recall below accuracy) and overperforming classes (recall above accuracy) for any pruning method other than Random Pruning (RP), hence showing that accuracy reduction alone does not lead to intensiﬁcation.
More surprisingly, however, is that we have observed the opposite phenomenon at lower pruning ratios.
Namely, we have observed a de-intensiﬁcation effect, by which model accuracy improves at the same time that the differences between class recall and model accuracy goes down.
To the best of our knowledge, no other work has identiﬁed this corrective effect as a positive externality of moderate pruning.
We have also introduced Undecayed Pruning (UP) as a variant of Gradient Pruning (GP) that discounts the effect of weight decay from the loss function, leading to an algorithm that is more closely aligned with the better acclaimed Magnitude Pruning (MP).
Since UP consists of a ﬁne adjustment between the MP and GP approaches, we are not surprised that this leads only to a minor improvement.
Nevertheless, it shows that a better alignment of the pruning method with the loss function alleviates recall balance.
We may explore in future how to deduct other forms of regularization before pruning.
One possible limitation of our study is that it does not include a vast list of pruning algorithms, which would nevertheless be prohibitive for a comprehensive statistical evaluation.
However, it emphasizes
[Table]
[Figure]
one m .
See Section 6.2.
the classic methods conventionally used and sheds light on how they can be combined through our new algorithm.
Nevertheless, we believe that the insight on how to develop better variations of simple algorithms such as MP and GP helps us understand how to design more complex network pruning algorithms as well.
In fact, additional experiments carried out by suggestion of the anonymous reviewers show that the intensiﬁcation effect can also be observed in modern pruning methods.
Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for the constructive discussion, including their suggestions to expand the scope of our work as well as to consider future directions of work.
Aidan Good, Jiaqi Lin, Hannah Sieg, and Thiago Serra were supported by the National Science
Foundation (NSF) grant IIS 2104583.
Hannah Sieg was supported by the H. Royer Undergraduate
Research Fund.
Mikey Ferguson was supported by Bucknell University’s Presidential Fellowship.
Xin
Yu was partially supported by the National Science Foundation (NSF) grant IIS 1764071.
Shandian
Zhe was supported by the National Science Foundation (NSF) grant IIS 1910983.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Instruction • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 6.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 6.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section6.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]