A Fair Loss Function for Network Pruning
Abstract
Model pruning can enable the deployment of neural networks in environments with resource constraints.
While pruning may have a small effect on the overall performance of the model, it can exacerbate existing biases into the model such that subsets of samples see significantly degraded performance.
In this paper, we introduce the performance weighted loss function, a simple modified crossentropy loss function that can be used to limit the introduction of biases during pruning.
Experiments using biased classifiers for facial classification and skinlesion classification tasks demonstrate that the proposed method is a simple and effective tool that can enable existing pruning methods to be used in fairness sensitive contexts
Introduction
Deep learning models are large, requiring millions of operations to make an inference [1].
Deploying large neural networks to environments with limited computational resources, such as mobile and embedded devices, may be infeasible.
Pruning is a simple and common method for reducing the size of a neural network [2].
It involves identifying parameters that do not significantly affect the model’s output and removing them from the network.
Pruning enables the deployment of performant neural networks in resource constrained environments [3, 4].
However, recent research has shown that while overall accuracy of the model may be maintained while the model is compressed, pruning can exacerbate existing model biases, disproportionately affecting disadvantaged groups [5].
Pruning methods that are designed to preserve overall model performance may not prioritize the preservation of parameters that are only important for a small subset of samples.
This effect has significant implications for the implementation of pruning in real-world situations.
Biases have been observed in artificial intelligence systems such as those used to classify chest X-ray images [6], recognize faces [7] and screen resumes [8].
Biases in models can increase the risk of unfair outcomes, preventing the implementation of the model.
If pruning exacerbated a model’s biases, it could increase the risk of unfair outcomes or limit the deployment of the pruned model.
It is therefore important to prune in a manner that does not aggravate a model’s biases.
In this paper, we propose the performance weighted loss function as a simple method for boosting the fairness of data-driven methods for pruning convolutional filters in convolutional neural network image classifiers.
The goal of our method is to enable the pruning of a significant number of model parameters without significantly exacerbating existing biases.
The loss function consists of two small tweaks to the standard cross-entropy loss function to prioritize the model’s performance for poorly-classified samples over well-classified samples.
These tweaks can be used to extend existing data-driven pruning methods without requiring explicit attribute information.
We demonstrate the effectiveness of our approach by pruning classifiers using two different pruning approaches for the CelebA [9] and Fitzpatrick17k [10] datasets.
Our results show that the performance weighted loss function can enable existing pruning methods to prune neural networks without significantly increasing model bias.
Related Work
Many different pruning approaches have been proposed to reduce the size of CNNs while minimally impacting model accuracy.
Pruning methods typically involve assigning a score to each parameter or group of parameters, removing parameters based on these scores and retraining the newly pruned network to recover lost accuracy [2].
The procedure by which parameters are identified to be pruned is the primary differentiator between pruning methods.
There are a wide variety of scoring approaches used to identify parameters that are unimportant or redundant and can be removed from the network.
Many approaches use parameter magnitudes to identify parameters to prune [11, 12].
Other approaches use gradient information [13],
Taylor estimates of parameter importance [14, 15, 16] and statistical properties of future layers [17].
Some approaches involve learning the scores via parameters that control the flow of information through the network [18, 19].
However, almost all novel pruning approaches focus on the overall accuracy of the model after pruning.
There are few papers that aim to improve or analyze the fairness of a pruned model.
Hooker et al.
[5] propose auditing samples affected by model compression, called Compression Identified
Exemplars, as an approach for identifying and managing the negative effects of model compression.
Paganini [20] demonstrate how class fairness can be affected by pruning approaches that only consider overall accuracy.
Wu et al.
[21] propose Fairprune, a method for improving model bias using pruning.
Instead of seeking to compress a model, Fairprune prunes parameters using a saliency metric to increase model fairness [21].
Xu and Hu [22] propose the use of knowledge distillation and pruning to reduce bias in natural language models.
Joseph et al.
[23] propose a multi-part loss function intended to improve the alignment between predictions between the original and pruned model.
They demonstrate that their method can have beneficial effects for fairness between classes.
Marcinkevi ˇ cs et al.
[24] propose a debiasing procedure that involves pruning parameters using a gradient based influence measure.
While not a pruning method, the work of Mahabadi and Henderson [25] is also relevant as their “Debiased Focal Loss” resembles the weighting scheme of the loss function proposed in this paper.
Instead of using their loss function for model compression, they aim to debias a model using the output of a trained bias-only model.
Method 3.1 Motivation
In the unfair pruning situation described by Hooker et al.
[5], model performance was more significantly impacted for certain sample subgroups.
The highly impacted subgroups were characterized by poor representation in the training data and worse subgroup performance by the original model when compared to unimpacted groups.
The performance decrement induced by the pruning process disproportionately impacts subgroups which are underrepresented and poorly classified.
To rectify this inequality, we can design a pruning process that prioritizes maintaining the performance of samples from the impacted subgroups.
However, we do not need to develop a new pruning method from scratch to achieve this objective.
Many existing pruning methods use data to identify which model parameters should be removed.
Some methods use parameters learned via a loss minimization process whereas others values derived from gradients calculated with respect to a loss function.
By modifying the loss function to prioritize samples from impacted subgroups, we can boost the fairness of existing pruning methods.
3.2 The Performance Weighted Loss Function
We make two different modifications to the standard cross-entropy loss function to transform it into the performance weighted loss function (PW loss).
We first apply sample weighting to ensure that samples from impacted groups have a larger contribution to the loss function.
We then transform the sample labels to ensure that we are not reinforcing undesirable model behaviours.
As the attribute information required to identify impacted subgroups is not always readily accessible, our weighting scheme does not depend on any external information.
We instead use the output of the original model to determine each sample weight.
We assign larger weights to samples for which the original model was not able to confidently classify.
The form of the scheme resembles the focal loss [26].
However, as the samples are weighted using the outputs of the original model the weights do not depend on the current output of the model and will not change during training.
The weight assigned to the i th data sample, w i , is given by the following equation:
[Equation]
where ˆ y i is the predicted probability given by the original model for the sample’s true class, θ ∈ [0 , 1] is the minimum weight value and γ ≥ 0 controls the shape of the relation between ˆ y i and w i . This weighting sch
We also emphasize the model performance through the use of corrected soft-labels in the cross-entropy function.
Rather than using the true labels of each sample, we use the output of the original model for the loss function in the pruning process.
Without this change, the preservation of an originally poorly classified sample’s prediction probability would result in a greater loss value than the preservation of an originally well classified sample’s prediction probability.
The use of true labels implicitly prioritizes the preservation of model performance for samples that have predictions closer to their true labels.
Using the model output as soft-labels alleviates this implicit prioritization.
However, as we are assigning higher weights to samples that are originally classified by the original model while also using the original model’s output as our labels, we are consequently assigning the highest weights to incorrect labels.
To avoid emphasizing incorrect behaviours we correct the soft-labels.
The corrected soft-label, ˆ y ∗ i is defined as:
[Equation]
where ˆ y i contains the prediction probabilities derived from the model output for the i th sample, y i is the true label vector of the i th sample, ˆ C
i is predicted class of the i th sample and C
i is the true class of the i th sample.
The corrected soft-label takes on the value of the model’s prediction probabilities when the prediction is correct and the true label when the prediction is incorrect.
By the application of the performance weighted scheme and corrected soft-labels onto the standard cross-entropy function, the performance weighted loss function, L PW , is defined by:
[Equation]
where ˆ y ′ i contains the prediction probabilities derived from the model output for the i th sample after pruning, l CE ( ˆ y ∗ i , ˆ y ′ i ) is the cross-entropy between the corrected soft-label and the prediction probabilities of the pruned model for the i th sample, and N is the number of samples in the batch.
By using this loss function with existing data-driven pruning methods, we can reduce the bias exaggerating effect of pruning by emphasizing samples that are more likely to be negatively affected by pruning.
Experiments 4.1 Experimental Set-up
We applied the PW loss to two different pruning methods.
The first method is AutoBot [18], an accuracy preserving pruning method that uses trainable bottleneck parameters that limits the flow of information through the model.
The second method uses an importance metric derived from the Taylor expansion of the loss function [14].
In both of our implementations, we pruned whole convolutional filters rather than individual neurons.
As pruned filters can be fully removed from the model, rather than being set to zero, filter pruning is a simple method for directly reducing the
FLOPS of a model.
In the AutoBot method, the bottlenecks are optimized by minimizing a loss function that includes the cross-entropy between the original and pruned model outputs, as well as terms that encourage the bottlenecks to limit information moving through the model, achieving a target number of FLOPS [18].
We applied the performance weighted loss function to the method by replacing the cross-entropy term in the loss function with the performance weighted loss function.
Additionally, we also used the performance weighted loss function when retraining the model after pruning.
The importance metric of the Taylor expansion method is formed using the gradient of the loss function with respect to each feature map and the value of each feature map [14].
This method alternates between training the network and pruning a filter.
In our implementation, a filter is pruned every five iterations.
We applied the performance weighted loss function by replacing the loss functions used in the gradient calculation and model training with the performance weighted loss function.
Once again, we also used the performance weighted loss function when retraining the model after pruning.
We also evaluated a random pruning method in which filters are selected and pruned from the network until only the desired number of FLOPS remain.
We use this method as a reference.
We implemented the methods using the PyTorch library [27].
The methods were implemented as three step pipelines in which the model is first pseudo-pruned by setting parameters to zero, fully pruned using the Torch-Pruning library [28] and retrained.
Pseudo-pruning allows for fast pruning during the pruning process while the full pruning step removes the unused parameters, reducing the number of operations required for prediction.
Due to dependencies between parameters introduced by structures such as residual layers, the achieved theoretical speedup often slightly differs from the target theoretical speedup.
All hyperparameters for the pruning methods were selected using a hold-out validation set.
Hyperparameters for the pruning methods were selected without the PW loss applied and were used for both unmodified and PW loss method variants.
We repeated each experiment three times.
All figures displaying model performance after pruning are displaying the average of all trials.
4.1.1 Metrics
Our primary concern is the degradation of a model’s behaviour towards different subgroups due to pruning.
We therefore evaluated the models by comparing the change in the areas under the receiver operator curves (ROC-AUC) for various subgroups for five different degrees of pruning.
As it is a threshold agnostic performance metric, the ROC-AUC is a good measure of the model’s understanding and separability for a subgroup [29].
For non-binary classification we used the one-vs-one ROC-AUC.
We measured the degree to which a model is pruned using the theoretical speedup, defined as the
FLOPS of the original model divided by the FLOPS of the pruned model.
4.2 Evaluating Fairness and Performance
All methods were tested with and without the PW loss on two different classification tasks.
Our first task was the celebrity face classification task using the CelebA dataset [9] as outlined by
Hooker et al.
[5], in which a model is trained to identify faces as blonde or non-blonde.
The CelebA dataset contains over 200 000 images of celebrity faces with various annotations.
While blonde non-male samples make up 14.05% of the training data, blond male samples make up only 0.85% of the training data.
We used the provided data splits with 80% of the available data being used for training with the remaining data split evenly for validation and testing.
Our second task is the skin lesion classification task using the Fitzpatrick17k dataset [10].
The
Fitzpatrick17k dataset consists of 16 577 images of skin conditions.
We trained our models to classify the samples as normal, benign or malignant.
Due to missing and invalid images we were only able to use 16 526 images.
Each sample in the dataset is assigned a Fitzpatrick score that categorizes the skin tone of the sample.
We trained our models on only samples with light skin tone scores of 1 or 2, and evaluated the model on medium skin tone scores of 3 or 4 as well as dark skin tone scores of 5 or 6.
We used a random 25% of the medium and dark skin tones as a validation set with the remainder used as a test set. [Figure] ResNet-18 With Taylor Pruning [Figure]
4.2.1 Pruning the CelebA Models
We trained a Resnet-18 [30] model and a VGG-16 [31] model for the CelebA task.
The ROC-AUCs for the male and non-male subgroups of the Resnet-18 model were 0.9639 and 0.9794 respectively.
The ROC-AUCs for the male and non-male subgroups of the VGG-16 model were 0.9679 and 0. respectively.
Both models were pruned using target theoretical speedups of 16, 32, 64, 128 and 256.
The change in ROC-AUC for all tested pruning methods for the Resnet-18 and VGG-16 models can be found in Figure 1.
Some of the VGG-16 models pruned using the AutoBot, AutoBot with performance weighting and random methods always predicted a single class.
These degenerate models were excluded from the figure.
All methods were able to significantly reduce the size of both models, but most of the results without performance weighting exhibited divergent performance between the male and non-male subgroups as the theoretical speedup increases.
Performance weighting was highly effective when pruning the Resnet-18 model for both the AutoBot and Taylor pruning methods.
We see an increase in AUC-ROC at all tested theoretical speedups for both the male and non-male subgroups.
The increase for the male subgroup is substantial and the subgroup AUC-ROC scores no longer diverage as the theoretical speedup increases.
We see similar improvements when performance weighting is applied to the AutoBot method for the
VGG-16 model, however the improvements are only substantial at the lowest theoretical speedups.
We do not see improvements when performance weighting is applied to the Taylor method.
This is likely due to the method not having significantly divergent performance for the VGG-16 model.
4.2.2 Pruning the Fitzpatrick17k Models
We trained a Resnet-34 [30] model and a EfficientNet-V2 Medium [32] model for the Fitzpatrick17k task.
The ROC-AUCs for the medium and dark subgroups of the Resnet-34 model were 0.8190 and 0.7329 respectively.
The ROC-AUCs for the medium and dark subgroups of the EfficientNet model were 0.8516 and 0.7524 respectively.
Despite a bias against dark skin tones existing in the original models, we do not see divergent
AUC-ROC scores as the theoretical speedup increases.
The medium skin tone subgroup actually saw greater changes in AUC-ROC due to pruning.
We only see slight benefits for using performance weighting with the Fitzpatrick17k models.
Performance weighting increased slightly improved performance after pruning for the ResNet-34 model with AutoBot pruning and the EfficientNet model [Figure] h a [Figure]
i n A
U C - R O
C ResNet [Figure] [Figure]
AutoBot Pruning
[Figure]
a
[Figure]
e i n A
U C - R O
C
ResNet-34 With Taylor Pruning
[Figure]
with AutoBot pruning at lower theoretical speedups.
It had negligible or detrimental effects for Taylor pruning with both models.
These results indicate that performance weighting is not an appropriate solution for all datasets and models that exhibit bias.
The lack of an increasing performance difference between subgroups may indicate that the pruning process was not introducing additional biases in the Fitzpatrick17k models.
This is in contrast to the CelebA models for which the initial bias was small but grew due to pruning.
Performance weighting may therefore only mitigate biases that are introduced from the pruning process.
It will not rectify biases that exist in the model before pruning.
4.3 Conditions for Bias
From our results in Section 4.2, we can see that utilizing the PW loss is not necessary in all circumstances.
The loss appeared to be more beneficial for models which saw increasing differences in performance between subgroups as the theoretical speedup increased.
To understand the properties of a dataset that would necessitate the use of the PW loss, we created three artificial datasets from the CelebA dataset by selected subsets of the training data.
The first subset was formed using 3.41% of the available training data such that it was fully balanced, containing an equal number of male and non-male samples as well as an equal number of blonde and non-blonde samples.
The second and third subsets were formed by adding additional samples to the first subset, altering the class or gender balance.
The second subset contained an equal number of blonde and non-blonde samples, but five times as many non-male samples as there were male samples.
The third subset contained an equal number of male and non-male samples, but five times as many non-blonde samples as there were blonde samples.
The entire test set was used to evaluate all subsets.
A ResNet-18 model was trained using each subset.
The AUC-ROCs for the male subgroup are 0.9562, 0.9479 and 0.9183 for the first, second and third subsets respectively. The AUC-ROCs for the non-male subgroup are 0.9713, 0.9732 and 0.9580 for the first second and third subsets respectively.
The models were pruned using the AutoBot and Taylor methods using target theoretical speedups of 8, 32 and 128.
The performance after pruning for these models can be found in Figure 3.
In the results using the fully balanced subset, we do see a divergence in subgroup performance for both methods, but the divergence is less than was seen when the full method was used.
In the results with the additional non-male samples, we see an increase in performance for all model/method 2222 7 Theoretical Speedup −0. −0. −0.
C h a n g e i n A
U C - R O C Fully Balanced 2222 7 Theoretical Speedup −0. −0.
−0.
C h a n g e i n A
U C - R O
C
Unequal Male/Non-Male Split 2222 7 Theoretical Speedup −0.
−0.
−0.
C h a n g e i n A
U C - R O
C
Unequal Label Split
AutoBot (Male)
AutoBot (Non-Male)
Taylor (Male)
Taylor (Non-Male) combinations.
For the AutoBot results, the increase is greater for non-male samples than it is for male samples.
We once again see a common increase in performance when we look at the subset with the additional non-blonde samples.
We do see additional instability in the Taylor results, but there are no clear findings with respect to differences in performance between subgroups.
A greater decrease in performance was seen for male samples for all model/method combinations, including those that were trained on data with a balanced gender split.
These results indicate that the dataset composition does influence the fairness of pruning results, but it does not fully explain it.
4.4 Ablation
To measure the effects of the components of the PW loss independently, we pruned our [-ResNet-
 CelebA-]ResNetCelebA model using the AutoBot method with only the corrected soft-labels and with only the weighting scheme described in equation 1.
We applied the modifications to the only the pruning process, and to both the pruning and retraining processes.
The ablation results can be found in Figure 4.
Both the modifications were more effective when applied to both the pruning and retraining process, indicating that simply modifying the process by which parameters are selected to be pruning is insufficient to mitigate the effects of bias.
Furthermore, the effect of using corrected soft-labels was larger than the effects of using our proposed weighting scheme.
While both changes boosted performance for the male subgroup when applied to both pruning and retraining, the effect of the corrected soft-labels was almost as large the effect of the full performance weighting method.
The full method did demonstrate less bias with a target theoretical speedup of 16.
Furthermore, as the AutoBot method already uses the outputs of the original model in its loss function, the improvement seen when the corrected soft-labels were only used for pruning can solely be attributed to the correction of the model outputs.
Unlike our proposed weighting scheme, the use of corrected soft-labels does not involve the selection of any parameters.
In situations in which parameter selection is not possible, the use of corrected soft-labels may be a simple yet useful method for reducing the effects of algorithmic bias in pruning.
Conclusion
In this paper we demonstrate how model pruning can exacerbate biases in models and present the performance weighted loss function as a novel method for mitigating this effect.
The performance weighted loss function is a simple modification that can be applied to any pruning method that uses the cross-entropy loss.
Our experimental results indicate that while the performance weighted loss function does not recitify model biases, it can help prevent those biases from becoming exaggerated by the pruning process.
The performance weighted loss function is a useful tool for practioners who seek to compress existing models without introducing new fairness concerns. [Figure] i [Figure]
A
[Figure] Acknowledgements This work was funded by the Natural Sciences and Engineering Research Council of Canada, the Ontario Graduate Scholarship and the Canada Research Chairs Program.
We thank them for their support. A Model Training and Pruning Parameters To ensure transparency and enable reproducability, all parameters and procedures used to train, prune and retrain the models can be found below.
All experiments were implemented using PyTorch 1.12. and torchvision 0.13.1 [27].
PyTorch Lightning {+1.7.
 [33]+} was also used to train the models.
The ResNet-18 [30] CelebA model was trained for 20 epochs using the AdamW [34] optimizer with an initial learning rate of 0.0001 and a CosineAnnealingLR learning rate scheduler with T max [-= 
-]=[35].
A batch size of 256 was used.
The model was initialized using the provided ImageNet weights from torchvision .
All parameters in layers except the final fully connected layer were frozen for the first 5 epochs after which they were unfrozen with a learning rate equal to 0.01 times the global learning rate.
Early stopping was applied such that the parameters that achieved the lowest validation loss were saved after training.
The VGG-16 [31] CelebA model was trained for 10 epochs using the AdamW [34] optimizer with an initial learning rate of 0.0005 and a CosineAnnealingLR learning rate scheduler with T max =[35].
A batch size of 64 was used.
The model was initialized using the provided ImageNet weights from torchvision .
All parameters in layers except the final fully connected layer were optimized with a learning rate equal to 0.01 times the global learning rate.
Early stopping was applied such that the parameters that achieved the lowest validation loss were saved after training.
The ResNet-34 [30] Fitzpatrick17k model was trained for 30 epochs using the AdamW [34] optimizer with an initial learning rate of 0.001 and a CosineAnnealingLR learning rate scheduler with T max = 30 [35].
A batch size of 64 was used.
The model was initialized using the provided ImageNet weights from torchvision .
All parameters in layers except the final fully connected layer were frozen for the first 5 epochs after which they were unfrozen with a learning rate equal to 0.001 times the global learning rate.
The EfficientNet V2 Medium [32] Fitzpatrick17k model was trained for 30 epochs using the AdamW [34] optimizer with an initial learning rate of 0.001 and a CosineAnnealingLR learning rate scheduler with T max = 30 [35].
A batch size of 32 was used.
The model was initialized using the provided
ImageNet weights from the torchvision .
All parameters in layers except the final fully connected layer were frozen for the first 5 epochs after which they were unfrozen with a learning rate equal to 0.01 times the global learning rate.
The parameter values used for our AutoBot [18] implementation can be found in Table 1.
β AB and γ AB refer to the parameters used by the AutoBot method to control the balance between the different terms of its loss function.
The parameter values used for our Taylor [14] implementation can be found in Table 2.
f prune refers to the frequency of the pruning.
That is, the number of batch iterations between the pruning of filters.
N filters refers to the number of convolutional filters that are pruned in each pruning instance.
The parameter values that are used for our PW losses can be found in Table 3.
Other parameters were not changed when the PW loss was introduced.
After pruning, all models were retrained using the AdamW [34] optimizer and CosineAnnealingLR learning rate scheduler with a T max value equal to the number of epochs.
The parameter values used to retrain the models can be found in Table 4.
[Table] [Table]
[Table] [Table]
B Detailed Results
For brevity, we only included figures displaying our results in the main body of this report.
For transparency, our full results can be found here.
Detailed results for all experiments performed in the main body of the paper can be found in Tables 5 through 10.
[Table]
± 0.
AutoBot + PW 67.6 M ± 0.
M 611.2 k ± 148.8 k 0.
± 0.
± 0. ± 0.
± 0.
AutoBot + PW 48.7 M ± 5.0 M 734.2 k ± 47.7 k 0.
± 0.
± 0. ± 0.
± 0.
AutoBot + PW 21.5 M ± 1.3 M 294.8 k ± 10.6 k 0.
± 0.
± 0. ± 0.
± 0.
AutoBot + PW 9.8 M ± 1.7 M 108.0 k ± 16.6 k 0.
± 0.
± 0. ± 0.
± 0.
Taylor 116.8 M ± 5.0 M 70.8 k ± 2.2 k 0.
± 0.
± 0. ± 0.
± 0.
Taylor 55.0 M ± 2.4 M 25.8 k ± 0.4 k 0.
± 0.
± 0. ± 0.
± 0.
Taylor 21.7 M ± 1.9 M 9.8 k ± 0.7 k 0.
± 0.
± 0. ± 0.
± 0.
Taylor 9.
[Table]
M 1022.8 k ± 41.3 k 0.
± 0.
± 0. ± 0.
± 0.
Random 66.4 M ± 5.6 M 496.6 k ± 30.6 k 0.
± 0.
± 0. ± 0.
± 0.
Random 32.4 M ± 0.7 M 176.3 k ± 19.7 k 0.
± 0.
± 0. ± 0.
± 0.
Random 16.4 M ± 1.7 M 120.9 k ± 5.6 k 0.
± 0.
± 0. ± 0.
± 0.
Random 7.0 M ± 0.7 M 24.1 k ± 10.9 k 0.
± 0.
± 0. ± 0.
± 0.
[Table] [Table]
± 0.
AutoBot + PW 1283.0 M ± 90.3 M 4078.3 k ± 112.2 k 0.
± 0.
± 0. ± 0.
± 0.
AutoBot + PW 677.3 M ± 39.6 M 1670.9 k ± 308.8 k 0.
± 0.
± 0. ± 0.
± 0.
AutoBot + PW 382.3 M ± 12.8 M 1283.4 k ± 126.6 k 0.
± 0.
± 0. ± 0.
± 0.
AutoBot + PW 180.4 M ± 12.5 M 68.5 k ± 27.4 k 0.
± 0.
± 0. ± 0.
± 0.
Taylor 2219.5 M ± 12.1 M 6614.4 k ± 90.9 k 0.
± 0.
± 0. ± 0.
± 0.
Taylor 1273.0 M ± 14.3 M 2271.0 k ± 60.2 k 0.
± 0.
± 0. ± 0.
± 0.
Taylor 732.5 M ± 7.1 M 879.8 k ± 22.5 k 0.
± 0.
± 0.
[Table]
± 0. ± 0.
± 0.
Random 2296.8 M ± 33.0 M 13762.6 k ± 114.3 k 0.
± 0.
± 0. ± 0.
± 0.
Random 1291.3 M ± 35.8 M 7771.6 k ± 333.6 k 0.
± 0.
± 0. ± 0.
± 0.
Random 664.0 M ± 36.7 M 4011.0 k ± 254.9 k 0.
± 0.
± 0. ± 0.
± 0.
Random 350.3 M ± 10.1 M 2122.7 k ± 113.5 k 0.
± 0.
± 0. ± 0.
± 0.
Random 173.6 M ± 2.8 M 925.2 k ± 86.2 k 0.
± 0.
± 0. ± 0.
± 0.
[Table]
± 12.8 M 672.0 k ± 8.8 k 0.
± 0.
± 0. ± 0.
± 0.
Random 4078.8 M ± 157.9 M 39661.8 k ± 539.2 k 0.
± 0.
± 0. ± 0.
± 0.
Random 3305.1 M ± 6.9 M 31406.7 k ± 869.5 k 0.
± 0.
± 0. ± 0.
± 0.
Random 2819.2 M ± 28.5 M 26370.3 k ± 515.0 k 0.
± 0.
± 0. ± 0.
± 0.
Random 1369.1 M ± 33.3 M 13060.2 k ± 157.0 k 0.
± 0.
± 0. ± 0.
± 0.
Random 681.8 M ± 6.3 M 6646.2 k ± 44.0 k 0.
± 0.
± 0. ± 0.
± 0.
[Table] [Table]
± 0.
Pruning Only
AutoBot + Corr.
Soft-Labels 17.2 M ± 1.4 M 199.0 k ± 11.0 k 0.
± 0.
± 0.
Pruning Only
AutoBot + Corr.
Soft-Labels 8.9 M ± 1.1 M 84.7 k ± 8.5 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Weights 112.2 M ± 2.7 M 194.8 k ± 47.8 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Weights 46.7 M ± 4.9 M 255.4 k ± 50.8 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Weights 47.7 M ± 2.9 M 730.3 k ± 29.5 k 0.
± 0. ± 0.
Pruning and
Retraining
AutoBot + Weights 31.5 M ± 12.2 M 486.7 k ± 205.6 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Weights 25.0 M ± 9.5 M 387.2 k ± 167.9 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Corr.
Soft-Labels 108.0 M ± 1.3 M 222.1 k ± 57.6 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Corr.
Soft-Labels 68.3 M ± 4.0 M 928.8 k ± 16.8 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Corr.
Soft-Labels 35.2 M ± 2.
M 474.5 k ± 19.8 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Corr.
Soft-Labels 17.5 M ± 0.2 M 199.9 k ± 10.5 k 0.
± 0.
± 0.
Pruning and
Retraining
AutoBot + Corr.
Soft-Labels 9.0 M ± 1.6 M 85.3 k ± 11.7 k 0.
± 0. ± 0.