G LOBAL
M AGNITUDE
P RUNING
W ITH
M INIMUM
T HRESHOLD I S
A LL W E N EED
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural network pruning remains a very important yet challenging problem to solve.
Many pruning solutions have been proposed over the years with high degrees of algorithmic complexity.
In this work, we shed light on a very simple pruning technique that achieves state-of-the-art (SOTA) performance.
We showcase that magnitude based pruning, speciﬁcally, global magnitude pruning (GP) is sufﬁcient to achieve SOTA performance on a range of neural network architectures.
In certain architectures, the last few layers of a network may get over-pruned.
For these cases, we introduce a straightforward method to mitigate this.
We preserve a certain ﬁxed number of weights in each layer of the network to ensure no layer is over-pruned.
We call this the Minimum Threshold (MT).
We ﬁnd that GP combined with MT when needed, achieves SOTA parameter-accuracy trade-off on all datasets and architectures tested including ResNet-50 and MobileNet-V1 on ImageNet.
Code is available at https://github.com/GPMT-Authors/Global-PruningWith-Minimum-Threshold.
I NTRODUCTION
Neural network pruning remains an important area from both practical perspective (deployment in real world applications) and academic perspective (understanding how to create an efﬁcient architecture).
It is a long standing area of exploration (LeCun et al., 1990a; Hassibi & Stork, 1993a), and was reinvigorated by Han et al.
(2015a).
Since then, much work has been done on trying to ﬁnd different ways of pruning neural networks, such as magnitude-based, gradient or second order based, and regularization-based methods amongst many others.
In this work, we shed light on an often overlooked method that has been seen as a mediocre baseline by the community — global magnitude pruning (GP), and show that it can achieve SOTA pruning performance.
We demonstrate that GP by itself is a strong pruning algorithm and outperforms SOTA pruning algorithms on benchmarks like ResNet-50 and MobileNet-V1 on ImageNet.
We also investigate the pruning behavior of GP and ﬁnd that a simple addition to GP can raise its performance even more.
In contrast to the idea of sparsifying each layer of a neural network to the maximum possible level, we ﬁnd that preserving a certain amount of weights in each layer actually leads to a better pruning scheme, achieving higher accuracy at the same sparsity level.
We call this the Minimum Threshold (MT).
When combined with GP, this technique enhances the pruning performance in most cases.
We conduct a range of experiments to showcase the above and also study detailed ablations to isolate the effects of GP and MT.
We obtain SOTA accuracies on all four sparsity targets on ResNet-50 on
ImageNet.
We obtain SOTA accuracies on other architectures and datasets tested as well.
Finally, GP with MT (GPMT) is very simple conceptually and very easy to implement.
It is a one-shot pruning method in which the weights to be pruned are decided in one-go without needing any iterative or gradual phases.
R ELATED W ORK
Compression of neural networks has become an important research area due to the rapid increase in size of neural networks (Brown et al., 2020), the need for fast inference on edge devices, e.g., a quadrotor’s onboard computer (Camci et al., 2020), and concerns about the carbon footprint of training large neural networks (Strubell et al., 2019).
Over the years, several compression techniques have emerged in the literature (Cheng et al., 2017), such as quantisation, factorisation, attention, knowledge distillation, architecture search and pruning (Almahairi et al., 2016; Ashok et al., 2017;
Iandola et al., 2016; Pham et al., 2018b).
Quantisation techniques which restrict the bitwidth of parameters (Rastegari et al., 2016; Courbariaux et al., 2016) and tensor factorisation and decomposition which aim to break large kernels into smaller components (Mathieu et al., 2013; Gong et al., 2014; Lebedev et al., 2014; Masana et al., 2017) are popular methods.
However, they need to be optimised for speciﬁc architectures.
Attention networks (Almahairi et al., 2016) have two separate networks to focus on only a small patch of the input image.
Training smaller student networks in a process called knowledge distillation (Ashok et al., 2017) has also proved effective, although it can potentially require a large training budget.
Architecture search techniques, such as new kernel design (Iandola et al., 2016) or whole architecture design (Pham et al., 2018a; Tan et al., 2019) have also become popular.
Nevertheless, the large search space size requires ample computational resources to do the architecture search.
Different from all these approaches, we focus on pruning deep neural networks in this work.
As compared to other categories, pruning is more general in nature and has shown strong performance (Gale et al., 2019).
Many pruning techniques have been developed over the years, which use ﬁrst or second order derivatives (LeCun et al., 1990b; Hassibi & Stork, 1993b), gradient based methods (Lee et al., 2018; Wang et al., 2020), sensitivity to or feedback from some objective function (Molchanov et al., 2017; Liu et al., 2020; Lin et al., 2020; de Jorge et al., 2021), distance or similarity measures (Srinivas &
Babu, 2015), regularization-based techniques (Kusupati et al., 2020; Savarese et al., 2020; Wang et al., 2021), and magnitude-based criterion (Str¨om, 1997; Zhu & Gupta, 2018; Park et al., 2020;
Evci et al., 2020; Lee et al., 2021).
Han et al.
(2015b) discovered a key trick to iteratively prune and retrain the network, thereby preserving high accuracy.
Gale et al.
(2019) adopt simple, magnitudebased pruning but employ gradual pruning that requires high computational budget and preset sparsiﬁcation schedules.
Runtime Neural Pruning (Lin et al., 2017) attempts to use reinforcement learning (RL) for compression by training an RL agent to select smaller sub-networks during inference.
He et al.
(2018) design the ﬁrst approach using RL for pruning.
However, RL training approaches typically require additional RL training budgets (Gupta et al.
or iterative pruning to achieve good accuracy (He et al.
In this work, we focus on a simple, effective, yet quite overlooked pruning method — global magnitude pruning (GP).
Although ﬁrst proposed in 1990s (Hoeﬂer et al.
, it has largely been ignored in recent years, generally being relegated to the position of a baseline for comparison (Zhu & Gupta, 2018; Blalock et al., 2020; Lee et al., 2021) rather than a strong pruning technique.
A few recent works use it as one in a possible pool of pruning techniques (See et al., 2016; Frankle &
Carbin, 2018; Gohil et al., 2020) but never study it in detail or adopt it as the main pruning method.
We delve deep into GP and showcase that it can achieve SOTA results with the addition of the
MT technique.
We present SOTA results on various architectures and datasets including ResNetand MobileNet-V1 on ImageNet, and include comprehensive ablation studies and insights on the workings of GP with MT (GPMT)
An advantage of GPMT is that it is a very simple and reliable approach.
There are a few levels of simplicities and robustness to GPMT.
Firstly, it is conceptually very simple and easy to implement.
It does not require any complex pruning frameworks like RL (He et al., 2018) or sparsiﬁcation schedules (Zhu & Gupta, 2018).
Secondly, it is one-shot and does not require any iterative procedure.
Thirdly, it is easily generalizable across architectures and datasets, as shown in the experiments.
Lastly, it is data-independent and does not access the dataset for determining the pruning mask.
M ETHOD
We conduct unstructured weight pruning using magnitude pruning.
Below we describe the key components of our algorithm in more detail.
3. G LOBAL M AGNITUDE P RUNING (GP)
GP is a magnitude based pruning approach whereby weights bigger than a given threshold are kept and weights smaller than the threshold are pruned.
Formally, for a given threshold t and each individual weight w in any layer, the new weight w new is deﬁned as follows:
[Equation]
In contrast to layer-wise pruning, the threshold is not set on a per layer basis but rather a single threshold is set for the entire network.
In this aspect, GP is much more efﬁcient than layer-wise pruning because the threshold does not need to be searched for every layer.
On the other hand, uniform pruning refers to setting the same sparsity target for each layer.
Thus, every layer is pruned by the same percentage.
3. M INIMUM T HRESHOLD (MT)
The Minimum Threshold (MT) refers to a ﬁxed number of weights that are preserved in every layer of the neural network post pruning.
The MT is a scalar value that is ﬁxed before the start of the pruning cycle.
The weights in a layer are sorted by their magnitude and the top MT number of weights are preserved.
For instance, an MT of 500 implies that 500 of the largest weights in every layer need to be preserved post pruning.
If a layer is smaller than the MT number, then it implies that all the weights of that layer must be preserved.
Therefore, the MT is a very simple concept to apply and also computationally inexpensive.
This corresponds to-
[Equation]
where W l ∈ R m denotes the weight vector for layer l , σ is the MT value in terms of the number of weights and min (cid:107) W l (cid:107) 0 indicates the number of non-zero elements in W l .
We explain in the below section how the actual pruning using MT is implemented.
3. T HE P RUNING W ORKFLOW
The pruning pipeline for GP and GP with MT (GPMT) is straightforward.
It consists of pruning the original model followed by ﬁne-tuning for a few epochs.
It is one-shot, therefore, the pruning & ﬁne-tuning cycle does not need to be repeated multiple times.
In terms of the pruning procedure itself, for GP, it consists of doing one pass over the network and pruning the weights according to their magnitude to reach the speciﬁed sparsity target.
For GPMT, it consists of two steps.
Firstly, the model is pruned using GP.
Secondly, the pruned model is evaluated to check if the MT condition is met by all layers or not.
If the condition is not met by a layer, then its sparsity ratio is reduced to meet the MT.
The slack arising from the decrease in sparsity is then redistributed amongst the other layers which do not violate the MT condition.
The redistribution is done in proportion to their existing sparsities so as to preserve their relative sparsities.
This ﬁnishes the pruning cycle and the network is then ﬁne-tuned.
Fig.
1 explains this procedure, while Algorithm 1 gives the pseudocode.
[Figure]
Method
WRN-22-8 on CIFARSparsity
Starting Acc.
Pruned Acc.
[Table] [Table]
MobileNet-
[Table]
MT improves performance even further.
E XPERIMENTS
Below we describe experiments related to ablations on global magnitude pruning (GP) and Minimum Threshold (MT), comparison with state-of-the-art algorithms and experiments on non-vision domains.
We report hyper-parameters and training related information for all the experiments in the appendix (section A.4).
4. I SOLATING AND UNDERSTANDING IMPACT OF GP AND MT OVER DIFFERENT ARCHITECTURES
We conduct detailed ablations to isolate and measure the impact of standalone GP and GP with
MT (GPMT) as compared to a uniform pruning baseline.
We ablate on multiple architectures and sparsity targets.
In addition, we report results averaged over multiple runs where each run uses a different pre-trained model to provide more robustness.
We ﬁrst prune a WRN-22-8 model on
CIFAR-10 at 95% sparsity.
We then ﬁne-tune the model for a few epochs and report the ﬁnal accuracy.
We experiment with different pruning schemes, i.e., uniform pruning, GP and GPMT (see Section 3 for details).
We ﬁnd that GP outperforms uniform pruning.
Furthermore, adding MT improves the performance even more, see Table 1.
Next, we do an experiment on the highly efﬁcient
MobileNet-V2 architecture to see if the above conclusion holds on it too.
We ﬁnd that indeed GP beats uniform pruning in this situation as well, and adding MT improves performance even further (Table 2).
This shows that GP by itself is superior to uniform pruning and adding MT aids GP even more.
[Table] [Table] We then conduct experiments under much tougher conditions, increasing the sparsity rates on both the above-mentioned models to see how the algorithm performs.
We ﬁnd that at 99.9% sparsity, the WRN is able to get decent accuracy and adding MT improves performance even at this sparsity rate (Table 3).
For MobileNet however, using GP only, accuracy drops to 10% and the model is not able to learn.
We discuss this in detail in Section 5.1.
However, on adding the MT, the model is able to learn and the accuracy jumps back to 83% (Table 4).
Alternatively, conducting GP gradually (Zhu & Gupta, 2018) also enables the model to learn well and achieves 87.36% accuracy.
We provide layer-wise weight snapshot for both the models before and after applying MT, to illustrate what MT does (Fig. 2 and Fig. 3).
Thus, in cases of high sparsity, MT is desirable to have.
N u m b e r o f r
e m a i n i n g w e i g h t s
[Figure]
MT (
[Figure]
[Figure]
4. S TATE - OF - THE - ART COMPARISON ON CIFAR-
We also conduct experiments to compare GP and MT to SOTA pruning algorithms on the CIFARdataset.
We compare with various algorithms including SNIP (Lee et al., 2018), SM (Dettmers &
Zettlemoyer, 2019), DSR (Mostafa & Wang, 2019) and DPF (Lin et al., 2020).
We start off with the original model having the same initial accuracy as the other algorithms.
For the WRN-28-8 model we ﬁnd that GP alone beats all the SOTA pruning algorithms at two out of the three sparsity levels (Table 5).
If we incorporate MT, then the accuracy increases further, and the algorithm is able to beat all the SOTA algorithms at all the sparsity levels.
For ResNet-32 as well (Table 6), both GP and
GPMT outperform all the other algorithms at 90% sparsity.
[Figure]
[Figure]
N u m b e r o f r
e m a
i n i n g w e i g h t s [Figure] 4. S TATE - OF - THE - ART COMPARISON ON I MAGE N ET
We apply GP and MT on ResNet-50 on the ImageNet dataset as well and compare to the SOTA pruning algorithms.
We compare with SOTA algorithms like GMP (Zhu & Gupta, 2018), DSR (Mostafa & Wang, 2019), DNW (Wortsman et al., 2019), SM (Dettmers & Zettlemoyer, 2019),
RigL (Evci et al., 2020), DPF (Lin et al., 2020) and STR (Kusupati et al., 2020).
We start with the same original accuracy as the other models.
We ﬁnd that GP alone gets good performance and surpasses all the other algorithms at all the sparsity levels (see Table 7) keeping the numbers of parameters constant.
Adding MT leads to roughly the same performance as GP (slightly higher and slightly lower in certain cases) and this is explained by the fact that all layers in ResNet-50 always maintain enough weights (i.e., ≥ 1,000) at all sparsity levels.
GP incurs higher FLOPs compared to some baselines as it prunes the last layers more vs. the initial layers (which have a higher FLOPs to parameters ratio) compared to some baselines. Optimization for parameters vs. FLOPs is usually a trade-off against each other and joint optimization of parameters and FLOPs can be implemented as future work to reduce the FLOPs of GP. GP still outperforms other baselines like DSR, SM, SM + ERK and RigL + ERK on both accuracy and FLOPs.
We also test another architecture on ImageNet, MobileNet-V1, which is a much smaller and more efﬁcient architecture than ResNet-50.
We start with the same original accuracy as the other models.
For MobileNet-V1, GP alone gets good results at 75% sparsity and outperforms SOTA algorithms by 2.44% on a constant parameter budget basis (Table 8).
On combining with MT, GPMT also surpasses the SOTA algorithms at 90% sparsity by 2.43%.
Since MobileNet-V1 is a very efﬁcient architecture (having only 4.2 million parameters compared to 25.6 million for ResNet-50), MT especially helps at higher sparsities when the layers are pruned very aggressively.
4. G ENERALIZING TO OTHER DOMAINS AND RNN ARCHITECTURES
We experiment with the GP rule on other domains and non-convolutional networks as well to measure the generalizability of the algorithm on different domains and network types.
We experiment on a FastGRNN model (Kusupati et al.
on the HAR-2 Human Activity Recognition dataset (Anguita et al.
More details on the dataset and the experimental setup can be found in
Section A.3.
We test the GP rule under different network conﬁgurations.
We ﬁnd that GP surpasses other pruning algorithms on all the conﬁgurations (Table 9) and successfully prunes the model on a very different architecture and domain.
I NSIGHTS ON GP AND MT 5. N ETWORK ARCHITECTURES CAN AFFECT PRUNING PERFORMANCE
We ﬁnd that network architectures can affect the results of how the pruning algorithm performs quite a lot, especially in the high sparsity domain.
We take the case of the high sparsity experiments between MobileNet-V2 and WRN-22-8, as mentioned in Section 4.1.
In the case of just using GP, we ﬁnd that WRN-22-8 is still able to learn, however, accuracy crashes for MobileNet-V2 and the model is unable to learn anything, getting a chance accuracy rate of 10%.
The reason for this wide discrepancy in learning behavior lies in the shortcut connections (He et al., 2015).
Both WRN-22and MobileNet-V2 use shortcut connections, however, their placement is different.
Referring to Fig.
4, WRN uses identity shortcut connections from Layer 20 to Layer 23.
Identity shortcut connections are simple identity mappings and do not require any extra parameters, and hence, they do not count towards the weights.
However, MobileNet-V2 uses a convolutional shortcut mapping from Layer 52 to Layer 57 and hence, it does add to the model’s weights, and thus, it is pruned by the pruning algorithm.
Both the models have the preceding two layers before the last layer, completely pruned.
However, because WRN uses identity mappings, it is still able to relay information to the last layer, and the model is still able to learn. [Table] [Table] [Table]
[Table]
outperforms other pruning algorithms in all the different network conﬁgurations
[Table] [Figure] [Figure] Pruning algorithms can be susceptible to such catastrophic network disconnection issues especially in the high sparsity domain.
Fortunately the MT can easily overcome this issue.
Retaining a small
MT of 0.02% was sufﬁcient for the MobileNet-V2 model to avoid disconnection and learn successfully.
Hence, retaining a small MT can help in the learning dynamics of models in high sparsity settings.
5. GP PRESERVES OUTPUTS BETTER THAN UNIFORM PRUNING
To understand why GP performs better than uniform pruning, especially at the layer-by-layer level, we measure the Frobenius output distortion between the original model and the pruned model (Park et al., 2020).
We compare the difference in the layer-wise outputs between the original model and the pruned model, and then compute the norm of this difference.
Thus, a smaller norm indicates that the output of the pruned model is closer to the original model and a larger norm indicates that the output is further away.
We compute and plot this layer-wise output distortion on the WRN-22model on CIFAR-10 for the GP and uniform pruning models as compared to the original model (Fig. 5).
As can be seen from the ﬁgure, the distortion is much higher for the uniform pruning model.
Layers 1 and 2 are especially impacted, with the distortion being around 3x that of GP in layer 1.
Thus, GP is able to preserve outputs much closer to the original model as compared to uniform pruning.
This points towards a new direction for empirically understanding the underlying differences between pruning algorithms.
5. W HEN IS MT REQUIRED AND HOW TO DETERMINE THE OPTIMAL VALUE
We ﬁnd that whether MT is required or not depends on the neural network architecture family.
For instance, in our experiments on multiple architectures, we found generally that ResNet architectures did not require MT (tested on ResNet-32 and ResNet-50).
Conversely, MobileNet and WideResNet architectures did require MT (tested on MobileNet-V1, MobileNet-V2, WideResNet-22-8 and
WideResNet-28-8).
Hence, we hypothesize that the requirement for MT depends upon the base family architecture used.
We also found that in architectures that require MT, a decent rule of thumb value for MT is 0.05% of the total weights.
Increments of 0.05% may be tried, i.e., 0.1%, 0.15% or 0.2%, if need be.
We never had to go beyond 0.2%.
In cases of high sparsity, where there is not enough capacity in the network to support 0.05% MT, lower values of MT can be tried, e.g., 0.02%, 0.01%, 0.005% or 0.002%.
We found that the above-stated values were sufﬁcient to get good performance across models and we never had to undertake a more exhaustive MT search.
In most cases, we only tried two to three MT values and achieved good results.
L IMITATIONS AND F UTURE W ORK
A limitation of the GP method is that it can incur higher FLOPs compared to some layer-wise baselines (Table 7). This is because it prunes the last layers more with respect to the initial layers (that have a higher FLOPs to parameter ratio) compared to some baselines. In applications where FLOPs reduction is equally important as parameter reduction, a way to mitigate this would be to add constraints on the FLOPs as well along with the parameters, and thus jointly optimize for both FLOPs and parameters.
We will look at this in future work. Another exciting extension to GP is to do GP gradually instead of one-shot.
As demonstrated in Table 4, doing GP gradually enhances the accuracy performance of GP. We believe this increase in performance is transferable to other datasets and architectures as well, and is another topic that we will look at in the future.
C ONCLUSION
Many methods have been proposed for neural network pruning over the years.
In this paper, we focus on an often overlooked pruning method — global magnitude pruning (GP).
We show that GP by itself is a strong pruning method and achieves SOTA performance on ResNet-50 and MobileNetV1 on ImageNet.
Furthermore, we investigate the pruning behavior of GP in depth and ﬁnd that in certain cases, e.g., at high sparsities, a few layers in the network may be over-pruned.
To rectify this, we propose a novel yet very simple mitigation mechanism called Minimum Threshold (MT).
MT preserves a certain ﬁxed amount of weights in every layer post pruning and ensures no layer is over-pruned.
We ﬁnd that adding MT on top of GP leads to higher performance across many architectures and datasets.
We achieve SOTA results on CIFAR-10 and ImageNet using the above pruning technique.
GP with MT (GPMT) is also easy to implement, is one-shot and is easily portable across architectures and datasets. GP can be further extended to be done gradually instead of oneshot and can lead to more performance gains. A limitation of GPMT is that it can incur higher
FLOPs compared to some layerwise methods and a possible mitigation for this is to add a FLOPs constraint on top of the parameter constraint when doing pruning.
Overall, we ﬁnd that GPMT can be used as a useful baseline for future pruning works given it’s simplicity and performance.
R EPRODUCIBILITY
To help with reproducibility we have released our codebase.
It can be found at https://github.com/GPMT-Authors/Global-Pruning-With-Minimum-Threshold.
Furthermore, we report in detail the experimental setup including all the hyper-parameters for all our experiments in the appendix (Section A.4).
The above two items ensure that all our experiments are reproducible.
Furthermore, our method is conceptually very simple and hence, very easy to implement.
It does not require additional algorithm-speciﬁc hyper-parameters or any architecture-speciﬁc tweaking.
We use standard PyTorch libraries (Paszke et al.
without any modiﬁcation to the model layers.
These make the GPMT method straightforward to implement, and hence, easy to replicate.
Lastly we show that GPMT gets stable results across multiple runs and pre-trained models (Figs.and 7) and hence, aids in reproducibility
A. S PARSIFYING F AST GRNN ON HAR-2 DATASET
HAR-2 dataset that is used in the FastGRNN pruning experiment is a binarized version of the 6class Human Activity Recognition dataset.
From the full-rank model with r W = 9 and r U =as suggested on the STR paper (Kusupati et al., 2020), we apply GP on the matrices W 1 and W 2 .
To do this, we ﬁnd the weight mask by ranking the columns of W 1 and W 2 based on their absolute sum, then we prune the 9 − r new W lowest columns and 80 − r new U lowest columns from W 1 and W respectively.
In the end, we ﬁne-tuned this pruned model by retraining it with FastGRNN’s trainer and applying the weight mask on every epoch.
A. H YPER - PARAMETERS AND EXPERIMENTAL SETUP
No data augmentation is done apart from standard data pre-processing.
Difference in batch size for training and testing in some experiments is due to GPU RAM availability.
Averages reported over three runs.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]