Pruning at a Glance: A Structured Class-Blind Pruning Technique for Model Compression
Anonymous Author(s)
AfﬁliationAddress email
Abstract
High performance of deep learning models typically comes at cost of considerablemodel size and computation time.
These factors limit applicability for deploymenton memory and battery constraint devices such as mobile phones or embeddedsystems.
In this work we propose a novel pruning technique that eliminates entireﬁlters and neurons according to their relative L1-norm as compared to the rest of thenetwork, yielding more compression and decreased redundancy in the parameters.
The resulting network is non-sparse, however, much more compact and requiresno special infrastructure for its deployment.
We prove the viability of our methodby achieving 97.4%, 47.8% and 53% compression of LeNet-5, ResNet-56 and
ResNet-110 respectively, exceeding state-of-the-art compression results reportedon ResNet without losing any performance compared to the baseline.
Our approachdoes not only exhibit good performance, but is also easy to implement on manyarchitectures.
Introduction
While deep learning models have become the method of choice for a multitude of applications, theirtraining requires a large number of parameters and extensive computational costs (energy, memoryfootprint, inference time).
This limits their deployment on storage and battery constraint devices,such as mobile phones and embedded systems.
To compress deep learning models without loss inaccuracy, previous work proposed pruning weights by optimizing network’s complexity using secondorder derivative information [2, 5].
While second order derivative introduces a high computationaloverhead, [8, 10] explored low rank approximations to reduce the size of the weight tensors.
Another line of work [4, 15], proposed to prune individual layer weights with the lowest absolute value(nonstructural sparsiﬁcation of layer weights).
[3] followed the same strategy while incorporatingquantization and Huffman coding to further boost compression.
While the aforementioned methodsconsidered every layer independently, [13] proposed to prune the network weights in a class-blindmanner, e.g. individual layer weights are pruned according to their magnitude as compared to allweights in the network.
Noteworthy, all approaches that prune weights non-structurally, generally result in high sparsitymodels that require dedicated hardware and software.
Structured pruning alleviates this by removingwhole ﬁlters or neurons, producing a non-sparse compressed model.
In this regard, [12] proposedchannel-wise pruning according to the L1-norm of the corresponding ﬁlter.
[16] learned a compactmodel based on learning structured sparsity of different parameters.
A data-free algorithm wasimplemented to remove redundant neurons iteratively on fully connected layers in [14].
In [7],connections leading to weak activations were pruned.
Finally, [17] pruned neurons by measuringtheir importance with respect to the penultimate layer.
Generally, in structured pruning, each layer ispruned separately, which requires calculation of layer importance before training.
This work follows the notion of structured pruning, with the key difference of considering allparameters in the network in a class-blind manner instead of each layer independently.
Such astrategy makes the layer importance calculation unnecessary, as stacking all ﬁlters and output featuresin the network provides a global importance insight.
The contribution of this paper includes: i) Proposing a structured class-blind pruning techniqueto compress the network by structurally removing whole ﬁlters and neurons, which results in acompact non-sparse network with the same baseline performance.
ii) Introducing a global importanceparameter as a part of the pruning technique, to devise the pruning percentage of each layer.
iii)
Exceeding state-of-the-art results reported on ResNet-56 and ResNet-110 on the CIFAR10 dataset[9].
Structured class-blind pruning
Consider a network with a convolutional (conv.) layer and a fully connected (fc.) layer.
We denoteall conv.
channels C i belonging to a certain ﬁlter i , where i ∈ [1 , F ] , and F is the total number ofﬁlters in the conv. layer.
Filter channels C i is a 3D ﬁlter, each channel containing 2D kernel weights.
For the fc layer, we denote W m , a 1-D feature space containing all the weights connected to certainoutput feature m , where m ∈ [1 , out ] where out is the number of output features.
We don’t prunethe classiﬁcation layer.
Each pruning iteration in our algorithm is structured as follows:
Algorithm loop over output features of a fc layer 5:
L 1 _ fc ( m ) ← sum ( | W m | ) (cid:46) for output features, calculate L1 of incoming weights 6: norm _ fc ( m ) ← L 1 _ fc ( m ) /nonzero
( W m ) (cid:46) normalize by non-zero weights connected 7: norms ← stack ( norm _ conv, norm _ fc ) (cid:46) stack all normalized norms from all layers 8: sorted ← sort ( norms ) (cid:46) sort ascendingly 9: threshold ← perc ( sorted, p ) (cid:46) threshold based on a percentage p of sorted norms values 10: for i < =
F ; i = i + 1 do 11: if norm _ conv ( i ) < threshold then 12: prune ( Filter i ) (cid:46) remove ﬁlter if its normalized norm is less than threshold 13: for m < = out ; m = m
+ 1 do 14: if norm _ fc ( m ) < threshold then 15: prune ( Neuron m ) (cid:46) same for fc, remove neuron
Importance calculation.
The importance and sensitivity calculation of each layer is part of thepruning method.
Pruning blindly implies constructing a hidden importance score, which correspondsto the normalized L1-norm.
For example, the relevant importance for a certain ﬁlter of a conv layerw.r.t.
all other layers is the ratio between its normalized norm and the sum of all normalized normsacross the network.
Normalization.
As we don’t copy the non-pruned parameters to a new compact model betweenpruning iterations, but only once after the maximum pruning percentage is reached, normalizationbecomes very important.
As we sum over the weights, normalization over the nonzero elementsis needed, otherwise, the most pruned layer will contain the least norms and will have an unfairadvantage to get much more pruned than the non-pruned layers.
Retraining process.
Pruning without further adaption, results in performance loss.
Therefore, inorder to regain base performance, it is necessary for the model to be retrained.
To this end, we applyan iterative pruning schedule that alternates between pruning and retraining.
This is conducted until amaximum compression is reached without losing the base accuracy.
[Table] [Table]
Experiment
In order to assess the efﬁcacy of the proposed method, we evaluate the performance of our techniqueon a set of different networks: ﬁrst, LeNet-5 on MNIST [11]; second, ResNet-56 and ResNet-110([6]) on CIFAR-10 [9].
We use identical training settings as [6], after pruning we retrain with learningrate of 0.05.
For ResNet, when a ﬁlter is pruned, the corresponding batch-normalization weight and bias appliedon that ﬁlter are pruned accordingly.
After all pruning iterations are ﬁnished, a new model with theremaining number of parameters is created.
We report compression results on the existing benchmark [12, 17].
As shown in Table 1, weoutperform the state-of-the-art compression results reported by [17] on both ResNet-56 and ResNet110 with a lower classiﬁcation error even compared to the baseline.
In Table 2, while using one-shot pruning, the inﬂuence of our method’s different components;structured pruning and blindness, is analyzed by removing a component each test, resulting in: i)"Non-Structured" - pruning applied on weights separately.
ii) "Non-Blind" - every layer is prunedindividually.
Then, the effect of the pruning strategy on the method with all its components isanalyzed by comparing: i) "BlindOneshot" - using one-shot pruning and ii) "BlindIterative" - usingiterative pruning .
By comparing the previous versions that are using one-shot pruning, our method has less number ofparameters compared to the other versions; ("Non-Structured" and "Non-Blind").
Finally, applying pruning iteratively is superior to one-shot pruning.
We also show that our method performs better than previously mentioned non-structured weight pruning techniques [4, 15].
Proposedstructured class-blind pruning offers comparable performance as [3], without requiring dedicatedhardware and software to realize compression.
Conclusion
We presented a novel structured pruning method to compress neural networks without losing accuracy.
By pruning layers simultaneously instead of looking at each layer individually, our method combinesall ﬁlters and output features of all layers and prunes them according to a global threshold.
Wehave surpassed state-of-the-art in our results that included ResNet-56 and ResNet-110 on CIFAR-10,compressing more than 47% and 53% respectively.
Also, we showed that only 11K parameters aresufﬁcient to exceed the baseline performance on LeNet-5, compressing more than 97%.
To realize theadvantages of our method, no dedicated hardware or libraries are needed.
It is worth to say that due toremoving whole ﬁlters and neurons, the pruning percentage reﬂects the effective model compressionpercentage.
For the future work, we are dedicated to proving the applicability of our method onseveral different architectures and datasets.
Hence, we plan to experiment on VGG-16, ResNet on
ImageNet and/or other comparable architectures.