Fast On-the-ﬂy Retraining-free Sparsiﬁcation of Convolutional Neural Networks
Abstract
Modern Convolutional Neural Networks (CNNs) are complex, encompassingmillions of parameters.
Their deployment exerts computational, storage and energydemands, particularly on embedded platforms.
Existing approaches to prune orsparsify CNNs require retraining to maintain inference accuracy.
Such retraining isnot feasible in some contexts.
In this paper, we explore the sparsiﬁcation of CNNsby proposing three model-independent methods. Our methods are applied on-theﬂy and require no retraining.
We show that the state-of-the-art models’ weightscan be reduced by up to 73% (compression factor of 3.7 × ) without incurring morethan 5% loss in Top-5 accuracy. Additional ﬁne-tuning gains only 8% in sparsity,which indicates that our fast on-the-ﬂy methods are effective.
Introduction
There has been a signiﬁcant growth in the number of parameters (i.e., layer weights), and the 12 corresponding number of multiply-accumulate operations (MACs), in state-of-the-art CNNs.
Thus,it is to no surprise that several techniques exist for “pruning” or “sparsifying” CNNs (i.e., forcing 14 some model weights to 0) to both compress the model and to save computations during inference.
Examples of these techniques include: iterative pruning and retraining (Cun et al.
[1990], Hassibi and 16 Stork [1993], Dong et al.
[2017], Sun et al.
[2016], Mathew et al.
[2017]), Huffman coding (Han et al.[2015]), exploiting granularity (Mao et al.
[2017b], Han et al.
[2016]), structural pruning of network 18 connections (Wen et al.
[2016], Mao et al.
[2017a], Anwar et al.
[2015], Niculae and Blondel [2017]),and Knowledge Distillation (KD) (Hinton et al.
A common theme to the aforementioned techniques is that they require a retraining of the model to 21 ﬁne-tune the remaining non-zero weights and maintain inference accuracy.
Such retraining, whilefeasible in some contexts, is not feasible in others, particularly industrial ones.
For example, for 23 mobile platforms, a machine learning model is typically embedded within an app for the platformthat the user directly downloads.
The app utilizes the vendor’s platform runtime support (often in 25 the form of a library) to load and use the model.
Thus, the platform vendor must sparsify the modelat runtime, i.e., on-the-ﬂy , within the library with no opportunity to retrain the model.
Further, the 27 vendor rarely has access to the labelled data used to train the model.
While techniques such as
Knowledge Distillation (Hinton et al.
[2015]) can address this lack of access, it is not possible to 29 apply it on-the-ﬂy.
In this paper, we develop fast retraining-free sparsiﬁcation methods that can be deployed for on-the-ﬂy 31 sparsiﬁcation of CNNs in the contexts described above.
There is an inherent trade-off between sparsityand inference accuracy.
Our goal is to develop model-independent methods that result in large sparsity 33 with little loss to inference accuracy.
We develop three model-independent sparsiﬁcation methods:ﬂat , triangular , and relative .
We implement these methods in TensorFlow and use the framework 35 to evaluate the sparsiﬁcation of several pretrained models: Inception-v3, MobileNet-v1, ResNet, VGG, and AlexNet.
Our evaluation shows that up to 81% of layer weights in some models may be 37 forced to 0, incurring only a 5% loss in inference accuracy.
While the relative method appears to bemore effective for some models, the triangular method is more effective for others.
Thus, predictive 39 modeling is needed to identify, at run-time, the optimal choice of method and it hyper-parameters.
Sparsiﬁcation Methods
Sparsity in a CNN stems from three main sources: (1) weights within convolution ( Conv ) and fully- 42 connected ( FC ) layers (some of these weights may be zero or may be forced to zero); (2) activationsof layers, where the often-applied ReLU operation results in many zeros (Sze et al.
[2017]); and (3) 44 input data, which may be sparse.
In this paper, we focus on the ﬁrst source of sparsity, in both Convand fully connected layers.
This form of sparsity can be determined a priori, which alleviates the 46 need for specialized hardware accelerators.
The input to our framework is a CNN that has L layers, numbered 1 . . .
L . The weights of each layer 48 l are denoted by ω l . We sparsify these weights using a sparsiﬁcation function, S , which takes as input 49 ω l and a threshold τ l from a vector of thresholds T . Each weight i of ω l is modiﬁed by S as follows:
distribute
Triangular
[Figure]
R e l a t i v e T H
S
R e l a t i v e T H S R e l a t i v e T H
S
R e l a t i v e T H S R e l a t i v e T H
S
R e l a t i v e T H S R e l a t i v e T H
S
R e l a t i v e T H S
[Figure]
R e l a t i v e T H S R e l a t i v e T H
S
R e l a t i v e T H S R e l a t i v e T H
S
R e l a t i v e T H S ( 3 ) R e l a t
i v e T H S
R e l a t i v
[Figure]
T H S
[Figure]
[Equation]
1) where τ l = T ( l ) is the threshold used for layer l .
Thus, applying a single threshold τ l forces weights 51 within − τ l and + τ l in value to become 0.
Our use of thresholds to sparsify layers is motivated by the 52 fact that recent CNNs’ weights are distributed around the value 0 .
The choice of the values of the elements of the vector T deﬁnes a sparsiﬁcation method .
These values 54 impact the resulting sparsity and inference accuracy.
We deﬁne and compare three sparsiﬁcationmethods.
The ﬂat method deﬁnes a constant threshold τ for all layers, irrespective of the distribution 56 of their corresponding weights.
The triangular method is inspired by the size variation of layers insome state-of-the-art CNNs, where the early layers have smaller number of parameters than latter 58 layers.
Finally, the relative method deﬁnes a unique threshold for each layer that sparsiﬁes a certainpercentage of the weights in the layer.
The three methods are depicted graphically in Figure 1a.
The 60 high level work-ﬂow of the sparsiﬁcation framework is depicted in Figure 1b.
Flat Method This method deﬁnes a constant threshold τ for all layers, irrespective of the dis- 62 tribution of their corresponding weights.
It is graphically depicted in the top of Figure 1a.
Theweights of the layers are proﬁled to determine the span σ min =
min ∀ linL ( max ( ω l ) − min ( ω l )) .
Thisspan corresponds to the layer k having the smallest range of weights within the pretrained model. This span is used as an upper-bound value for our ﬂat threshold τ . Since using σ min as a thresholdeliminates all the weights in layer k and is likely to adversely affect the accuracy of the sparsiﬁed 67 model, we use a fraction δ , 0 ≤ δ ≤ 1 , of the span: [Equation]
where δ is a parameter of the method that can be varied to achieve different degrees of model sparsity.
Triangular Method The triangular method is deﬁned by two thresholds τ min and τ max for respec- 70 tively the ﬁrst convolution layer (i.e, layer 1) and the last fully connected layer (i.e., layer L ).
Theyrepresent the thresholds at the tip and the base of the triangle in middle part of Figure 1a.
These 72 thresholds are determined by the span of the weights in each of the two layers.
Thus,
[Equation]
where σ conv is the span of the weights in the ﬁrst convolution layer, deﬁned in a similar way as for the 74 ﬂat method, and it represents an upper bound on τ min .
Thus, δ conv is a fraction that ranges between0 and 1 .
Similarly, σ fc is the span of the weights in the last fully connected layer and it represents an 76 upper bound on τ max .
Thus, δ fc is a fraction that ranges between 0 and 1 .
Relative Method In particular, it uses the δ th percentile of distribution’s weight in layer l denoted 78 by δ l .
Thus each element of the vector T ( l ) =
τ l is deﬁned as:
[Equation]
where 0 ≤ δ l ≤ 1 deﬁnes the desired percentile of the δ l of zero weights in each layers.
We evaluate our sparsiﬁcation methods using TensorFlow v1.4 with CUDA runtime and driver v8.
The evaluation of Top-5 accuracy is done on an NVIDIA’s GeForce GTX 1060 with a host running [Table]
[Figure] Ubuntu 14.04, kernel v3.19 using ImageNet (Jia Deng et al.
Figures 2a, 2b, 2c, 2d, and 2e 84 show the inference accuracy as a function of the introduced sparsity by each method.
They reﬂect thatsigniﬁcant sparsity can be obtained with a small reduction in inference accuracy.
With less than 5% 86 reduction in accuracy, we gain 51% sparsity (2.04 × compression factor), 50% (2 × ), 62% (2.63 × ),70% (3.33 × ), and 73% (3.7 × ) for the models.
This validates our approach.
Further, the ﬁgures reﬂect that the relative method outperforms the other two methods for the 89 Inception-v3, VGG and ResNet, but the triangular one outperforms the other two for MobileNet-v1and Alexnet.
This is likely due to the structure of the models.
MobileNet-v1 and Alexnet have a 91 gradual linear increase in the size of the convolution layers, making the triangular method moreeffective.
In contrast, the other models have no such increase, making the relative method more 93 effective.
As a case-in-point, ResNet has 152 conv layers with variable sizes, which makes thetriangular method less effective, as seen by the drop in accuracy in Figure 2c.
An interesting observation is that for Alexnet, introducing the ﬁrst 50% sparsity suffers little drop 96 in accuracy.
Sparsity value for the the same little drop are 35%, 30%, 41%, and 42% for the othermodels, This shows there exists signiﬁcant redundancy within CNNs.
Han et al.
[2015] observe the 98 same with their Caffe implementation of Alexnet, for which they iteratively prune and retrain.
Ourevaluation validates their result across other models using faster on-the-ﬂy methods.
The authors 100 also do gain around 80% sparsity by pruning (i.e., without retraining)
Alexnet with L2 regularization.
Their method with no retraining is not speciﬁed and it is unclear if it applies to other CNNs or not.
Fine-tuning .
We explore what can be achieved by some ﬁne-tuning of our methods, still with no 103 retraining, in order to gain more sparsity. We do so to determine the effectiveness of our on-the-ﬂymethods, since the ﬁne-tuning is not likely feasible in out context.
We focus on the relative method 105 and start with a baseline sparsity.
We then vary the degree of sparsity of each layer in turn around thebase sparsity, attempting to maintain a no more than 5% drop in inference accuracy.
The results for 107 only AlexNet (due to space limitations) are shown in the table in Figure 2f.
The baseline sparsityis selected as 70%.
It is possible for some layers, particularly larger ones, to have higher sparsity, 109 while smaller layers are more sensitive to sparsiﬁcation and must have lower sparsity.
Nonetheless,there is a gain of 8% in overall model sparsity.
This value is 4%, 3%, 2% , and 5% for Inception-v3, 111 Mobilenet-v1, ResNet, and VGG, respectively.
Since this gain comes at the expense of an explorationof different sparsity ratios for the layers and thus more computations, it is not feasible in the contexts 113 we explore.
However, the gain is not signiﬁcant to render our on-the-ﬂy methods inefﬁcient on theirown without further tuning.
Concluding Remarks
We explored sparsiﬁcation without retraining of CNNs and proposed three model-independent 117 methods doing so.
We experimentally evaluated these methods and showed that they can resultin up to 73% sparsity with less than 5% drop in inference accuracy.
However, there is no single 119 method that works best for all models.
There are two key directions for future work.
The ﬁrst is toexplore heuristics for selecting a sparsiﬁcation method based on the CNN model and possibly ﬁne 121 tune the parameters of the methods.
The second is to realize the beneﬁt of the sparsity in the model’simplementation on the NNlib library, which