I TERATIVE D EEP C OMPRESSION : C OMPRESSING D EEP N ETWORKS FOR C LASSIFICATION AND S EMANTIC S EGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Machine learning and in particular deep learning approaches have outperformed many traditional techniques in accomplishing complex tasks such as image classﬁcation (Krizhevsky et al., 2012), natural language processing or speech recognition (Hinton et al., 2012).
Most of the state-of-the art deep networks have complex architecture and use a vast number of parameters to reach this superior performance.
Though these networks use a large number of learnable parameters, those parameters present signiﬁcant redundancy (de Freitas, 2013).
Therefore, it is possible to compress the network without much affecting its accuracy by eliminating those redundant and unimportant parameters.
In this work, we propose a three stage compression pipeline, which consists of pruning, weight sharing and quantization to compress deep neural networks.
Our novel pruning technique combines magnitude based ones with dense sparse dense Han et al.
(2016) ideas and iteratively ﬁnds for each layer its achievable sparsity instead of selecting a single threshold for the whole network.
Unlike previous works, where compression is only applied on networks performing classiﬁcation, we evaluate and perform compression on networks for classiﬁcation as well as semantic segmentation, which is greatly useful for understanding scenes in autonomous driving.
We tested our method on LeNet-5 and FCNs, performing classiﬁcation and semantic segmentation, respectively.
With LeNet-5 on MNIST, pruning reduces the number of parameters by 15.3 times and storage requirement from 1.7 MB to 0.007 MB with accuracy loss of 0.03%.
With FCN8 on Cityscapes, we decrease the number of parameters by 8 times and reduce the storage requirement from 537.47 MB to 18. MB with class-wise intersection-over-union (IoU) loss of 4.93% on the validation data.
I NTRODUCTION
Deep learning found its importance in different domains to solve tasks ranging from small-scale to large-scale problems.
It has remarkably achieved human-level performance in image recognition tasks (He et al., 2016).
Existing deep neural networks are very powerful but they require considerable storage and memory bandwidth.
For example, AlexNet by Krizhevsky et al.
(2012) hasmillion parameters, which is over 100 times more than LeCun et al.
(1989) conventional model LeNet - 5 (LeCun et al., 1998).
More parameters require more storage space and more computation.
This makes it difﬁcult to deploy deep neural networks on embedded devices and mobile platforms performing real time processing with limited memory and processing units.
Still, not all weights in a network are important and there is in fact high redundancy in these parameters (Guo et al., 2016).
Choosing the right and important parameters is essential to do the optimization between the network efﬁciency and resources used with minimal accuracy loss.
While some success has been achieved in compressing deep neural networks performing classiﬁcation, it has not been studied for networks performing semantic segmentation, where each pixel in the image is classiﬁed to a category making it possible to recognize multiple objects in an image.
Semantic segmentation is extremely relevant for the self driving domain, where scenes have to be understood in real time, and is an important target for compression as deep learning methods have achieved signiﬁcant success on it (Wu et al., 2016).
To achieve this goal, we present “iterative deep compression”: a three stage pipeline as illustrated in ﬁgure 1, similar to “deep compression” proposed by Han et al. (2015).
As is the case for many pipelines in this ﬁeld, ﬁrst we prune the number of parameters in the network by removing redundant and unimportant connections.
To the remaining connections, we apply weight sharing so that the same weight can be shared by multiple connections across the network.
Thus, only the shared weights and the indices mapping each remaining weight to its shared weight need to be stored.
Finally, we apply quantization to reduce the number of bits required to store these shared weights.
Our major contribution is the development of a pruning method that combines magnitude pruning with ideas from dense sparse dense (Han et al., 2016).
That is done by iteratively ﬁnding for each layer its achievable sparsity instead of selecting a single threshold for the whole network and then applying dense sparse retraining while pruning.
Moreover, we also investigate several options for weight sharing, such as sharing weights within layer and across the layers, examining the impact of each on network performance.
We also explore different clustering techniques comparing their optimal number of clusters and resulting network performance.
In this work we compressed two state-of-the-art deep neural network architectures.
The ﬁrst performs classiﬁcation on MNIST dataset (LeCun et al., 1998) while the second does semantic segmentation on Cityscapes (Cordts et al., 2016).
While in some stages of our pipeline we may explore more than one technique, we always select the one performing best for our next stage.
R ELATED W ORK
In order to compress deep networks, a variety of methods have been proposed.
Chen et al.
(2015) accomplish network compression by HashNets, where they group the parameters of the network into hash buckets.
These parameters are tuned with standard backpropagation during training, however, their binning is pre-determined by a hash function.
They exploit the inherent redundancy in neural networks to achieve drastic reductions in model sizes.
Gong et al.
(2014) proposed another way to compress deep networks by vector quantization and found that the product quantization gives a good balance between model size and accuracy.
They achieve 16 - 24 times compression on the state-ofthe art CNN by classiﬁcation accuracy loss of 1%.
But both the compression methods consider only fully connected layers in the network.
Lin et al.
(2013) attempted to reduce the number of parameters by global average pooling the feature maps from the last convolutional layer of the network.
Hubara et al.
(2016) recently proposed binarized neural networks, where they use binary weights and activations during training.
This bit wise operations substantially improve power-efﬁciency, but on the cost of accuracy loss.
Our work is based on the idea of network pruning.
It helps to reduce both network complexity and over ﬁtting (Hubara et al., 2016).
Hassibi & Stork (1993) perform network pruning by using the information from second derivatives of the loss function and propose that such pruning is more accurate than magnitude based pruning, where smaller magnitude weights are eliminated.
Their method does not require retraining the network after pruning, however, it is computationally expensive for large networks.
Han et al.
(2015) proposed magnitude based pruning while simultaneously retraining the network.
They signiﬁcantly reduce the number of parameters in the network without much impacting the accuracy, however, there is always the risk of deleting the important parameters.
Guo et al. (2016) incorporate connection splicing in order to avoid incorrect pruning.
In their splicing operation they enable the recovery of pruned connections if they are found important at any time, but they do not account for correlation between weights (Yang et al., 2016b).
Recently, Han et al.
(2015) proposed a deep compression pipeline.
First they prune the unimportant connections based on their magnitude, then perform quantization to enforce weight sharing and ﬁnally apply Huffman encoding for lossless data compression.
Their experiment on AlexNet reduced the number of parameters by 9x without loss of accuracy.
Because of their signiﬁcant effectiveness, our work is partly inspired by their work.
However, we contribute to this work by improving the weight sharing technique and introducing dynamic threshold calculation for pruning.
We also contribute to the work by Han et al.
on dense sparse dense training by calculating the achievable sparsity in each layer.
M ETHOD
In this section we present our compression pipeline also depicted in ﬁgure 1, with its Pruning ,
Weight Sharing and Quantization steps.
Pruning, From Dense to Sparse: The objective of the pruning technique is to identify and remove unimportant and redundant weights that would least affect the performance of the network.
For this, we use the simple heuristic of quantifying the importance of weights using their absolute values (Han et al., 2015).
In other words, the bigger the absolute weight value, the more important it is.
But it is crucial to select one threshold weight value below which all the weights could be considered unimportant.
We start with a pre-trained fully connected network.
One approach is to choose a single threshold value for the entire network.
However, this does not take into account the difference in the weight distribution across the layers.
For example, as depicted in ﬁgure 2, the weight distribution of two different dense layers of the LeNet architecture differs.
Therefore, we select the threshold value layerwise which means a different threshold value for each layer.
To select the threshold values we introduced a hyper parameter to the network, the sparsity .
This is the starting sparsity, initialized to 100% for each layer.
It gives the percentage of weights of the network that are pruned, that is, set to 0 (Han et al., 2016).
We determine the sparsity for each layer using algorithm 1 .
We start from output to ﬁrst convolutional layer.
This process gives the approximate value of achievable sparsity in each layer without much impacting the accuracy of the network.
In this algorithm 1, ﬁrst, we initialize the sparsity for each layer and receive as input the pre-trained network and a parameter called ‘error tolerance’ for the network.
Error tolerance is computed by adding a very small degradation tolerance to baseline error of the model, where baseline error is the error of the pre-trained network before pruning.
Then, we initialize sparsity to 100% for a layer, that is, setting all its weights to zero and then we evaluate error of the network as depicted in step 3 and 5.
In step 6, we compare this error with the deﬁned error tolerance of the network.
If the current error is more than the deﬁned error tolerance, we train the network and again prune it at same sparsity level.
We repeat this process until the error of the network converges at that sparsity level and it stops improving by further retraining.
We decrease the sparsity in the layer by 1% after every such convergence, as shown in step 12.
Once we got the approximated sparsity level for each layer, we start pruning our network, depicted in algorithm 2.
The achieved sparsity for each layer and the overall error tolerance of the network are received as parameters from the last step.
We also receive the original pre-trained weights and an extra sparsity for each layer.
This extra sparsity will be added to the starting sparsity found in the previous algorithm.
Then we calculate the threshold value for each layer λ L that is needed to make the layer Sparsity L sparse, pruning the weights of each layer.
However, mistakenly pruning important connections or over pruning could cause high accuracy loss (Guo et al., 2016).
In order to compensate the unexpected loss, we retrain the network and enable the connection recovery, that is, a pruned value is not henceforth always zero as is done in Han et al. (2015), but can regain a positive absolute value after retraining.
Finding important connections in a certain network is also extremely difﬁcult, therefore, we conduct pruning and training iteratively and continually maintain the old weights from the previous iteration.
After each iteration, a pruning threshold is selected from trained weights of the previous iteration.
Hence, we are dynamically calculating the threshold for each layer.
After pruning, we evaluate the pruned model and compare this current error with the deﬁned error tolerance of the network.
If the current error is greater than the error tolerance, we retrain the network and prune it again as can be seen in step 11 and 12.
All the layers of the network are pruned and trained together because of the dependence of each layer to its previous layer.
We keep track of the error from every iteration and we stop retraining if the network converges, that is, error is not improving any more at deﬁned sparsity level.
This error has been compared with the error tolerance of the network as depicted in step 15.
We decrease the sparsity by 1% and again start the iterative process pruning and retraining if the current error is still less than the error tolerance.
Otherwise, it returns pruned network as can be seen in step 19.
This sums up pruning of the network where we iteratively undergo the dense sparse phase under the constraint of sparsity and error tolerance.
before and after codebook pruning
Weight Sharing limits the number of effective weights to store by ﬁnding the weight values that could be shared across multiple connections without affecting the accuracy of the network.
Weights can be shared, ﬁrst, only within a layer, that is ﬁnding shared weights among the multiple connections within a layer and second, across all the layers, that is, ﬁnding shared weights among multiple connections across all the layers.
This can be done using clustering.
The idea is that all the weights that belong to one cluster share the same weight value, partitioning the n original weights into k clusters such that n >> k .
The performance of the network depends upon the quality of the clustering algorithm.
The value of the cluster’s centroid is assigned to all the weight values within that cluster.
So at the end we need to store only the centroid values to represent the weight values of the network instead of storing all the weight values individually.
We examine two different clustering algorithms: mean shift clustering and k-means.
K-means is very sensitive to the initial position of the cluster centers (Celebi et al., 2013), so we examine two different initialization methods: random and linear initialization (Han et al., 2015).
The stepwise algorithm for k-means with linear initialization is illustrated in algorithm 3.
Reﬁnement of the shared weights As depicted in ﬁgure 3, linear initialization results in cluster centers scattered over the entire range.
However, cluster centers at the extreme ends of the distri- bution represent few original weights.
It could be the case, however, that those underrepresented weight values are really important to the network’s performance.
Thus we will evaluate the effects of merging them with the other shared weights, by pruning the code book.
Pruning of the codebook To check the possibility of reducing down the number of shared weights, we ﬁrst examine the number of weights mapped to each center in the codebook.
We found that there are some codes that are being assigned to zero or very few weight values.
So we prune such codes.
For this pruning, we empirically chose a ﬁxed threshold of 25.
Therefore, any code which is assigned to less than 25 weight values would be pruned and removed from the codebook.
All the weight values assigned to such codes/cluster centers would be reassigned to next closest cluster center.
In our experiments we have observed that this does not impact much the accuracy of the network.
The cluster centers before and after codebook pruning are shown in ﬁgure 4.
Fine tuning of the codebook To ﬁne tune the cluster centers we use the gradient approach (Han et al., 2015).
First, gradients for each weight value are calculated using Theano symbolic differentiation.
Then, the gradients of the weight values that belong to one cluster are grouped together.
All the calculated gradients are grouped according to the cluster they belong and summed to give one value per cluster.
This value is then multiplied with the learning rate and subtracted from the cluster centers obtained from the previous step.
These new values are called ﬁne-tuned cluster centers.
Now these ﬁne-tuned values are used as shared weights for the network.
Quantization Quantization is performed to reduce the number of bits required to represent each shared weight value.
We use ﬁxed point quantization to convert ﬂoating point weight values to ﬁxed point weight values.
Fixed point implementation facilitates the potential deployment on embedded systems (Lin et al., 2016).
Based on state-of-the-art by Han et al. we convert the 32-bit ﬂoating point weight values to 8-bit ﬁxed point values for each fully convolutional layer and to 5-bit ﬁxed point values for each fully connected layer.
The conversion of ﬂoating point to ﬁxed point is done using the following formulation, where q format is the user deﬁned bit width (Lin et al., 2016):
[Equation]
E XPERIMENTS AND R ESULTS
We experimentally analyzed the three stage pipeline discussed above and applied it to some popular pre-trained networks.
In section 4.1 and 4.2 we present our experiment results for LeNet-5 on
MNIST and FCN8 on cityscapes respectively.
4. R ESULTS FOR L E N ET -5 ON MNIST
We trained LeNet on the training set of MNIST to accomplish the classiﬁcation task.
The optimizer
Adam was used to train the network.
We trained the network for 15 epochs with learning rate of 10 − 3 .
Our trained LeNet has achieved the accuracy of 99.30% on the validation set of MNIST.
So, we have 99.30% as baseline accuracy and consequently 0.70% as baseline error rate of the network.
Pruning The ﬁrst step was to select the threshold used to prune the network.
Table 1 gives the comparison between our two threshold selection methods.
In the ﬁrst, we ﬁxed one threshold value for all layers while in the second we dynamically calculate a different threshold for each layer.
Dynamic threshold selection achieved substantially better results.
[Table]
While initializing the sparsity levels for each layer, we increased the above mentioned percentage by 5 to 6% following our algorithm.
Based on these deﬁned sparsity levels, the initial threshold has been calculated for each layer.
We prune each layer of the network with these threshold values.
A plot of the weight distribution of second convolutional layer before and after ﬁrst pruning is shown in ﬁgure 5.
We iteratively perform pruning and retraining using the same optimizer and learning rate on all network weights until the current validation tolerance.
Our pruning runs for 42 iterations in total.
Figure 6 depicts the convergence of error rate and accuracy rate after each iteration at initialized sparsity level.
As it can be seen, after a few iterations the accuracy of the network is not improving anymore so we decreased the sparsity level by 1% in each layer and again start iteratively pruning and retraining at new decreased sparsity levels.
In table 2 we compare layer by layer pruning results between our method and Han et al. [Table] [Figure]
Weight Sharing
Next, we applied weight sharing techniques on these remaining weights of the network.
We perform clustering on all the non-zero weights.
Weight sharing further reduced the number of weights to be stored, as depicted in table 3, where the comparison of each weight sharing technique we explored is shown in terms of number of clusters and accuracy achieved.
[Table]
So k-means clustering with linear initialization across all the layers gives the best results.
We ﬁne tune these cluster centers by gradient approach discussed in section 3.
The accuracy of the network remained unchanged after ﬁne tuning.
Quantization Next
[Table]
we applied quantization to reduce the number of bits required to store the tuned cluster centers.
We applied 8-bit ﬁxed point quantization for convolutional layers and 5-bit ﬁxed point quantization for fully connected layers.
Table 4 gives the accuracy statistics and reduction in size after each stage of pipeline [Table] [Table] codebook
[Table]
4. R ESULTS FOR FCN ON CITYSCAPES
We conducted second experiment on FCN (Yang et al., 2016a) performing semantic segmentation task on the Cityscapes dataset.
Our trained FCN8 has achieved the baseline IU of 64.75% and baseline error rate of 35.25% on the validation set of cityscapes.
Pruning In this experiment, dynamic threshold calculation is used to calculate the threshold as it is evident from table 1 that dynamic threshold selection achieved substantially better results.
We iteratively perform the pruning and retraining operation with the same optimizer and learning rate until the current validation error of the network becomes less than the deﬁned error tolerance.
It runs for 32 iterations.
Figure 7 depicts the convergence of error rate and accuracy rate after each iteration at initialized sparsity level.
Plot of the weight distribution of ﬁrst convolutional layer before and after ﬁrst pruning is shown in ﬁgure 8.
Table 5 depicts the compression statistics after pruning and table 6 depicts the mean IU statistics and reduction in storage after pruning.
[Figure]
Weight sharing To ﬁnd the shared weights, we applied k-means clustering with linear initialization as it achieved substantially better results in the previous experiment.
To ﬁnd the optimum number of clusters, we evaluate the number of clusters ranging from 10 to 1200.
Figure 9 depicts the achieved IOU score corresponding to number of clusters found in this range.
It can been seen that as the number of clusters increases the IOU score is improving, however, the improvement is very slow.
Due to time constraints, we did not continue with weight sharing and directly applied quantization to the non-zero weights after pruning.
[Table]
Quantization We applied 8 bit quantization to pruned weights.
The size of the network reduced from 537.47 MB to 18.23 MB as depicted in table 6.
Stages of pipleline
[Table]
C ONCLUSION AND F UTURE W ORK
Deep learning approaches have demonstrated that they can outperform many traditional techniques, but because of their complex architecture in terms of more stacked layers and a large number of parameters, it is challenging to deploy these deep networks on mobile devices with limited hardware requiting real time predictions.
This work contributes to the previous research on compression of deep networks performing classiﬁcation.
Moreover, we have also presented the compression of a network that performs semantic segmentation.
We implemented a three stage deep compression pipeline of pruning, weight sharing, and quantization.
Using different sparsity levels, we calculate different thresholds in each layer to perform pruning.
Our “layerwise threshold” initialization method has shown promise in providing a good trade-off between sparsity and network performance.
We also examined two different weight sharing possibilities: ﬁnding shared weights within a layer or across all the layers of the network.
We extend the previous work by Han et al.
(2015) with kmeans using linear initialization by merging underrepresented shared weights.
Finally, we quantize these shared weights, based on state-of-the-art by Han et al.
(2015), in which they used ﬁxed point quantization.
The experimental results show that our method compresses the number of parameters in LeNet - 5 and FCN by 15.3x and 8x, respectively.
We reduce the storage requirement for LeNet from 1.7 MB to 0.007 MB and for FCN from 547 MB to 18.23 MB.
The reduction in storage for the FCN has extended network compression to more sophisticated tasks such as object detection and segmentation.
In our work, iterative pruning is performed to get rid of unimportant connections.
This process takes 35 days for LeNet with 430180 parameters on MNIST and 42 days for FCN with approximately 134M parameters on Cityscape to reach an optimum level of sparsity and performance.
While not prohibitive, this process could be sped up by different approaches such as masking by Guo et al.
(2016), where they have a gradient based heuristic for determining which weights do not come back.
It would also be interesting to carry out the experiment on bigger networks and datasets for classiﬁcation.
Also recently, smaller deep neural network architectures, such as SqueezeNet, by Han et al. achieved AlexNet-level accuracy on ImageNet with 50x fewer parameters and is thus feasible to be deployed on FPGAs and other hardware with limited memory (Iandola et al., 2016).
Compressing already smaller deep neural networks could further reduce the number of parameters and make them even more efﬁcient on embedded systems.