Neural Network Compression using Transform Coding and Clustering
Abstract
With the deployment of neural networks on mobile devices and the necessity of transmitting neural networks over limited or expensive channels, the ﬁle size of trained model was identiﬁed as bottleneck.
We propose a codec for the compression of neural networks which is based on transform coding for convolutional and dense layers and on clustering for biases and normalizations.
With this codec, we achieve average compression factors between 7.9–9.3 while the accuracy of the compressed networks for image classiﬁcation decreases only by 1%–2%, respectively
Introduction
Deep neural networks spread to many scientiﬁc and industrial applications (1; 2; 3; 4).
Often, the necessity of large amounts of training data, long training duration and the computational complexity of the inference operation are noted as bottlenecks in deep learning pipelines.
More recently, the memory footprint of saved neural networks was recognized as challenge for implementations in which neural networks are not executed on servers or in the cloud but on mobile devices or on embedded devices.
In these use cases, the storage capacities are limited and/or the neural networks need to be transmitted to the devices over limited transmission channels (e.g. app updates).
Therefore, an efﬁcient compression of neural networks is desirable.
General purpose compressors like Deﬂate (combination of Lempel-Ziv-Storer-Szymanski with Huffman coding) perform only poorly on neural networks as the networks consist of many slightly different ﬂoating-point weights.
In this paper, we propose a complete codec pipeline for the compression of neural networks which relies on a transform coding method for the weights of convolutional and dense layers and a clusteringbased compression method for biases and normalizations.
Our codec provides high coding efﬁciency, negligible impact on the desired output of the neural network (e.g. accuracy), reasonable complexity and is applicable to existing neural network models, i.e. no (iterative) retraining is required.
Several related works were proposed in the literature.
These works mainly rely on techniques like quantization and pruning.
The tensorﬂow framework provides a quantization method to convert the trained ﬂoating-point weights to 8 bit ﬁxed-point weights.
We will demonstrate that considerable coding gains on top of those due to quantization can be achieved by our proposed methods.
Han et al. proposed the Deep Compression framework for the efﬁcient compression of neural networks (5).
In addition to quantization, their method is based on an iterative pruning and retraining phase.
In contrast to Deep Compression, we aim at transparent compression of existing network models without the necessity of retraining and without modifying the network architecture.
It is known from other domains like video coding that transparent coding and coding modiﬁed content are different problems (6; 7).
Iandola et al.
propose a novel network architecture called SqueezeNet which particularly aims at having as few weights in the network as possible (8).
We will demonstrate that our method can still reduce the size of this already optimized SqueezeNet network by a factor of up to 7.4.
[Figure]
CodeBook 2n
In this section, we describe our complete codec design with reference to the pipeline illustration in Fig.
The trained neural network model is the input of the codec.
All layers of the network are processed individually.
This simpliﬁes partly retroactive updates of individual layers without transmitting the complete network again.
Weights, biases and normalizations are pre-scaled to use the complete dynamic range of the number representation.
It is observable that the ﬁlters in neural networks contain structural information not completely different from blocks in natural pictures.
Reasoned by this observation, the encoder base for convolutional ﬁlters consists of a two-dimensional discrete cosine transform (2D DCT) followed by a quantization step.
This combination is often referred to as transform coding.
For the DCT, the transformation block size is set accordingly to the size of the ﬁlter (e.g. a 7 ×[-
-]
DCT for a 7 × 7 ﬁlter).
Subsequent to the transformation, the coefﬁcients are quantized.
The bit depth of the quantizer can be tuned according to the needs of the speciﬁc application.
Typical values are 5-6 bit/coefﬁcient with only a small accuracy impact.
The weights of dense layers (also referred to as fully-connected layers) and of 1 × 1 convolutions (no spatial ﬁltering but ﬁltering over the depth of the previous layer, typically used in networks for depth reduction) are arranged block-wise prior to transform coding.
K-means clustering is used for the coding of the biases and normalizations.
The number of clusters is set analogously to the quantizer bit depth according to the quality settings.
Code books are generated for biases and normalizations.
Thereby, the usage of the clustering algorithm is beneﬁcial if less bits are needed for coding the quantizer indices and the code book itself than for coding the values directly.
The clustering approach has the advantage that the distortion is smaller than for uniform quantization.
In consequence, the accuracy of the network is measured to be higher for a given number of quantizer steps.
However, the occurrence of code book indices is also more uniformly distributed.
Due to the higher entropy of this distribution, the compression factor is considerably smaller (see Sec. 3).
In particular the Burrow-Wheeler transform and the move-to-front transform which are both invoked for entropy coding are put at a disadvantage by the uniform distribution.
We chose to use use the same number of quantizer steps for all parameters.
For this reason the clustering was chosen for those network parameters which are too sensible to the higher distortion caused by uniform quantization.
The processed data from the transform coding and from the clustering are entropy coded layer-wise using BZip2, serialized and written to the output ﬁle.
In addition, meta data is stored.
It includes the architecture of the layers in the network, shapes and dimensions of the ﬁlters, details on the block arrangements, scaling factors from the pre-scaling, scaling factors and offsets from the quantizer, and the code books for the clustering.
Evaluation
We evaluate our method using four image classiﬁcation networks: ResNet50 (2), GoogLeNet (3),
AlexNet (9) and SqueezeNet (8).
Rate-distortion analysis is a typical procedure for the evaluation of compression algorithms (10).
The performance of neural networks for image classiﬁ-
[Figure]
networks are encoded, decoded and then used for classiﬁcation downstream
As data, we use the ILSVRC-2012 validation set (50,000 images in 1,000 classes).
To study which algorithms from our overall pipeline contribute how much the the ﬁnal result, we evaluate three subsets of our technology:
In the ﬁrst subset, only quantization is applied to the network weights.
In the second subset, we apply the clustering algorithm to all parameters of all layers.
In the third set, we use our complete pipeline with transform coding and clustering.
The resulting RD curves are visualized in Fig.
The ﬁndings for the two state-of-the-art networks (Fig. 2(a) and 2(b)) are quite clear: The results for the complete pipeline are superior to those of the subsets.
This indicates that all methods in the pipeline have a reason for existence and their coding gains are to some extend additive.
Compression factors of ten or higher are observed without a considerable decrease in accuracy.
AlexNet has the special property that it contains an extraordinary high number of weights and that more than 90% of the weights are located in the ﬁrst dense layer.
As suggested by Han et al.
(5), this disadvantage in the design of the network can only be ﬁxed by pruning and retraining.
Hence, we observe in Fig. 2(c) that transform coding and clustering do not bring any gain on-top of quantization.
Interestingly, our methods enables compression factors of more then ﬁve without much decrease in accuracy even for SqueezeNet in Fig.
2(d).This is an indication that our framework is also beneﬁcial for networks with memory-optimized architecture.
From the underlying data of Fig. 2, we calculate numerical values for the compression factor.
In average, our codec achieves compression factors of 7.9 and 9.3 for accuracy decreases of 1% and 2%, respectively.
We analyze the computational complexity of our algorithms by measuring the encoder and decoder run times using our unoptimized Python code.
For the ﬁnal codec (transform coding for convolutional and dense weights, clustering for biases and normalizations), we measure 29s for the encoder and 7.6s for the decoder.
Conclusion
In this paper, we proposed a codec for the compression of neural networks which is based on transform coding and clustering.
The codec enables a low-complexity and high efﬁcient transparent compression of neural networks.
The impact on the neural network performance is negligible.