Universal Deep Neural Network Compression
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this paper, we investigate lossy compression of deep neural networks (DNNs)by weight quantization and lossless source coding for memory-efﬁcient deployment.
Whereas the previous work addressed non-universal scalar quantization andentropy coding of DNN weights, we for the ﬁrst time introduce universal DNNcompression by universal vector quantization and universal source coding.
Inparticular, we examine universal randomized lattice quantization of DNNs, whichrandomizes DNN weights by uniform random dithering before lattice quantizationand can perform near-optimally on any source without relying on knowledge ofits probability distribution.
Moreover, we present a method of ﬁne-tuning vectorquantized DNNs to recover the performance loss after quantization.
Our experimental results show that the proposed universal DNN compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the AlexNet (trained on
ImageNet) with compression ratios of 47 .
1 and 42 .
5 , respectively.
Introduction
Compression of deep neural networks (DNNs) has been actively studied in deep learning to developcompact DNN models for memory-efﬁcient and computation-efﬁcient deployment.
Han et al.
[1]showed impressive compression results by weight pruning, k -means clustering and Huffman coding.
It is further optimized in [2] using Hessian-weighted k -means clustering.
Recently, it is shown howsoft weight sharing or soft quantization can be employed for DNN weight quantization in [3, 4].
Onthe other hand, weight pruning is also extensively studied, e g., in [5–9].
In this paper, we focus on
DNN weight quantization, which can be used together with pruning to generate compressed models.
Vector quantization reduces the gap to the rate-distortion bound by jointly quantizing multiple symbols.
Since conjectured by Gersho in [10], lattice quantization has been presumed to be the mostefﬁcient entropy coded vector quantization in the high resolution regime asymptotically, as the rategoes to inﬁnity and the distortion diminishes [11].
Although lattice quantizers are simple and empirically shown to perform well even at ﬁnite rates, their efﬁciency depends on source statistics.
Thus,we consider universal quantization that provides near-optimal performance for any source distribution [12].
Of particular interest is randomized lattice quantization, where uniform random ditheringmakes the distortion independent of the source, and the gap of its rate from the rate-distortion boundat any distortion level is provably no more than 0 .
754 bits per sample for any ﬁnite dimension, andno more than 0 .
5 bits per sample asymptotically as the dimension grows for any sources [13].
From the classical lossy compression results, this paper establishes a universal DNN compressionframework consisting of universal quantization and universal lossless source coding such as Lempel–
Ziv–Welch [14–16] and the Burrows–Wheeler transform [17, 18].
In order to recover any accuracyloss resulting from weight quantization, we furthermore propose a ﬁne-tuning algorithm for vectorquantized DNNs.
The gain of ﬁne-tuning becomes larger as the vector dimension increases, due tothe fact that the number of shared quantized values that are tunable (trainable) in a vector quantizedmodel increases as the dimension increases.
[Figure]
Universal DNN compression
For vector quantization, given N weights denoted by w 1 , . . .
, w N , we generate n -dimensional vectors v 1 , . .
. , v ⌈ N/n ⌉ by concatenating n distinct weights into one vector, e.g., as follows:
[Equation]
where w j = 0 for N + 1 ≤ j ≤ ⌈ N/n ⌉ n , and ⌈ x ⌉ is the smallest integer larger than or equal to x .
Vector quantization partitions these n -dimensional vectors into a ﬁnite number of clusters, and thevectors in each cluster share their quantized value, i.e., the cluster center.
Randomized lattice quantization : Randomized lattice quantization [12] achieves universally goodperformance regardless of source statistics at any rates, and this leads us to the following universal
DNN quantization method.
• We randomize the n -dimensional vectors in (1) by adding uniform random dithers as follows:
[Equation]
where each dithering vector u
i consists of n repetitions of a single uniform random variable
U i ,and U 1 , . . .
U ⌈ N/n ⌉ are independent and identically distributed (i.i
d.) uniform random variablesof support [ − ∆ / 2 , ∆ / 2] ; ∆ is the quantization bin size.
In each dimension, dithering values are
i i.d. uniform, which is sufﬁcient to make the quantization error independent of source statistics.
• After dithering, uniform quantization in each dimension
(i
e., lattice quantization
follows,
i
e.,
[Equation]
where q i is the quantized vector of ˜ v i ; the rounding and scaling are element-wise operations in(3), where the rounding yields the closest integer values of the input.
Remark 1 .
Vector quantization theoretically provides a better rate-distortion trade-off.
However, inpractice, for compression of a ﬁnite number of data, the gain of vector quantization is limited by thecodebook overhead, which becomes more considerable as dimension n increases and becomes thedominant factor that degrades the compression ratio after some point (see Figure 2 and compare thecases of n = 4 and n = 6 ).
Fine-tuning of vector quantized DNNs : We ﬁne-tune the vector quantization codebook to recoverthe loss after quantization.
Each element of a shared quantized vector in the codebook is ﬁne-tunedseparately.
That is, if we have k VQ clusters of n -dimensional vectors, we effectively divide weightsinto nk VQ groups and ﬁne-tune their shared quantized values separately.
The average gradient of thenetwork loss function with respect to weights is computed in each group and used to update theirshared quantized value, as will be clariﬁed below.
Let c i =
[ c i, 1 · · · c i,n ] T be the shared quantized vector present in the codebook for cluster i , and let
I i,j be the index set of all weights that are quantized to the same value
c
i,j from (3), for 1 ≤ i ≤ k VQand 1 ≤ j ≤ n .
The gradient descent for the shared quantized value
c i,j is then given by
[Equation]
4) where t is the iteration time, L is the network loss function, and η is the learning rate.
The individualquantized vectors from (3) are also updated following their shared quantized vectors in the codebook.
In randomized lattice quantization, we would like to clarify that the average gradients are computedfrom the network loss function evaluated at the quantized weights after canceling random dithers(see (5)), while we ﬁne-tune the shared values obtained before canceling random dithers (see (3)).
[Figure]
Universal source coding : Universal source coding algorithms are more convenient in practice than
Huffman coding since they do not require us to estimate source statistics as in Huffman coding.
Moreover, they utilize dictionary-based coding, where the codebook (i.e., dictionary) is built fromsource symbols adaptively in encoding and decoding, and therefore the codebookoverhead is smallerthan Huffman coding (see Remark 1).
For universal compression, the indices in the codebook of the lattice quantization output are passedas an input stream to a universal source coding scheme, which produces a compressed stream.
Thedecoder needs to deploy the codebook that contains the indices and their corresponding ﬁne-tunedshared quantized values for decompression.
Decompression : Using randomized lattice quantization, the encoder and the decoder are assumedto share the information on random dithers or their random seed.
Under this assumption, the decoderdecompresses the ﬁne-tuned lattice quantization output from the compressed stream and then cancelsthe dithering vectors to obtain
[Equation]
(5) which yields the deployed weights of the universally quantized DNN at the inference step.
Experiments and discussion
[Table] [Table]
We evaluate the proposed DNN compressionscheme ﬁrst without pruning for the 32-layer
ResNet [19] (ResNet-32) model on CIFAR10 dataset [20].
We consider two cases for(randomized) lattice quantization where theuniform boundaries in each dimension areset from (a) { i ∆ , i ∈
Z } and (b) { (2 i +1)∆ / 2 , i ∈ Z } , respectively; Z is the set ofintegers.
The quantization bin size ∆ is thesame for both cases, but case (a) has the zeroat a bin boundary while in case (b) the zeroelement is at the center of the middle bin.
For unpruned models, we often have a highvolume of weights concentrated around zero,and thus case (b) that assigns one bin to include all the weights near zero is expected to outperform case (a), which is aligned with our latticequantization results in Figure 2.
However, it is interesting to observe that randomized lattice quantization provides similarly good performance in both cases, which is the main beneﬁt of randomizingthe source by uniform dithering before quantization.
Figure 2 also shows that vector quantizationprovides additional gain over scalar quantization particularly when the compression ratio is large.
Finally, Table 1 summarizes the compression ratios that we obtain from our universal DNN compression method for pruned ResNet-32 and AlexNet [21] models.
The proposed universal DNN compression scheme with the bzip2 [18] universal source coding algorithm yields 47 .
10 × and 42 .
46 ×compression for ResNet-32 and AlexNet, respectively.
Compared with [1, 2, 4] which need to optimize and/or calculate source statistics for compression, we achieved a better trade-off between rate(compression ratio) and distortion (loss in accuracy) through the universal compression of DNNs. {+A AppendixA.1 Experimental details for the results in Figure 
+}
Figure 1 shows the compression results of the unpruned ResNet-32 model.
In particular, we comparethe trade-off between the accuracy and the compression ratio for lattice quantization and randomizedlattice quantization with different values of vector dimension n (see (1)); here, uniform quantizationcorresponds to lattice quantization with dimension n = 1 .
Given the vector dimension n , the weightsfrom all layers of the pre-trained ResNet-32 model are vectorized as in (1) for vector quantization.
Then, lattice quantization or randomized lattice quantization follows.
{+In plain lattice quantization,no random dithering is added before quantization, i.e., we set u i = 
+}
for all i in (2).
We ﬁne-tunethe quantization codebook as explained in lines 61–74.
Finally, we simply use Huffman coding onlyin this experiment to get the compressed models.
The gain of randomized lattice quantization over lattice quantization can be found in Figure 1(a) inparticular for n ≥ 2 and large compression ratios.
We note that randomized lattice quantizers provide similarly good performance in both cases (a) and (b).
Lattice quantization performs well onlyin case (b), where the quantization bins are optimized for given weight distribution.
We emphasizethat randomized lattice quantization is applicable for any network models blindly, regardless of theirweight distribution and with no optimization, while it is guaranteed to yield a good rate-distortiontrade-off close to the optimum within a ﬁxed gap [12].
A.2 Additional results that show the impact of ﬁne-tuning
In Figure 3, we show the performance of universally quantized ResNet-32 models before and afterﬁne-tuning the codebook.
The gain of vector quantization becomes more signiﬁcant after ﬁne-tuning,in particular, as the vector dimension n increases, since there are a more number of shared quantizedvalues trainable in vector quantized models, as we pointed out in lines 36–38.
A.3 Contribution of pruning, quantization, and source coding
Table 2 shows the incremental improvement of the compression ratio for the AlexNet model.
Givena pre-trained AlexNet model, we prune its 90 .
4 % weights and ﬁne-tune remaining unpruned weights,as suggested in [22], which yields the compression ratio of 9 .
11 with the top-1 accuracy of 57 .
The accuracy improves slightly after ﬁne-tuning the pruned model.
Using universal quantizationand universal source coding bzip2 that deploys the Burrows–Wheeler transform [17] for the pruned
AlexNet model, we achieve the compression ratio of 42 .
We also compare Huffman coding tothe universal source coding algorithms, i.e., Lempel–Ziv–Welch (LZW) and bzip2 , in Table 2.
Both
LZW and bzip2 provide better compression ratios than Huffman coding in our experiments.