From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper15/Authors
comment: Dear Reviewer,

We appreciate the reviewer for your kind review and suggestions. Please find our revised paper that we have prepared to address your questions and suggestions.

1. We first would like to clarify that the results in Figure~2 are obtained "with no pruning" for the ResNet-32 model. However, in Table~1, we show the compression ratios for "pruned" ResNet-32 and AlexNet models. That is the reason why the compression ratio in Table~1 is better than the ones in Figure~2. We clarified this point in our revised paper (see Figure~2).

2. For the pruned ResNet-32 model, we included additional results with better accuracy in the revised paper (see Table~1). To get better accuracy, we need to decrease the pruning ratio to 87.48%, and then we have the top-1 accuracy of 92.57% with the compression ratio of 32.87, which is still better than the ones from the previous work. We have updated Table~1 with this result in our revised paper.

3. The benefit of using universal source coding is explained in lines 75-83. We are clarifying it more below. In Table~2, we also compared the performance of Huffman coding to universal source coding algorithms, i.e., Lempel-Ziv-Welch (LZW) and bzip2, for the pruned AlexNet model (see Appendex~A.3 and Table~2).

Huffman codes are known to be optimal for encoding individual source symbols into binary codewords. However, its optimality comes with the assumptions that source symbols are independent and that their statistics are known. Exploiting the dependency of source symbols, if exists, can improve the compression efficiency. For example, one could encode blocks of source symbols instead of individual symbols. Such an extension of Huffman coding is called as extended Huffman coding or block Huffman coding. However, extended Huffman coding may be even difficult to use since they require us to know or to accurately estimate the joint distribution of source symbols. Given a specific neural network, empirical approximation of the source statistics can be obtained by computing weight histograms, but it is time-consuming, inaccurate, and inapplicable to other networks or even to the same network if the weight distribution changes after fine-tuning.

In contrast, universal source codes asymptotically achieve the optimal performance for any source belonging to a broad class of sources, without needing to know its statistics. Furthermore, they utilize dictionary-based coding, where the codebook (i.e., dictionary) is built from source symbols while encoding and decoding, and therefore the codebook overhead is smaller than Huffman coding.

As the reviewer suggested, we compared the compression ratios of Huffman coding and universal source coding algorithms, i.e., LZW and bzip2 that deploys the Burrows--Wheeler transform, for the pruned AlexNet model and observed some gain when using LZW or bzip2. The compression ratios are 41.43, 41.82, and 42.46 when using Huffman coding, LZW, and bzip2, respectively, at the same top-1 accuracy of 57.02%. I added this result in our revised paper (see Appendex~A.3 and Table~2).

4. We discussed the benefit of randomized lattice quantization in lines 22-31. We will clarify it more here.

Randomized lattice quantization achieves universally good performance regardless of source statistics at any rates by dithering the source uniformly before quantization. The quantization loss becomes independent of source statistics after uniform random dithering, and it results in a universally good rate-distortion trade-off, which is close to the optimum Shannon's rate-distortion bound within a fixed gap, for any sources at any distortion levels. Hence, it is applicable to any neural network models to produce their near-optimal compressed models without optimization and/or calculation of their weight distribution.

The gain of randomized lattice quantization over lattice quantization can be found in Figure~2(a) in particular for $n>=2$ and large compression ratios. Moreover, observe that randomized lattice quantizers provide similarly good performance in both cases (a) and (b). Lattice quantization performs well only in case (b), where the quantization bins are optimized for given weight distribution.

We have clarified this point in our revised paper (see Appendix~A.1).

Thank you again for your valuable comments and suggestions. We look forward to further comments.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper15/Authors
comment: Dear Reviewer,

We thank you for your kind review and suggestion. We have revised our paper and uploaded it.

1. The experimental setup and more discussion for the results in Figure~2 are added in our revised paper (see Appendix~A.1). As the reviewer requested, we also updated Figure~2 to be more visible.

In Figure~2, we compare the trade-off between the accuracy and the compression ratio for lattice quantization and randomized lattice quantization with different values of vector dimension $n$  (see (1)). Here, uniform quantization corresponds to lattice quantization with dimension $n=1$. Given the vector dimension $n$, the weights from all layers of the pre-trained ResNet-32 model are vectorized as in (1) for vector quantization. Then, lattice quantization or randomized lattice quantization follows. In plain lattice quantization, no random dithering is added before quantization, i.e., we set $\mathbf{u}_i=\mathbf{0}$ for all $i$ in (2). We fine-tune the quantization codebook as explained in the paper. Finally, we use Huffman coding only in this experiment to get the compressed models.

The gain of randomized lattice quantization over lattice quantization can be found in Figure~2(a) in particular for $n>=2$ and large compression ratios. We note that randomized lattice quantizers provide similarly good performance in both cases (a) and (b). Lattice quantization performs well only in case (b), where the quantization bins are optimized for given weight distribution. We emphasize that randomized lattice quantization is applicable for any network models blindly, regardless of their weight distribution and with no optimization, while it is guaranteed to yield a good rate-distortion trade-off close to the optimum within a fixed gap~[12] (see lines 22-31).

2. We added the incremental improvement of compression ratio for the AlexNet model in the revised paper (see Appendix~A.3) to show the contribution of pruning, quantization, and source coding. We also updated Table~1.

Given a pre-trained AlexNet model, we prune its 90.4% weights and fine-tune remaining unpruned weights, as suggested in [22], which yields the compression ratio of 9.11 with the top-1 accuracy of 57.28%. The accuracy improves slightly after fine-tuning the pruned model. Using universal quantization and bzip2 that deploys the Burrows--Wheeler transform [17] on the pruned AlexNet model, we achieve the compression ratio of 42.46.

We note that quantization itself does not reduce the model size since the quantized weights are still 32-bit floating-point numbers although they are clustered into a small number of bins. The compression comes with source coding.

We also compare Huffman coding to the universal source coding algorithms, i.e., Lempel-Ziv-Welch (LZW) and bzip2, in Table~2. The compression ratios are 41.43, 41.82, and 42.46 when using Huffman coding, LZW, and bzip2, respectively, at the same top-1 accuracy of 57.02%. Both LZW and bzip2 provide better compression ratios than Huffman coding in our experiments.

3. Finally, we compared the accuracy before and after fine-tuning in Figure~3 in our revised paper (see Appendix A.2). The gain of vector quantization becomes more significant after fine-tuning, in particular, as the vector dimension $n$ increases, since there are a more number of shared quantized values trainable in vector quantized models, which we pointed out in lines 36-38.

Thank you again for your valuable comments and suggestions. We look forward to further comments.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper15/AnonReviewer3
title: Intuitions behind the proposed method seem unclear
review: This paper introduces a pipeline for network compression. Similar with deep compression [1], it also consists of three steps: pruning, quantization and source coding. The major difference is using randomized lattice quantization instead of the classical vector quantization, and using universal source coding (bzip2) instead of Huffman coding. My comments are:

a> Insights and intuitions behind introducing the universal quantization/source coding strategy seem unclear, both theoretically and experimentally. From Figure 2, one can tell the difference when the compression rate exceeds 20 and the accuracy drops under 90%, but none of the model shown in Table 1 is as that bad. Further, in comparison with deep compression, the proposed method is not consistently better; it shows higher compression rate but lower top-1 accuracy on ResNet-32.
b> Also, ablation study for the source coding strategy is necessary (i.e., why not Huffman coding). 


rating: 2: Marginally below acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper15/AnonReviewer2
title: Interesting idea but experiments can be improved
review: This paper proposes to use universal vector quantization and source coding for neural network compression. With the proposed method, the authors got good compression results on ResNet-32 (trained on CIFAR-10 dataset) and AlexNet (trained on ImageNet classification dataset).


Adding randomly sampled dithering values (following a uniform distribution) to the network weight values to enhance neural network quantization is interesting. Also, random uniform quantization is a good choice for real deployment of neural networks on hardware.    

However, some critical questions are not well studied.

1 The result difference between the strategies with/without adding dithering values to the weights. It seems that the results of Fig. 2 are provided to analyze this question, but they are not well clarified (e.g., details on 8 different settings). Also, Fig. 2 can be modified to make blue curves be visible.
    
2 To the final compression ratios, how about the contributions of pruning, proposed quantization and source coding. 

3 The accuracy improvement by using fine-tuning of quantized vectors, in comparison to naïve implementation (without fine-tuning).

rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct