From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Ok, thanks for the clarification.

With the changes you will apply for the final version, I will change my rating from 'borderline' to 'weak accept'

final rating: 'weak accept'.

Reviewer confidence: 3 / 5

From: ~Simon_Wiedemann1
comment: We also thank very much the reviewer nr. 2 for her/his thoughtful comments and criticism, which will further help us to improve our final version of the abstract.

1) Comment: The related works about sparse ...
3) Comment: CSR is not introduced very well...

Reply: Because of space constraints, we did not include an introduction of the sparse representations and assumed that the reader has the required background knowledge. We believed that this was ok, since sparse representations such as CSR are fairly common and popular in many fields of science where expensive dot product operations are involved in the calculations. However we thank the reviewer for her/his comment, and we will include proper citations in our final version in order to point to the respective background knowledge.

2) Comment: Table 1 is very hard to understand...

Reply: Table 1 expresses only one accuracy result since all representations compress the weight matrices in a lossless manner and therefore, no accuracy loss results in converting them into the different formats. Hence, table 1 reports the gains attained after we performed the following steps: 1- get all weight matrices of the network (including convolutional layers) and quantized their range of values uniformly down to 7 bits. 2- then, convert each of them into the different representations (CSR, CER, CSER), 3- and benchmark the respective matrix-vector multiplication under the 4 mentioned measures and finally, 5- aggregated the results over the layers. 
Hence, each entry in  the table reports the gains attained relative to its original form (thus, it's dense representation) in the following order: storage gains, time gains or speedups, energy gain and gain in total number of operations. 

4) Comment: The describe of CER and CSER is also not very clear... 

Reply: Again, we ask the reviewer for understanding that due to space constrains, we did not include the specific workings of the CER and CSER representation in the abstract, and provided instead the motivation/ideas/principles that apply to them. We believe that it requires a more careful and thoughtful explanation to fully grasp the details of the CER and CSER representations, as well as their associated dot product algorithms, which we provided these in our main manuscript [1].

5) Comment: There are few grammatical mistakes...

Reply: We will correct all of them for our final version of the abstract.

[1] Compact and computationally efficient representation of deep neural networks. CoRR, abs/1805.10692, 2018.
title: Reply to reviewer

From: ~Simon_Wiedemann1
comment: We thank very much the reviewer for her/his thoughtful comments and criticism, which will help us to improve our final version of the abstract.

In the following we will we provide replies and comments to all issues raised by the reviewer.

1) Comment: There are a couple of...

Reply: We thank the reviewer to point them out. We will correct them for the final version of the abstract.

2) Comment: E.g. the term 'sparse matrix ...

Reply: Thanks for the constructive criticism. We will change this for the final version of our abstract.

3) Comment: In the abstract, the phrase 'when ...

Reply: We defined the set of high compressible matrices as the set of matrices whose entropy values are low, relative to the maximum likelihood estimate of the probability mass distribution of their weight elements. We also specified this in the next lines 10-12 and in the lines 34-35 and 55-63. However, the phrase 'set of highly compressible matrices' may be too abstract, therefore we will rephrase it more concretely in our new version of the abstract.

4) Comment: In section 1, regarding phrase 'the cardinality ...

Reply: If one considers a uniform prior over a finite set, say W, one can trivially assign log2(|W|) bits to each element. Hence, there is a direct relation between the bit-size of each element and the cardinality of the entire set. We thought that this was clear from the context, but it appears to not be so. We will accordingly describe it more specifically.

5) Comment: In section 1, regarding your claim that mixed-precision ...

Reply: Common hardware such as NVIDIA GPUs support only very particular types of mixed-precission arithmetic (such as reported in [1]). These methods mostly do not generalize to other quantization schemes. For instance, in our provided example (figure 1) we were able to quantize the weight values down to 7 bits only because we did not quantize the activation values. Quantizing the activation values down to 7 bits would result in a considerable harm in the prediction accuracy of the network. Hence, even in this scenario, we cannot perform efficient dot product operations since no common hardware supports multiplications on mixed-numerical representations between 7-bit weight elements and single precision floating point activation values. In addition, we would like to remind the reviewer that one can achieve even higher compression ratios if one applies more general quantization schemes [2], where the final representations can be exploited even less trivially. In contrast, our proposed CER and CSER representations are able to accommodate to all these scenarios.

6) Comment: Table 1 which summarizes the key results of your paper ...

Reply: To clarify, with 'performing a matrix-vector multiplication of different state-of-the-art neural network' we meant that we benchmarked the matrix-vector multiplication of all weight matrices of the networks. Therefore, yes, we included all layers in our calculations, as well convolutional layers. Also, the gains are reported relative to the original network, hence, not relative to the 7-bit dense representation of it. 

7) Comment: In section 2, the claim that CER and CSER data structurees can ...

Reply: Sparse representations are most optimal if and only if the probability mass distribution of the weight elements is similar to a spike-and-slap distribution (that is, a spike at 0 and the rest uniformly distributed). On the other hand, dense representations are most optimal when the elements are uniformly distributed. For any other statistic, the dense and sparse representations are suboptimal. In contrast, the CER and CSER data structures become more optimal as the entropy of the element distribution is being minimized, regardless of the sparsity level of the matrix. Consequently, the CER and CSER representations are able to exploit a wider set of matrices since the set of distributions with the property of having low entropy is larger than the set of spike-and-slab distributions.

8) Comment: In section 3, regarding the comparison of Alexnet ...

Reply: As stated in the abstract (lines 96-97), we benchmarked the AlexNet architecture provided in [3], which is the final compressed model after applying the deep compression scheme [4] on it. In our implementation, the time required for loading the activation values was considerably higher than the time required for the rest of the operations. When this occurs, the CSR and CER and CSER representations converge to the same gains, since both require to load as many input elements.

[1] https://devblogs.nvidia.com/mixed-precision-programming-cuda-8/
[2] Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Universal deep neural network compression. CoRR, abs/1802.02271, 2018.
[3] https://github.com/songhan/Deep-Compression-AlexNet
[4] Song Han, Huizi Mao, and William J. Dally.  Deep compression:  Compressing deep neural network with pruning, trained quantization and huffman coding. arXiv:1510.00149, 2015.
title: Reply to reviewer

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: This paper proposes the idea about new efficient matrices representations and its application on the compression of deep neural networks. This paper also shows that their superiority of the high compression ratios and the low computational complexity.

But there are few problems:

(1)The related works about sparse representations of matrices do not be introduced very well. I think this is one of the most important parts of your related works. 

(2)Table 1 is very hard to understand. And I suppose that this table might be the most important expression of your results. For example, I am confused with that there are four types of compression strategies to each network, but only one accuracy result is mentioned. 

(3)CSR is not introduced very well. And corresponding references are also not mentioned. I suppose that many readers would be puzzled about the CSR.

(4)The describe of CER and CSER is also not very clear. What are the differences about their assumptions? And how to specifically use them in the representation of a neural network?

(5)There are few grammatical mistakes. For example, in the second paragraph of the abstract, ‘lossless’ is improperly used, because it is one adjective.

Score: 3/5 (Marginally above acceptance threshold)
Confidence: 2/3

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: summary: the authors propose two new representations (compressed shared weight row, compressed entropy row ) for efficient compression of quantized weight matrices with low entropy.
For these representations, efficient dot product algorithms can be designed by taking into account the distributive law of multiplications.

There are a couple of phrases with wrong grammatic which should be corrected:
- page 1: "when we lossless ..."
- page 1: "Deep neural networks nowadays became ..."
- page 1: "computational resource which difficult or even prohbiit ..."
- page 2: "we can not only save a ..."

In the literature review, the relevant SoA papers for model compressions are cited.

There are several unclear formulations / missing information and questionable claims in the paper, which make it difficult to read and to assess the originality / novelty of the paper.

E.g. the term 'sparse matrix data structures' is not optimal. Better terms might be 'sparse matrix represenation' / 'sparse matrix storage formats' or 'sparse matrix compression format'.

In the abstract, the phrase 'when we consider the entire set of highly compressible matrices' is unclear -> what defines / characterizes this set ?

In section 1, regarding phrase 'the cardinality of the weight elements is in this case smaller than the original size (7 bits as opposite to 32)': It seems for me that you mix
up the size (in bit) of the quantized weight (7 bit) with its cardinality. 

In section 1, regarding your claim that mixed-precision dot product with low-bit (e.g. 8 bit) types is not supported on most conventional hardware: I would strongly disagree with that.
Deep learning inference is usually done on the GPU, and all NVIDIA GPUs since Pascal architecture have a dedicated instruction DP4A for mixed-precision dot product with 8-bit weights
(https://devblogs.nvidia.com/mixed-precision-programming-cuda-8/).

Table 1 which summarizes the key results of your paper is difficult to understand, and I suppose has wrong wording. First of all, the phrase 'performing a matrix-vector multiplcation of different state-of-the-art neural network' is wrong I suppose. From the numbers for the column 'dense', it seems that that the storage and number of operations is meant is for the _whole_ network and not for a part of it (e.g., a fully connected layer). Furthermore, it is unclear whether the reported gain factors are with respect to the original (not quanitzed) network or with respect  to the 7-bit quantized network.

In section 2, the claim that CER and CSER data structurees can efficiently exploit a wider set of matrices is questionble. Furthermore, it is unclear for me which class of matrices have element statistics which 'fail to be efficiently executed by the dense and sparse representation'.

In section 3, regarding the comparison of Alexnet with CSER representation and CSR representation, it is unclear from which paper the given performance numbers for CSR format are taken.
Furthermore, I would question your result for CER / CSER representation for Alexnet. This is, because it is strange that you report significantly higher compression ratio and energy
saving when you compare with CSR representation, but the reported speedup factor of CER/CSER is identical to the one for CSR. That does not fit together IMO.

Overall, I would give the paper a 'borderline' rating.

Reviewer confidence: 3 / 5