From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper11/AnonReviewer3
title: Review from comment of 21st of Oct
review: My comment from 21st of Oct was intended as a review. Back then, the review functionality was either not available, or I somehow missed it. Instead of repeating the whole text, I will just update the scores and refer to the original comment for the detailed review.
rating: 3: Marginally above acceptance threshold
confidence: 1: The reviewer's evaluation is an educated guess

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper11/Authors
comment: Thanks for the review. Here are some clarifications regarding the points noted by the reviewer.

- While lossless compression of the weights of the neural network can very well be done using standard compression techniques like Huffman codes, those compression techniques are optimal only when the data is sequential in nature, e.g. a sequence of numbers/words. But if there is a structure associated to the data that is not sequential (like network structures), like that of the arrangement of the weights in a neural network, it is not obvious how to use Huffman codes efficiently so as to achieve an optimal compression rate. It is even more non-trivial to use such lossless compressed structures for inference purpose without fully decompressing them. Basu and Varshney [1] propose a detailed optimal lossless compression algorithm that compresses a structure such as a neural network optimally. Our main contribution in this paper is to build on that paper and to provide a novel inference algorithm that can directly use such a compressed structure and provide inference without full compression. Further note that this algorithm works on any form of quantized network which adds to its benefits.
 
- On computational efficiency, please see responses to other reviewers.  Due to the similarity to arithmetic decoding, there are well-known computationally efficient implementations, e.g. from the H.264/AVC and HEVC standards.
 
- We thank the reviewer for the wonderful suggestion to write a further long and detailed journal paper that expands on this 3-page extended abstract.  We will plan to do so, with perhaps the IEEE Transactions on Information Theory as the target journal.  Notwithstanding, we believe it is important to also present this work in a setting that this workshop would provide: the community involved in the practice of neural network compression.  This would allow us to learn what application scenarios that the reviewer and others envision to demonstrate the practical utility of this approach (which we have demonstrated is optimal in the sense of information-theoretic bounds and succinctness, as well as through basic experiments).
 
- If the reviewer could provide more specific guidance on how to improve the English prose, we would be most appreciative.  We would take that into account when preparing the final version.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper11/Authors
comment: Thanks very much for the review and for leaning towards acceptance. Please see the following clarifications regarding issues pointed out in the comment.
 
- Regarding a high-level explanation into why the technique works, please note that ref [1] introduced a compressed representation that is created by a “sorting” operation that takes advantage of permutation-invariance of the node operation of a neural network, followed by arithmetic coding.  The basic insight for why it is possible to further achieve succinctness in the present paper is that the way signals downward traverse a feedforward NN during inference is well-matched to the downward traversal of the arithmetic decoding algorithm.  As such, the dynamic memory needed for inference is negligible. We will plan to add this high-level view in a final version of the workshop paper.
 
- The reviewer correctly notes that the present paper builds upon ref [1], which introduced the novel representation and also showed that it has a rate lower bound that is not just the naïve entropy bound but in fact the much lower entropy bound computed upon recognizing the permutation-invariance.  This previous work did not deal with inference at all.  The central contribution of the present paper is in developing a novel joint decompression-inference algorithm that further achieves succinctness by requiring only negligible dynamic memory.  We will revise the writing in the final version of the workshop paper to reduce overlap with [1].
 
- Notice the static memory requirement closely approaches not just the naïve entropy bound but in fact the much lower entropy bound computed upon recognizing the permutation-invariance.  This is demonstrated empirically in Table 1.  Given the static memory is very close to entropy and the dynamic memory is negligible, the total memory requirement is as essentially as small as can be for lossless representation.
 
- Regarding computational efficiency (e.g. flops/time), the computational efficiency of our algorithm depends directly on the computational efficiency of arithmetic coding. It is known that arithmetic codes are very efficient, e.g. high-throughput implementations of arithmetic coding/decoding have been developed for video, e.g. as part of the H.264/AVC and HEVC standards (Sze and Budagavi, 2012; Sze and Marpe, 2014).  We have revised the introductory section to convey this point.
 
- The technique, as currently developed is indeed for feedforward neural networks rather than convolutional neural networks.  Extending to CNNs will require developing an appropriate “sorting” operation for the encoder to take advantage of structural invariances.  Once that is done, however, developing a joint decoding-inference algorithm to achieve succinctness should follow in a much similar way as proposed in the present paper.  We will revise the conclusion of the final version of the workshop paper to provide this information.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper11/Authors
comment: Thanks for the review and for finding the ideas interesting. Here are some clarifications regarding the concerns raised in the comment.
 
- As far as we know, our proposed inference algorithm without completely decompressing a losslessly compressed deep neural network is indeed novel and has never been done before.  Achieving this formal (but practically relevant) notion of succinctness is indeed the key contribution being presented.  We have emphasized this point in a revised introductory section.
 
- The computational efficiency of our algorithm depends directly on the computational efficiency of arithmetic coding. It is known that arithmetic codes are very efficient, e.g. high-throughput implementations of arithmetic coding/decoding have been developed for video, e.g. as part of the H.264/AVC and HEVC standards (Sze and Budagavi, 2012; Sze and Marpe, 2014).  We have revised the introductory section to convey this point.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper11/AnonReviewer2
title: Need to improve writing
review: This paper has proposed a succinct source coding of deep neural networks that direcctly uses lossless compression by using partiall-labeled bipartite graphs which encode the connections of nerons of different layers. 

pros: (1) novel ideas. 
cons:
 (1) I find the writing of the paper is difficult to understand. I can only understand that maybe the authors are trying to encode connections of the neural network in lossless compression, but I am not sure enough. All details are difficult to read. The English also needs to be improved. Finally, I don't think it is a good idea to show several theorems in a workshop paper with only 3 pages. I encourage the authors put the theorems and detailed proofs into appendix, or consider to submit to a journal or conference without a strigent page limit.
(2) Because I don't fully understand the paper, I can only say that maybe this method require much computation to decode the connections of neurons in different layers, which would make the inference slower. In summary, maybe it is not effective to encode connections. Maybe a better way is to encode weights, just as a lot of methods did. The author needs to validate their method in application senarios. 

rating: 2: Marginally below acceptance threshold
confidence: 1: The reviewer's evaluation is an educated guess

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating: 3 / 5

Review:
The author proposed an inference algorithm for a (lossless-)compressed representation based on the permutation-invariant characteristics of a feed-forward neural network. The idea is interesting that one does not have to decode the compressed representation to conduct inference, which is needed for devices that have stringent storage. However, there is no free lunch, the algorithm essentially trades computation for space, which leads to concerns about the overhead of the algorithm and it is not discussed in the paper. Additionally, I had a hard time following the paper, which might due to my unfamiliarity of the sub-field. Overall, I like the idea of doing inference without fully-decompress the network but I'm not certain if this itself is novel. If it is, the author could point it out claiming this is the first attempt and make the contribution clear; otherwise, the author should then compare to work with related ideas.

Pros
- The idea seems interesting to me.

Cons
- Paper is a little bit hard to follow without referring to reference (1). I think the phrasing could be enhanced.
- The experiment does memory overhead analysis while no runtime overhead analysis, which is important since the method is trading off computation for storage.

Confidence: 1 / 3

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: The authors propose an algorithm for performing inference in a neural network that has been compressed in a lossless fashion. The advantage of the algorithm is that inference can be performed without requiring full decompression of the network. The main idea of the paper is based on exploiting permutation-invariance of neural network nodes and heavily builds on Ref. 1 from the paper. The method is evaluated on a small fully connected network on MNIST. An appealing conceptual advantage of the method is that it can be used in conjunction with (i.e. on top of) other (lossy) network compression algorithms. 

The problem addressed by the paper is very timely and important, since widely used lossless compression schemes for neural networks (e.g. Huffman-coding) require full decompression during forward-pass operation - thus not leading to memory-savings during operation. Unfortunately I have a hard time following the paper, which might be partly due to being unfamiliar with the very closely related literature. The paper builds heavily on Reference (1), where parts of the abstract have even been literally copied (which does not necessarily ensure integrity of double-blind review). While the method is evaluated on MNIST, the significance of the results is somewhat hard to interpret, since the paper does not report improvements in terms of absolute memory requirements, FLOPS, wall-clock-time or similar. To me personally, this is a very border-line case where I can neither strongly support acceptance or rejection. At the main conference I would thus vote for rejection, but for a Workshop as a venue of presenting cutting-edge, on-going work, I am slightly in favor of acceptance.

Rating: 5 of 10  (barely above the threshold for acceptance)
Confidence: 1 of 5  (it’s well possible that I did not understand important parts of the paper, not very familiar with related literature)


Major issues:
1) I have a hard time understanding the high-level view of how the method operates. This is partially because it builds heavily on Ref 1. But even after reading Ref. 1 I would like to encourage the authors to polish the high-level story of the paper and perhaps add some intuitive figures/explanations. Perhaps other readers of the deep learning community with a similar background in network compression feel similar.

2) The work seems a bit incremental, even though Ref. 1 does not report how to do inference, which is of course very important. Perhaps one problem is that Ref 1 and this paper are very similar (some parts are in fact literal copies). This is bad practice as it somewhat breaks double-blind review, but a paper (even a workshop paper) should never be copy&paste. 

Minor issues:
3) The experimental section is quite weak, even for a workshop paper. It would be nice to see additional experiments on different datasets, or comparisons against relevant methods, or even the compression of an already compressed network using some lossy compression algorithm.
3a) The results provided are a bit hard to interpret. Can you provide some numbers on actual memory savings (not just queue-length) or compression rates. Perhaps also report on the timing/FLOPS, i.e. is the method computationally more or less expensive to run and by how much?

4) If I understand correctly, the method cannot be straightforwardly applied to convolutional neural networks. If so, please mention this a bit more explicit in the paper.