From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper6/Authors
comment: We are thankful to the reviewer and appreciate the constructive feedback. We believe that this feedback helps us to further strengthen our manuscript and our research.

We agree with the assessment of the reviewer regarding our method and our evaluation.

It is our believe that neural networks gained considerable attention by communities far beyond the machine learning and computer vision communities. Amongst these communities, there is the traditional compression community which is for example represented in MPEG. MPEG, which is known for popular video and audio compression standards like HEVC (also referred to as H.265), AVC (also referred to as H.264) and MP3, started to explore the compression of other data types recently. One of the considered data types is neural networks. Precisely, a sub-group was formed to push forward a potential standard for the representation and compression of neural networks (topology and weights). The requirements for this potential future standard include the compression of existing neural networks model and partly updates of compressed neural networks. Both requirements are fulfilled by our proposed method. This standard would be able to truly enable interoperability of systems which rely on neural networks as MPEG has accomplished for video and audio with previous standards. An overview on the activity in this community (use cases, requirements, Call for Evidence) can be found on the MPEG website: https://mpeg.chiariglione.org/standards/exploration/digital-representation-neural-networks.

We aim at transparent compression of neural networks (i.e. the goal is to compress an existing model without modifications to its architecture) because we believe that machine learning researchers and engineers often put great know-how in the design of neural networks. Then, it is desired to maintain this architecture for the product roll-out and deployment. Also, there are cases in which the neural network design and the delivery of the trained network models need to work independently. 

Still, we strongly believe that the other approach (design of light-weight architectures and pruning) is very valuable as well. Therefore, we demonstrated in our manuscript that our pipeline can be applied successfully on top of such light-weight architectures to even further reduce the storage requirement.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper6/Authors
comment: We would like to thank the reviewer for the competent and critical review. We acknowledge that a substantial amount of time was spend looking over our paper.

We agree with the reviewer that our transparent compression of neural networks (i.e. the goal is to compress an existing model without modifications to this model) puts a new perspective on neural network compression in addition to existing perspectives like pruning. We also agree that it was demonstrated in our manuscript that our coding pipeline can be applied on top of light-weight neural networks. We will follow the reviewer’s suggestion to add more discussion on the merits of combining these two approaches.

Furthermore, we agree that our method is of interest for the upcoming neural network representation standard which is currently under exploration within MPEG: https://mpeg.chiariglione.org/standards/exploration/digital-representation-neural-networks. Here, it is also of interest that the neural network layers can be processed independently, which enables not only efficient parallel processing but also allows to update only one layer in a network without retransmission of the other parts of the network. 

We agree that the complexity of our method is reasonable. The encoding time of less than a minute can neglected compared to the typical training durations of neural networks which can last for days or weeks. For real-world applications, the decoding time matters more. For decoding a neural network, we measured 6.7 seconds with our unoptimized Python code. We believe that it is safe to assume that this time can be reduced considerably by performance-oriented implementations. Hence, on-the-fly decoding is possible with our approach.

Due to the 3-page limit we were not able to discuss all of our experimental results in depth. We hope to discuss the remaining experiments with other experts at the workshop.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating :  4 （1-5）

Review:
  This paper proposes an effective neural network compression method with DCT transform and quantization by considering the scale factor to achieve great compression ratio while maintaining good performance over the ImageNet dataset. This work comes up with another perspective of compressing deep models without fine-tuning, which has been well validated not only on large deep model such as AlexNet and VGG, but also on light-weight deep model SqueezeNet, which has demonstrated great technical potentials for further combining with state-of-the-art light-weight deep model compression approaches. Moreover, the compression (encoding) approach consists of transform and clustering with great encoding efficiency, which is expected to fulfill the requirements towards the future deep model communication and transmission standard. Overall, the framework towards light weight model encoding pipeline with uniform quantization and clustering has yielded great compression performance, which can be further combined with existing deep model compression approaches towards light-weight models.
Pros:
 + A new perspective of better model encoding algorithm by leveraging DCT, quantization and clustering with satisfying performance without fine-tuning.
 + The codec pipeline considering the effects of scaling as well as inter-operability, has shown good potentials for the deep neural network representation standard. 
 + The proposed pipeline is focused more on the weight distribution, such as transforming, quantization and clustering, which may be combined with other state-of-the-art deep model compression algorithms to generate not only light-weight but also well encoded deep models.
 + The pipeline with acceptable computation complexity as well as the compression performance without fine-tuning is evaluated, even with light-weight model (SqueezeNet), which is supposed to support deep model communication paradigm.
Cons:
  - In the introduction, the authors have declared that their proposed pipeline is contrast to the Deep Compression [S. Han ICLR 2016]. However, the pruning and retraining based compression are still powerful approaches to generate light-weight models, and it is suggested to add more discussions on the combination of merits.
  - It is desirable to add more comparison experiments.

 Confidence: 3 (1-3)

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating: 4 of 5

This paper deals with “neural network compression”; but “compression” here is used in the same sense as in “video compression” for video – it does not mean making a more compact neural network by e.g. reducing the number of nodes. It means generating a compressed file that can be transmitted to a mobile device without consuming so much bandwidth (similarly to an mp4 file versus the raw video); but then, the encoded neural network has to be fully decoded/decompressed on the mobile device before it can be used again for inference.

The paper is technically sound, and reasonably well-written. The authors make clear what they mean (and what they don’t mean) by “neural network compression”, and they examine the benefits of applying this kind of compression not only on network architectures that are not optimized for size (e.g. GoogLeNet), but also on an optimized architecture (SqueezeNet).

Overall, this is a sensible idea and solid work; but, compared to methods that aim to transform the network to a more compact one (rather than a coded representation of the original network), it is probably of less interest to the community. 

Confidence: 2 of 3

From: ~Lixin_Fan1
comment: Authors please send author information to workshop chairs (lixin.fan@nokia.com), so that you submission can be assigned to reviewers without conflict of interests.
title: Authors please send author information to workshop chairs (lixin.fan@nokia.com).