From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper38/Authors
comment: Thanks for your constructive comments. Before we start, we would like to mention that PocketFlow is an open-source framework (sorry that we failed to emphasize this in the submission) and will be publically available in the next few weeks. PocketFlow is flexible to use self-defined models, data sets, and model compression methods, and we provide detailed documentation as guidelines for developers.

1. The supported compression methods are limited.
As a flexible framework, new model compression methods can be easily implemented in PocketFlow. Developers only need to use the data input pipeline and network definition (either standard or self-defined) provided by PocketFlow to build up the training process of model compression methods. We are glad to include network slimming [1] as another channel pruning based algorithm ASAP and presents its detailed performance at the workshop.

2. The system should be flexible to use user-specified data sets for evaluation.
PocketFlow supports users to define their own data input pipeline for data sets other than CIFAR-10 and ILSVRC-12. Developers only need to implement two functions, *build_dataset_train()* for the training subset (or training subset + validation subset) and *build_dataset_eval()* for the evaluation subset.

3. Use network slimming for comparison on CIFAR-10.
We would like to mention that the baseline model we used on CIFAR-10 is ResNet-56, which is different from those used in network slimming [1], so that the results may not be directly comparable. We will implement network slimming (as mentioned in Q1) in PocketFlow and present detailed comparison results at the workshop.

[1] Liu et al., Learning Efficient Convolutional Networks through Network Slimming, ICCV 2017.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper38/AnonReviewer5
title: A practical system development with applications in industry.
review: This paper presents a practical system that supports fast and easy deployment of DNNs for mobile applications.

Pros:
.  Well motivated system design with solid implementation
. Automatic search for the optimal hyper-parameters of DNNs compression methods
. Fast and easy deployment of DNNs supported

Cons:
. The supported compression methods are limited, with many state of the art methods excluded e.g.Net-Slimming.

Detailed comments:
1. Empirical evaluation are based on CIFAR-10 [10] and ILSVRC-12 [13] data sets only. While these datasets are widely used, the optimal hyper-parameters found as such might not be the optimal ones for real use cases. Ideally, the system should be flexible to use users specified datasets for evaluation. 

2. The reported performance of the compressed models seemed not comparable with the state of the art e.g. for CIFAR-10. Would you use e.g. Net-slimming for comparison?



rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper38/Authors
comment: Thanks for your constructive comments.

1. We develop PocketFlow as an easy-to-use framework for model compression, and will be released to the open-source community shortly, so that more people can contribute and improve this framework (sorry that we failed to emphasize this in the submission). The main idea is not to propose new model compression methods. Instead, we want to simplify the model compression workflow for more users and developers.
2. Yes, users still need to choose which model compression method to be used. Meanwhile, PocketFlow will give suggestions based on user's requirements. For instance, if you want a dramatically smaller network and do not care about run-time speed, then you should use "WeightSparseLearner" or "NonUniformQuantLearner". Or, if you want a  network that also runs faster, then you should use one of the other three learners.
3. Such hyper-parameters include learning rate schedule, pruning ratios of each layer (or # of quantization bits of each layer), etc.
4. For the moment, these three hyper-parameter optimizers cannot be used together.
5. In Table 2, we compare the classification accuracy, model size, and run-time speed of original and quantized networks, based on the same network architecture.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating: 2/5
Confidence: 2/3

This paper proposes the systematic way to find the optimal hyperparameters for a given specific model compression method. Candidates of model compression methods are existing parameter pruning, parameter quantization methods. Also, when the model compression method is specified, it optimizes the hyperparameters of the compression method by using existing optimizer. 

Comments:
- This paper merely implements the existing model compression method and optimizer. 
- If I understand the section 2.1 correctly, users should choose one of the model compression method (not automatically suggested by the system). Then, users still suffer from the difficulty of choice of method.
- In section 2.2, it says "there are several hyper-parameters that may have a large impact on the final compressed model’s performance", but readers cannot know which hyperparameters are important for model compression. 
- In section 2.2, it suggests three hyperparameter optimizers  (GP, TPE, DDPG), but it is not clear whether all three optimizers are combined or a single optimizer is used. 
- In section 3, the paper shows performance comparison for the compressed model and other networks. However, it seems more meaningful to compare the results of compression of the same original network with those presented in the paper (using hyperparameter optimizer) and the results presented in the original model compression paper (e.g. [17] in the paper), rather than the completely unrelated networks such as MobileNet, Inception-V1, etc.