From: NIPS.cc/2018/Workshop/CDNNRIA/Paper8/Authors
comment: We thank the reviewer for finding NetScore to be potentially useful as a single metric to select best model at edge usage.

With regards to NetScore being a straightforward metric, we feel that is more of a pro than a con, because it makes the metric very transparent, hardware agnostic, and easy to understand.  Similar the last work such as information density, you can easily understand the tradeoffs between the different factors and what the units of the metric are.

With regards to arbitrary choice of exponents, they were specifically chosen to: 1) emphasize the importance of accuracy while still accounting for the importance of architecture and computational efficiency within NetScore, and 2) maintaining more understandable units (by maintaining powers of two).  NetScore is not designed to be used for scoreboard purposes, but should be used to give practitioners a quick estimate of network performance across a wide selection of models so that they can narrow down a set of models for further deep analysis on target hardware.  That is essentially the main goal for NetScore, in a similar vein as big-O for algorithmic complexity, and allows for a quick, hardware agnostic way to compare network architectures with each other so further model selection can be done on a smaller set of models.   This has now been described clearly in the revised paper.

Unlike information density and other metrics, there are no perfect metrics (or perfect choice of exponents) but the goal is to push the conversation forward for better ways to look at the wealth of models being made available everyday and identify a tractable subset of models that is suitable for particular edge usage for further analysis on target hardware.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper8/AnonReviewer1
title: Useful but straightforward metric, arbitrary choice of exponents
review: The paper proposes a single metric to jointly evaluate the accuracy, computational complexity and network architecture complexity of a deep neural network.

Pros:
-Might be useful as a single metric to select best model at edge usage


Cons:
-Very straightforward metric
-Arbitrary choice of exponents, another selection with slightly different parameters might lead to change in the scoreboard

rating: 2: Marginally below acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper8/Authors
comment: We thank the reviewer for finding NetScore to be a valuable effort and can give practitioners a quick estimate of a network's performance.  That is essentially the main goal for NetScore, in a similar vein as big-O for algorithmic complexity, and allows for a hardware agnostic way to compare network architectures with each other.  We have now added clarification on this in the revised paper. 

Given the huge number of deep neural network architectures out there, it is right now very difficult to narrow down just a handful of network architectures for more comprehensive analysis on the target device.  With NetScore, it allows for a quick way to compare a huge number of different network architectures to pick and choose a couple of good potential networks to analyze for hardware-specific usage.  As such, it can be of value for practical use for narrowing down choices as opposed to making the final decision on the most appropriate network for a particular piece of hardware.  We have revised the paper (see updated paper) to better emphasize this particular usage for NetScore as a quick way to assess a large number of networks to narrow down for further exploration for target hardware for practical scenarios.

In terms of the choice of the functional form, NetScore actually takes a similar approach as information density in that the NetScore represents the capacity of a specific network architecture to utilize their full learning and computing capacity.  As such, NetScore, like information density, preserves the original units (in this case, the unit of NetScore is the squared percentage accuracy per root parameter per root MAC operation in the logarithmic scale).  As such, it does indeed take scale into account in a similar light as information density.  Therefore, the most important thing to note is that since the units are defined, NetScore allows for a more objective comparison.  The explanation regarding this have be added to the revised paper.  

The choice of exponents were specifically chosen to: 1) emphasize the importance of accuracy while still accounting for the importance of architecture and computational efficiency within NetScore, and 2) maintaining more understandable units (by maintaining powers of two).  

With regards to issues about accuracy, for practical usage of NetScore will be based on consistent measurement of accuracy, as that is not a problem with the metric (or any other metrics such as information density) and essentially no metric can possibly be created to address inconsistencies in the user input for accuracy and it is therefore on the user to use it in a consistent way.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Score 2/5

The authors present a metric for evaluating the computational efficiency of deep neural networks. The NetScore metric is based on accuracy, memory footprint, and computational load (number of MAC operations). My biggest concern is the arbitrariness in the choice of the functional form of the metric (eq. 1). In particular, the choice of exponents. A different choice of exponents should lead to a different NetScore order for the different networks investigated. Another issue is accuracy; top-1 accuracy on imagenet is obtained using different methods, single crop, multi-crop, etc.. As mentioned in ref.[1] in the paper, different test method lead to different ordering of the information density metric, and I suspect that will also be the case for the netscore metric introduced here.

I believe the metric as defined has too many arbitrary knobs. A more valuable metric in my opinion would have non-arbitrary hardware-related knobs, for example: weight access energy/latency, energy/MAC, attainable parallelism, etc... For real-world applications, I suspect the NetScore metric will be of little help, as every compute architecture is different and it is unclear how this metric can be used to reflect the suitability of a network to a particular hardware architecture. Of course, a particular network can be obviously better than another (by being better in terms of memory, computation, and accuracy). The issue is with the non-obvious comparisons, and this is where unfortunately, we are still stuck with actually trying the different networks out on the target device to see which performs better and I do not believe that a metric like NetScore can change that. 

In summary, while I believe this is a valuable effort and can give practitioners a quick, rough, and arbitrary estimate of a network's performance, it is too vague and arbitrary for practical use.

confidence 2/3

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper8/Authors
comment: We thank the reviewer for appreciated the effort of NetScore for providing a metric for quantitatively assessing neural networks in the sense of accuracy, architectural complexity, and computational complexity.  We are happy to address the issues mentioned by the reviewer for the work to be accepted (please see updated paper, which addresses the comments made by the reviewer).  In particular:

1> The NetScore takes inspiration, as described in the paper, from the PSNR index in signal processing only in the sense that a logarithmic scale enables more readible comparison between network architectures.  NetScore actually takes a similar approach as information density in that the NetScore represents the capacity of a specific network architecture to utilize their full learning capacity (characterized by p(N)) and their full computing capacity (characterized by m(N)).  As such, NetScore, like information density, preserves the original units (in this case, the unit of NetScore is the squared percentage accuracy per root parameter per root MAC operation in the logarithmic scale). As such, it does indeed take scale into account in a similar light as information density.  The explanation regarding units have been added to the revised paper.  The choice of 2, 0.5, and 0.5 were specifically chosen to: 1) emphasize the importance of accuracy while still accounting for the importance of architecture and computational efficiency within NetScore, and 2) maintaining more understandable units (by maintaining powers of two).  

2> We thank the reviewer for clarifying this point, and we can indeed emphasize the superiority of NetScore compared to information density in a number of empirical results (see revised paper).  For example, CondenseNet(G=C=8) has slightly lower information density than ZynqNet (24.5 vs. 24.9) but CondenseNet(G=C=8) has 2x lower computational complexity and much higher accuracy.  The NetScore, in this case, is much higher for CondenseNet(G=C=8) compard to ZynqNet (75 vs. 70.7).    In another example, MobileNetv1 (0.75-224) has more than 2x parameters than SqueezeNet, and based on information density it would appear that SqueezeNet is a more efficient network architecture than MobileNetv1 (0.75-224).  However, the computational cost of SqueezeNet is more than 26x greater than MobileNetv1 (0.75-224) (which is not reflected by information density) and SqueezeNet also has much lower accuracy compared to MobileNetv1 (0.75-224), and as such the balance between accuracy, architectural complexity, and computational complexity is better for MobileNetv1 (0.75-224) compared to SqueezeNet, and is thus reflected in MobileNet having a higher NetScore than SqueezeNet.    We have now added these empirical results to the revised paper.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Overall rating: 2/5

This paper presents NetScore, a "metric" designed for quantitatively assessing neural networks in the sense of accuracy and computational complexity. I appreciate the effort, but some issues need to be addressed for accepting this work.

1> The formula in Eq (1) seems plausible, but why it is in its form is not carefully explained. It looks similar to the PSNR index in signal processing, but a(N), p(N) and m(N) are not in the same range and their scales are not considered. Also, the numerical setting for alpha, beta and gamma is not explained either. Why 2, 0.5 and 0.5 are specifically chosen instead of the other?
2> The authors claim that their NetScore considers the computation complexity of deep networks while the information density metric does not, but didn't emphasize the superiority of their method in experiments. Some empirical studies showing their method works in a more reasonable way (e.g., networks with lower complexity and higher accuracy indeed shows higher NetScore but maybe lower information density) should be very helpful. 

Confidence score: 3/3

From: ~Lixin_Fan1
comment: Authors please send author information to workshop chairs (lixin.fan@nokia.com), so that you submission can be assigned to reviewers without conflict of interests.
title: Authors please send author information to workshop chairs (lixin.fan@nokia.com).

From: ~Lixin_Fan1
comment: Authors please send author information to workshop chairs (lixin.fan@nokia.com), so that you submission can be assigned to reviewers without conflict of interests.
title: Authors please send author information to workshop chairs (lixin.fan@nokia.com).

From: ~Lixin_Fan1
comment: Authors please send author information to workshop chairs (lixin.fan@nokia.com), so that you submission can be assigned to reviewers without conflict of interests.
title: Authors please send author information to workshop chairs (lixin.fan@nokia.com).