NetScore: Towards Universal Metrics for Large-scale Performance Analysis of Deep Neural Networks for Practical On-Device Edge Usage
Abstract
Much of the focus in the design of deep neural networks had been on improvingaccuracy, leading to more powerful yet highly complex network architectures thatare difﬁcult to deploy in practical scenarios.
As a result, there has been a recentinterest in the design of quantitative metrics for evaluating deep neural networksthat accounts for more than just model accuracy as the sole indicator of networkperformance.
In this study, we continue the conversation towards universal metricsfor evaluating the performance of deep neural networks for practical on-device edgeusage by introducing NetScore , a new metric designed speciﬁcally to provide aquantitative assessment of the balance between accuracy, computational complexity,and network architecture complexity of a deep neural network.
In what is one ofthe largest comparative analysis between deep neural networks in literature, the
NetScore metric, the top-1 accuracy metric, and the popular information densitymetric were compared across a diverse set of 60 different deep convolutional neuralnetworks for image classiﬁcation on the ImageNet Large Scale Visual Recognition
Challenge (ILSVRC 2012) dataset.
The evaluation results across these three metricsfor this diverse set of networks are presented in this study to act as a referenceguide for practitioners in the ﬁeld.
Introduction
There has been a recent urge in both research and industrial interests in deep learning [4], with deepneural networks such as deep convolutional neural networks [6, 5] demonstrating state-of-the-artperformance across a wide variety of applications [19, 22, 11].
However, the practical industrialdeployment bottlenecks associated with the powerful yet highly complex deep neural networks inresearch literature has become even increasingly visible and as a result, the design of deep neuralnetworks that strike a strong balance between accuracy and complexity become a very hot area ofresearch focus [18, 14, 34, 33, 26, 28, 36].
One of the key challenges in designing practical deepneural networks lies in the difﬁculties with assessing how well a particular network architectureis striking that balance.
One of the most widely cited metrics is the information density metricproposed by [1], which attempts to measure the relative amount of accuracy given network size.
However, information density does not account for computational requirements for performingnetwork inference (e.g., MobileNet [14] has more parameters than SqueezeNet [18] but has lowercomputational requirements for network inference).
Therefore, the exploration and investigationtowards universal performance metrics that account for accuracy, architectural complexity, andcomputational complexity is highly desired as it has the potential to improve network model searchand design.
In this study, we introduce NetScore , a new metric designed speciﬁcally to provide aquantitative assessment of the balance between accuracy, computational complexity, and networkarchitecture complexity of a deep neural network.
NetScore: Design Principles
The proposed NetScore metric (denoted here as Ω ) for assessing the performance of a deep neuralnetwork N for practical usage can be deﬁned as:
[Equation]
40 where a ( N ) is the accuracy of the network, p ( N ) is the number of parameters in the network,m ( N ) is the number of multiply–accumulate (MAC) operations performed during network inference,and α , β , γ are coefﬁcients that control the inﬂuence of accuracy, architectural complexity, andcomputational complexity of the network on Ω .
Control coefﬁcients We set α = 2 to better emphasize the importance of model accuracy inassessing the overall performance of a network in practical usage, as networks that have unreasonablylow accuracy remain unusable in practical scenarios, regardless how small or fast the network is.
Furthermore, we set β = 0 .
5 and γ = 0 .
5 since, while architectural and computational complexity areboth very important factors to assessing the overall performance of a network in practical scenarios,the most important metric remains the model accuracy given that, as eluded to before, networkswith unreasonably low model accuracy are not useful in practical scenarios regardless of size andspeed.
Given these coefﬁcients, NetScore is in the units of squared percentage accuracy per rootparameter per root MAC operation, and represents the capacity of a network architecture to utilize itsfull learning and computing capacity.
Logarithmic scaling : A difﬁculty in comparing the overall performance of different deep neuralnetworks with each other is their great diversity in their model accuracy, architectural complexity, andcomputational complexity.
This makes the dynamic range of the performance metric quite large andunwieldy for practitioners to compare for model search and design purposes.
To account for this largedynamic range, we take inspiration from the ﬁeld of signal processing; in particular, the logarithmicscale commonly used to express the ratio between one value of a property to another.
Here, wetransform the ratio between the model accuracy property ( a ( N ) ) and the model architectural andcomputational complexity ( p ( N ) and m ( N ) ) into the logarithmic scale to reduce the dynamic rangeto within a more readily interpretable range.
Experimental Results and Discussion
To get a better sense regarding the overall performance of the huge wealth of deep convolutionalneural networks introduced in research literature in the context of practical usage, we perform a largescale comparative analysis across a diverse set of 60 different deep convolutional neural networksdesigned for image classiﬁcation using the following quantitative performance metrics: i) top-1accuracy, ii) information density, and iii) the proposed NetScore metric.
The dataset of choice forthe comparative analysis in this study is the ImageNet Large Scale Visual Recognition Challenge(ILSVRC 2012) dataset [23], which consists of 1000 different classes.
To the best of the author’sknowledge, this comparative analysis is one of the largest in research literature and the hope is thatthe results presented in this study can act as a reference guide for practitioners in the ﬁeld.
The set of deep convolutional neural networks being evaluated in this study are: AlexNet [19],
AmoebaNet-A (4, 50) [24], AmoebaNet-A (6, 190) [24], AmoebaNet-A (6, 204) [24], AmoebaNetB (3, 62) [24], AmoebaNet-B (6, 190) [24], AmoebaNet-C (4, 50) [24], AmoebaNet-C (6,228) [24], CondenseNet (G=C=4)
[16], CondenseNet (G=C=8)
[16], DenseNet-121 (k=32) [17],
DenseNet-169 (k=32) [17], DenseNet-161 (k=48) [17], DenseNet-201 (k=32) [17], DPN-131 [2],
GoogleNet [31], IGC-L100M2 [35], IGC-L16M16 [35], IGC-L100M2 [35], Inception-ResNetv2 [30],
Inceptionv2 [32], Inceptionv
[32], Inceptionv4 [30], MobileNetv1 (1.0-224) [14], MobileNetv1 (1.0192) [14], MobileNetv1 (1.0-160) [14], MobileNetv1 (1.0-128) [14], MobileNetv1 (0.75-224) [14],
MobileNetv2 [26], MobileNetv2 (1.4) [26], NASNet-A (4 @ 1056)
[38], NASNet-A (6 @ 4132)
NASNet-B (4 @ 1536)
[38], NiN [20], OverFeat [27], PNASNet-5 (4, 216) [21], PolyNet [37],
PreResNet-152 [13], PreResNet-200 [13], PyramidNet-101 (alpha=250) [9], PyramidNet-200(alpha=300) [9], PyramidNet-200 (alpha=450) [9], ResNet-152 [12], ResNet-50 [12], ResNet101 [12], ResNeXt-101, SENet [15], ShufﬂeNet (1.5) [36], ShufﬂeNet (x2) [36], SimpleNet [10],
SqueezeNet [18], SqueezeNetv1.1 [18], SqueezeNext (1.0-23v5) [7], SqueezeNext (2.0-23) [7],
SqueezeNext (2.0-23v5) [7], TinyDarkNet [25], VGG16 [29], Xception [3], ZynqNet [8].
In thisstudy, the units used for p ( N ) and m ( N ) are in M-Params (millions of parameters) and G-MACs(billions of MAC operations), respectively.
The top-1 accuracies across 60 different networks (shown in Fig. 1(left)) clearly illustrate the signiﬁcant progress made in network design for image classiﬁcation over the past six years, with thedifference between the network with the highest top-1 accuracy in this study (i.e., AmoebaNet-C(6, 228)) and that of AlexNet exceeding 25%.
The information densities across 60 different deepconvolutional neural networks for the ILSVRC 2012 dataset, shown in Fig.
1(middle), clearly illustrates that the deep convolutional neural networks that were speciﬁcally designed for efﬁciency(e.g., MobileNetv1, MobileNetv2, ShufﬂeNet, SqueezeNet, Tiny DarkNet, and SqueezeNext) havesigniﬁcantly higher information densities compared to networks that were designed purely with accuracy as a metric.
More speciﬁcally, the SqueezeNext (1.0-23v5), Tiny DarkNet, and the SqueezeNetfamily of networks had the highest information density by a wide margin compared to the other testeddeep convolutional neural networks, which can be attributed to their signiﬁcantly lower architectural complexity in terms of number of network parameters.
Another notable observation from theresults in Fig.
1(middle) is that the dynamic range of the information density metric is quite largeacross the diverse set of 60 deep convolutional neural networks evaluated in this study.
Finally, the
NetScore across 60 different deep convolutional neural networks for the ILSVRC 2012 dataset isshown in Fig.
1(right).
Similar to the trend observed in Fig.
1(middle), it can be clearly observed thatmany of the deep convolutional neural networks that were speciﬁcally designed for efﬁciency havesigniﬁcantly higher NetScores compared to networks that were designed purely with accuracy as ametric.
However, what is interesting to observe is that the NetScore ranking amongst these efﬁcientnetworks are quite different than that when using the information density metric.
In particular, the topranking deep convolutional neural networks with the highest NetScores are SqueezeNext (1.0-23v5),
CondenseNet (G=C=8), and MobileNetv2.
A number of examples illustrate the efﬁcacy of NetScore over information density for providing amore complete proﬁle of network efﬁciency and performance.
For example, CondenseNet(G=C=8)has slightly lower information density than ZynqNet, but has ∼ 2 × lower computational complexityand much higher accuracy.
The NetScore, in this case, is much higher for CondenseNet(G=
C=8)compared to ZynqNet (higher by >4 units).
In another example, MobileNetv1(0.75-224) has morethan 2 × parameters than SqueezeNet, and thus has much lower information density.
However,the computational complexity of SqueezeNet is > 26 × greater than MobileNetv1(0.75-224) andaccuracy much lower, and as such is reﬂected by a much higher NetScore for MobileNetv1(0.75-224)compared to SqueezeNet (higher by >14 units).
The proposed NetScore metric, which by no means is perfect, could potentially be useful for guidingpractitioners in model search and design and hopefully push the conversation towards better universalmetrics for evaluating deep neural networks for use in practical scenarios. NetScore can, for example,be used to narrow down a selection of network architecture candidates from a huge number of networkarchitectures available to evaluate deeper on target hardware for hardware-speciﬁc usage.