HW-NAS-B ENCH : H ARD W ARE - AWARE N EURAL A RCHITECTURE S EARCH B ENCHMARK
Anonymous authors
Paper under double-blind review
A BSTRACT
HardWare-aware Neural Architecture Search (HW-NAS) has recently gained tremendous attention for automating the design of DNNs to be deployed into more resource-constrained daily life devices.
Despite their promising performance, developing optimal HW-NAS solutions can be prohibitively challenging as it requires cross-disciplinary knowledge in the algorithm, micro-architecture, and device-speciﬁc compilation.
First, to construct the hardware cost to be incorporated into the NAS process, existing works mostly adopt either pre-collected cost look-up tables or device-speciﬁc hardware cost models.
The former can be time-consuming due to the needed learning about the device’s compilation method and how to set up the measurement pipeline, while the latter is often a barrier for non-hardware experts like NAS researchers.
Both of them limit the development of HW-NAS innovations and impose a barrier-to-entry to non-hardware experts.
Second, similar to generic NAS, it can be notoriously difﬁcult to benchmark HW-NAS algorithms due to the required signiﬁcant computational resources and the differences in their adopted search space, hyperparameters, and hardware devices.
To this end, we develop HW-NAS-Bench, the ﬁrst public dataset for HWNAS research which aims to democratize HW-NAS research to non-hardware experts and make HW-NAS research more reproducible and accessible.
To design
HW-NAS-Bench, we carefully collected the measured/estimated hardware performance (e.g., energy cost and latency) of all the networks in the search space of both NAS-Bench-201 and FBNet, considering six hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC).
Furthermore, we provide a comprehensive analysis of the collected measurements in HW-NASBench to provide insights for HW-NAS research.
Finally, we demonstrate
exemplary user cases where HW-NAS-Bench (1) allows non-hardware experts to perform HW-NAS by simply querying our pre-measured dataset and (2) verify that dedicated device-speciﬁc HW-NAS can indeed often provide optimal accuracycost trade-offs.
All the codes and data will be released publicly upon acceptance.
I NTRODUCTION
The recent performance breakthroughs of deep neural networks (DNNs) have attracted an explosion of research in designing efﬁcient DNNs, aiming to bring powerful yet power-hungry DNNs into more resource-constrained daily life devices for enabling various DNN-powered intelligent functions.
Among them, HardWare-aware Neural Architecture Search (HW-NAS) has emerged as one of the most promising techniques as it can automate the process of designing optimal DNN structures for the target applications, each of which often corresponds to a different hardware device and requires a different hardware efﬁciency metric (e.g., prioritizes latency or energy).
For example,
HW-NAS in (Wu et al., 2019) develops a differentiable neural architecture search (DNAS) framework and discovers state-of-the-art (SOTA) DNNs balancing both accuracy and hardware efﬁciency, by incorporating a loss that consists of both the cross-entropy loss that leads to better accuracy and the latency loss that penalizes the network’s latency on a target device.
Despite the promising performance achieved by SOTA HW-NAS, there exist paramount challenges that limit the development of HW-NAS innovations.
First, HW-NAS requires the collection of hardware efﬁciency data corresponding to (all) the networks in the search space.
To do so, current practice either pre-collects these data to construct a hardware-cost look-up table or adopts device-
[Figure]
HW
[Figure]
Our Proposed
No Hardware Expertise/ Knowledge Needed
HW-NAS w/o Our Proposed HW-NAS-Bench
HW-NAS w/ Our Proposed HW-NAS-Bench
Possible
[Figure]
Q u e r
y
i ng
F ee db ack HW-NAS Algorithms
Searched Architectures
A cc
Efﬁciency
HW-NAS Researcher Focus
On
EfﬁciencyPerformance
Latency
[Figure]
Hardware Expertise/ Knowledge
[Figure]
Q u e r
y
i ng
F ee db
[Figure]
speciﬁc hardware-cost estimators/models, both of which can be time consuming to obtain and impose a barrier-to-entry to non-hardware experts.
This is because it requires knowledge about devicespeciﬁc compilation and properly setting up the hardware measurement pipeline for hardware-cost data collection.
Second, similar to generic NAS, it can be notoriously difﬁcult to benchmark HWNAS algorithms due to the required signiﬁcant computational resources and the differences in their (1) hardware devices, which are speciﬁc for HW-NAS, (2) adopted search space, and (3) hyperparameters.
Such a difﬁculty is even higher for HW-NAS considering the numerous choices of hardware devices, each of which can favor very different network structures even under the same target hardware efﬁciency, as discussed in (Chu et al., 2020).
While the number of ﬂoating-point operations (FLOPs) have been commonly used to estimate the hardware-cost, many works have pointed out that DNNs with fewer FLOPs are not necessarily faster or more efﬁcient (Wu et al., 2019).
For example, NasNet-A has a comparable complexity in terms of FLOPs as MobileNetV1 yet can have a slower latency than the latter due to NasNet-A’s adopted hardware-unfriendly structure.
It is thus imperative to address the aforementioned challenges in order to make HW-NAS more accessible and reproducible to unfold HW-NAS’s full potential.
Note that although pioneering NAS benchmark datasets (Ying et al., 2019; Dong & Yang, 2020; Klyuchnikov et al., 2020; Siems et al., 2020; Dong et al., 2020) have made a signiﬁcant step towards providing a uniﬁed benchmark dataset for generic NAS works, they all either merely provide latency on server-level GPUs (e.g., GTX 1080Ti) or do not provide hardware-cost metrics on real hardware, limiting their applicability to
HW-NAS (Wu et al., 2019; Wan et al., 2020;
Cai et al., 2018) which primarily targets commercial edge devices, FPGA, and ASIC.
To this end, as shown in Figure 1, we develop HW-NAS-Bench and make the following contributions in this paper:
• We have developed HW-NAS-Bench , the ﬁrst public dataset for HW-NAS research aiming to (1) democratize HW-NAS research to non-hardware experts and (2) provide a uniﬁed benchmark for HW-NAS to make HW-NAS research more reproducible and accessible, covering two SOTA NAS search spaces including NAS-Bench-201 and FBNet, with the former being the most popular NAS search space and the latter having been shown to be the most hardware friendly.
• We provide hardware-cost data collection pipelines for six commonly used hardware devices that fall into three categories (i
e., commercial edge devices, FPGA, and ASIC), in addition to the measured/estimated hardware-cost (e.g., energy cost and latency) on these devices for all the networks in the search space of both NAS-Bench-201 and FBNet.
• We conduct comprehensive analysis of the collected data in HW-NAS-Bench, such as studying the correlation between collected hardware-cost and accuracy-cost trade-offs of all the networks on the six hardware devices, which can provide insights to not only HWNAS researchers but also DNN accelerator designers.
Other researchers can extract other useful insights from HW-NAS-Bench that have not been discussed in this work.
• We demonstrate exemplary user cases to show (1) how HW-NAS-Bench can be easily used by non-hardware experts to develop HW-NAS solutions by simply querying the collected data in our HW-NAS-Bench and (2) that dedicated device-speciﬁc HW-NAS can indeed
often provide optimal accuracy-cost trade-offs, demonstrating the great necessity of HWNAS benchmark frameworks like our proposed HW-NAS-Bench.
R ELATED WORKS 2. H ARD W ARE - AWARE N EURAL A RCHITECTURE S EARCH
Driven by the growing demand for efﬁcient DNN solutions, HW-NAS has been proposed to automate the search for efﬁcient DNN structures under the target efﬁciency constraints.
For example, (Tan et al., 2019; Howard et al., 2019; Tan & Le, 2019) adopts reinforcement learning based
NAS with a multi-objective reward consisting of both task performance and efﬁciency, achieving promising results yet suffering from prohibitive search time/cost.
In parallel, (Wu et al., 2019; Wan et al., 2020; Cai et al., 2018; Stamoulis et al., 2019) explore the design space in a differentiable manner following (Liu et al., 2018) and signiﬁcantly improve the search efﬁciency.
The promising performance of HW-NAS has motivated a tremendous interest in applying it to more diverse applications (Fu et al., 2020; Chen et al., 2019; Marchisio et al., 2020) paired with target hardware devices, e.g., EdgeTPU
(Xiong et al., 2020) and NPU (Lee et al., 2020), in addition to the widely explored mobile phones.
As discussed in (Chu et al., 2020), different hardware devices can favor very different network structures under the same hardware efﬁciency metric, and the optimal network structure can differ signiﬁcantly when considering different application driven hardware efﬁciency metrics on the same hardware devices.
As such, it would ideally lead to the optimal accuracy-efﬁciency trade-offs if the HW-NAS design is dedicated for the target device and hardware efﬁciency metrics.
However, this requires a good understanding of both device-speciﬁc compilation and hardware-cost characterization, imposing a barrier-to-entry to non-hardware experts, such as many NAS researchers, and thus limiting the development of optimal HW-NAS results for numerous applications, each of which often corresponds to a different application driven hardware efﬁciency and adopts different types of hardware devices.
As such, our proposed HW-NAS-Bench will make HW-NAS more friendly to
NAS researchers, who are often non-hardware experts, as it consists of comprehensive hardwarecost data in a wide range of hardware devices for all the networks in two SOTA NAS search spaces, expediting the development of HW-NAS innovations.
2. N EURAL A RCHITECTURE S EARCH B ENCHMARKS .
The importance and difﬁculty of NAS reproducibility and benchmarking has recently gained increasing attention.
Pioneering efforts include (Ying et al., 2019;
Dong & Yang, 2020; Klyuchnikov et al., 2020; Siems et al., 2020; Dong et al., 2020).
Speciﬁcally, NAS-Bench-101 (Ying et al., 2019) develops the ﬁrst large-scale, open-source architecture dataset for NAS, in which they provide the ground truth test accuracy on CIFAR-10 (Krizhevsky et al., 2009) of all the architectures (i.e., 423k) in their search space.
Later, NAS-Bench-201 (Dong & Yang, 2020) further extends NAS-Bench101 to support more NAS algorithm categories (e.g., differentiable algorithms) and more datasets (e.g., CIFAR-100 (Krizhevsky et al., 2009) and ImageNet16-120 (Chrabaszcz et al., 2017)).
Most recently, NAS-Bench-301 (Siems et al., 2020) and NATS-Bench
(Dong et al., 2020) were developed to support benchmarking on larger search spaces.
However, all of these works either merely provide latency on the server-level GPU (e.g., GTX 1080Ti) or do not consider hardware efﬁciency metrics on real hardware, limiting their applicability to HW-NAS (Wu et al., 2019; Wan et al., 2020;
Cai et al., 2018) that primarily target the commercial edge devices, FPGA, and ASIC.
This has motivated us to develop our HW-NAS-Bench, aiming to make HW-NAS more accessible and reproducible.
T HE P ROPOSED HW-NAS-B ENCH F RAMEWORK 3. HW-NAS-B ENCH ’ S C ONSIDERED S EARCH S PACES
To ensure a wide applicability, our HW-NAS-Bench considers two representative NAS search spaces: (1) NAS-Bench-201’s cell-based search space and (2) FBNet search space.
Both contributes valuable aspects to ensure our goal of constructing a comprehensive HW-NAS benchmark framework.
Speciﬁcally, the former enables HW-NAS-Bench to naturally integrate the ground truth ac-
curacy data of
, while the latter ensures that
HW-NAS-Bench includes the most commonly recognized hardware friendly search space.
NAS-Bench-201 Search Space.
Inspired from the search space used in the most popular cell-based
NAS, NAS-Bench-201 adopts a ﬁxed cell search space, where each architecture consists of a predeﬁned skeleton with a stack of the searched cell that is represented as a densely-connected directed acyclic graph (DAG).
Speciﬁcally, it considers 4 nodes and 5 representative operation candidates for the operation set, and varies the feature map sizes and the dimensions of the ﬁnal Fully-connected layers to handle its considered three datasets (i.e., CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and ImageNet16-120 (Chrabaszcz et al., 2017)), leading to a total of 3 × 5 6 = 46875 architectures.
Training log and accuracy are provided for each architecture.
However, NAS-Bench-201 can not be directly used for HW-NAS as it only includes theoretical cost metrics (i.e., FLOPs and the number of parameters (#Params)) and latency on a server-type GPU (i.e., GTX 1080Ti).
HW-NASBench enhances NAS-Bench-201 by providing all the 46875 architectures’ measured/estimated hardware-cost on six hardware devices, that are primarily targeted by SOTA HW-NAS works .
FBNet Search Space.
FBNet (Wu et al., 2019) constructs a layer-wise search space with a ﬁxed macro-architecture, which deﬁnes the number of layers and the input/output dimensions of each layer and ﬁxes ﬁrst and the last three layers with the remaining layers to be searched.
In this way, networks in FBNet (Wu et al., 2019) search space have more regular structure than those in NASBench-201, and have been shown to be more hardware friendly (Fu et al., 2020; Ma et al., 2018).
The 9 considered re-deﬁned cell candidates and 22 unique positions lead to a total of 9 22 ≈ 10 21 unique architectures.
While HW-NAS researchers can develop their search algorithms on top of FBNet (Wu et al., 2019) search space, tedious efforts are required to collect the hardware-cost look-up tables or models on each target devices.
HW-NAS-Bench provides measured/estimated hardware-cost on six hardware devices for all the 10 21 architectures of FBNet search space, aiming to make
HW-NAS research more friendly to non-hardware experts .
3. H ARDWARE - COST C OLLECTION P IPELINE AND T HE C ONSIDERED D EVICES
To collect hardware-cost data for all the architectures in both NAS-Bench-201 and FBNet spaces, we construct a generic hardware-cost collection pipeline (see Figure 2) to automate the process.
Speciﬁcally, the pipeline mainly consists of the target devices and corresponding implementation tools (e.g., compilers), where it takes all the networks as its inputs, compile the networks to (1) convert them into the device’s required execution format and (2) optimize the execution ﬂow, the latter of which aims to optimize the hardware performance on the target devices.
For example, for collecting hardware-cost in Edge GPU, we ﬁrst set the device in the Max-N mode to fully make use of all available resources following (Wofk et al., 2019), and then set up the embedded power rail monitor (Texas Instruments Inc.) to obtain real-measured latency and energy via sysfs (Patrick Mochel and Mike Murphy.) averaged over 50 runs.
We can see that the hardware-cost collection process requires various hardware domain knowledge including machine learning development frameworks, device compilation, embedded systems, and device measurements .
Next, we brieﬂy introduce the six considered hardware devices and the speciﬁc conﬁguration required to collect hardware-cost data using each device.
[Figure]
TensorRT
C Implementation
Hardware Pragma
Conﬁgure Devices in Advance Using Hardware Experts Knowledge
[Table]
Edge GPU: NVIDIA Edge GPU Jetson TX2 (Edge GPU) is a commercial device with a 256-core
Pascal GPU and a 8GB LPDDR4 targeting IoT applications (NVIDIA Inc., a).
When plugging Edge
GPU into the above hardware-cost collection pipeline, we ﬁrst compile the network architectures in both NAS-Bench-201 and FBNet spaces to (1) convert them to the TensorRT format and (2) optimize the inference implementation within NVIDIA’s recommended TensorRT runtime environment, and then execute them in Edge GPU to measure the consumed energy and latency.
Raspi 4: Raspberry Pi 4 (Raspi 4) is the latest Raspberry Pi device (Raspberry Pi Limited.), consisting of a Broadcom BCM2711 SoC and a 4GB LPDDR4.
To collect hardware-cost operating on it, we compile the architecture candidates to (1) convert them into TensorFlow Lite (TFLite)
(Abadi et al., 2016) format (2) optimize the implementation using the ofﬁcial interpreter (Google LLC., 2020) in Raspi 4, which will be conﬁgured in advance.
Edge TPU:
Edge TPU Dev Board (Google LLC.
, a) is a dedicated ASIC accelerator developed by
Google for targeting Artiﬁcial Intelligence (AI) inference for edge applications.
Similar to the case when using Raspi 4, all the architectures are converted into TFLite format.
After that, an Edge TPU compiler will be used to convert the pre-built TFLite model into a more compressed format which is compatible to the pre-conﬁgured Edge TPU runtime environment in the Dev Board.
Pixel 3: Pixel 3 is one of the latest Pixel mobile phones (Google LLC.
, e) that are widely used as the target platform by recent NAS works (Xiong et al., 2020; Howard et al., 2019; Tan et al., 2019).
To collect hardware-cost in Pixel 3, we ﬁrst convert all the architectures into TFLite, then use TFLite’s ofﬁcial benchmark binary ﬁles to obtain the latency, when conﬁguring the Pixel 3 to only use big cores for reducing the measurement variance as in (Xiong et al., 2020; Tan et al., 2019).
ASIC-Eyeriss: For collecting hardware-cost data in ASIC, we consider a SOTA ASIC accelerator (Chen et al., 2016).
Speciﬁcally, we adopt SOTA ASIC accelerator performance simulators (1)
Accelergy (Wu et al., 2019)+Timeloop (Parashar et al., 2019) and (2) DNN-Chip Predictor (Zhao et al., 2020), which automatically identify the optimal algorithm-to-hardware mapping methods for each architecture and then provide the estimated hardware-cost when being executed in Eyeriss.
FPGA: FPGA is a widely adopted AI acceleration platform featuring higher hardware ﬂexibility and decent hardware efﬁciency.
To collect hardware-cost in this platform, we ﬁrst develop a SOTA chunk based pipeline structure (Zhang et al., 2018;
Shen et al., 2017) implementation, compile all the architectures using standard Vivado HLS toolﬂow (Xilinx Inc., a), and then obtain the hardware-cost on a Xilinx ZC706 board with Zynq XC7045 SoC (Xilinx Inc., b).
More details about the pipeline for each device are provided in the Appendix D.
Note that to estimated the hardware-cost of networks in the FBNet search space (Wu et al., 2019) when being executed on the commercial category of edge devices (i.e., edge GPU, Raspi 4, edge
TPU, and Pixel 3) in our HW-NAS-Bench, we sum up hardware-cost of all unique blocks (i
[Table]
e., “block” in the FBNet space (Wu et al., 2019)) within the network architectures.
To validate that
[Table]
such an approximation is close to real-measured results, we conduct the experiments summarized in Table 2, which calculates two types of correlation coefﬁcients between the measured and the approximated hardware-cost based on 100 randomly sampled architectures from the FBNet search space.
We can see that our approximated costs are highly correlated with the real-measured ones, except for the case on EdgeTPU, which we conjecture is caused by our adopted in-house EdgeTPU compiler (Google LLC.
, c).
More visualization results can be found in Appendix A.
A NALYSIS ON HW-NAS-B ENCH
In this section, we provide analysis and visualization of the hardware-cost and corresponding accuracy data (the latter only for architectures in NAS-Bench-201) for all the architectures in the two considered search spaces.
Speciﬁcally, our analysis and visualization conﬁrm that (1) commonly used theoretical hardware-cost metrics such as FLOPs do not correlate well with measured/estimated hardware-cost; (2) hardware-cost of the same architectures can differ a lot when executed on different devices; and (3) device-speciﬁc HW-NAS is necessary because optimal architectures resulting from HW-NAS targeting on one device can perform poorly in terms of hardware efﬁciency when being executed on another device.
4. C ORRELATION B ETWEEN C OLLECTED H ARDWARE - COST AND T HEORETICAL O NES
To conﬁrm whether commonly used theoretical hardware-cost metrics align with real measured/estimated ones, we summarize the calculated correlation between the collected hardware-cost in our HW-NAS-Bench and the theoretical metrics (i.e., FLOPs and #Params), based on data for all the architectures in both search spaces on all the six considered hardware devices where a total of four different datasets are involved.
As summarized in Tables 3 - 4, commonly used theoretical hardware-cost metrics (i.e., FLOPs and #Params) do not always correlate well with measured/estimated hardware-cost for the architectures in both the NAS-Bench-201 and FBNet spaces.
For example, there exists at least one coefﬁcient < 0.5 on all devices, especially for the cases with measured hardware-cost on commonly considered edge platforms including Edge GPU, Edge TPU
[Table]
and ASIC-Eyeriss.
As such, HW-NAS based on theoretical hardware-cost might lead to sub-optimal results, motivating HW-NAS benchmark frameworks like our HW-NAS-Bench.
Note that we consider the Kendall Rank Correlation Coefﬁcients (Abdi, 2007), which is a commonly used correlation coefﬁcient in both recent NAS and NAS benchmark frameworks (You et al., 2020;
Siems et al., 2020;
Yang et al., 2020)
[Table] [Table]
[Table]
4. C ORRELATION A MONG C OLLECTED H ARDWARE -C OST ON D IFFERENT D EVICES
To check how much the hardware-cost of the same architectures on different devices correlate, we visualize the correlation between the hardware-cost collected from every two paired devices base on data for all the architectures on both the NAS-Bench-201 and FBNet search spaces with each of the architecture being associated with 9 different hardware-cost metrics.
The visualization in Figures 3 - 4 indicates that hardware-cost of the same network architectures can differ a lot when being executed on different devices.
More speciﬁcally, the correlation coefﬁcients can be as small as -0.00 (e.g., Edge GPU latency vs. ASIC-Eyeriss energy), which is a result of the large difference in (1) their underlying hardware architectures and (2) the amount of available resources.
Thus, the resulting architecture of HW-NAS targeting one device might perform poorly when being executed on other device, motivating device-speciﬁc HW-NAS, and it is crucial to develop comprehensive hardware-cost datasets like our HW-NAS-Bench to enable fast development and optimal results of HW-NAS for different applications.
4. O PTIMAL A RCHITECTURES ON D IFFERENT H ARDWARE D EVICES
To conﬁrm the necessity of performing device-speciﬁc HW-NAS from another perspective, we summarize all architectures’ test accuracy vs. hardware-cost on ImageNet16-120 of NAS-Benchand focus on the architectures with the optimal accuracy-cost trade-offs.
As shown in Figure 5, such optimal architectures for different devices are not the same.
For example, the optimal architectures on Edge GPU (marked as red points) can perform poorly in terms of hardware efﬁciency in other devices, especially in ASIC-Eyeriss and Edge TPU whose hardware-cost have exactly the smallest correlation coefﬁcient with the hardware-cost measured in the Edge GPU, which is shown in Figure 3.
Again, this set of analysis and visualization conﬁrms that HW-NAS targeting on one device can perform poorly in terms of hardware efﬁciency when being executed on another device, thus motivating the necessity of device-speciﬁc HW-NAS.
U SER C ASE : B ENCHMARK SOTA HW-NAS A LGORITHMS
In this section, we will demonstrate the user cases of our HW-NAS-Bench to show (1) how nonhardware experts can use it to develop HW-NAS solutions by simply querying the hardware-cost data (2) dedicated device-speciﬁc HW-NAS can indeed often lead to optimal accuracy-cost tradeoffs, again showing the need for HW-NAS benchmark frameworks like our HW-NAS-Bench to enable more optimal HW-NAS solutions via device-speciﬁc HW-NAS.
Benchmark Setting.
We adopt a SOTA HW-NAS algorithm, ProxylessNAS (Cai et al., 2018) for this experiment.
As an example to use our HW-NAS-Bench, we use ProxylessNAS to search over the FBNet (Wu et al., 2019) space on CIFAR-100 (Krizhevsky et al., 2009), when targeting different devices in our HW-NAS-Bench by simply querying the corresponding device’s measured/estimated hardware-cost, which has negligible overhead as compared with the NAS algorithm itself, without the need for hardware expertise or knowledge during the whole HW-NAS.
[Figure]
ImageNet16-
[Figure] [Figure]
[Figure]
points in red denote the architectures with the optimal “accuracy on ImageNet16-120 vs. smallest latency measured on Edge GPU” trade-offs of which the architectures represent the ground truth of
HW-NAS targeting Edge GPU
. 5. O PTIMAL A RCHITECTURES R ESULTING FROM D EVICE - SPECIFIC HW-NAS
[Table]
Table 5 illustrates that the searched architectures achieve the lowest latency among all architectures, when the target devices of HW-NAS are the same as the one used to measure the architecture’s ondevice inference latency.
Speciﬁcally, when being executed on Edge GPU, the architecture targeting on Raspi 4 during HW-NAS leads to about a 50% higher latency while the architecture targeting on FPGA during HW-NAS introduces over a 100% higher latency, than the architecture speciﬁcally target on Edge GPU during HW-NAS, under the same inference accuracy.
This set of experiment show that non-hardware experts can easily use our HW-NAS-Bench to develop optimal HW-NAS solutions, while demonstrating the need of device-speciﬁc HW-NAS solutions.
C ONCLUSION
We have developed HW-NAS-Bench, the ﬁrst public dataset for HW-NAS research aiming to (1) democratize HW-NAS research to non-hardware experts and (2) provide a uniﬁed benchmark for
HW-NAS to make HW-NAS research more reproducible and accessible.
Our HW-NAS-Bench covers two representative NAS search spaces, provides all network architectures’ hardware-cost data on six commonly used hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC).
Furthermore, we conduct comprehensive analysis of the collected data in HW-NAS-Bench, aiming to provide insights to not only HW-NAS researchers but also DNN accelerator designers.
Finally, we demonstrate exemplary user cases of HW-NAS-Bench to show (1) how HW-NAS-Bench can be easily used by non-hardware experts to develop HW-NAS solutions by simply querying the collected data and (2) that dedicated device-speciﬁc HW-NAS can indeed often provide optimal accuracy-cost trade-offs, demonstrating the great necessity of HW-NAS benchmark frameworks like our proposed HW-NAS-Bench. A CKNOWLEDGEMENT The work is supported by the National Science Foundation (NSF) through the CNS Division of Computer and Network Systems (Award number: 2016727).
A M ORE V ISUALIZATION ON M EASURED H ARDWARE - COST FOR THE FBN ET S EARCH S PACE
M eas u r e d
E dg e G
P U L a t e n cy
M eas u r e d
E dg e G
P U E n e r g y
M eas u r e d
R as p i
L a t e n cy
M eas u r e d
E dg e T P
U L a t e n cy
M eas u r e d
P i xe l
L a t e n cy
M eas u r e d
E dg e G
P U L a t e n cy
M eas u r e d
E dg e G
P U E n e r g y
M eas u r e d
R as p i
L a t e n cy M eas u r e d E dg e T P U L a t e n cy
M eas u r e d P i xe l L a t e n cy [Figure]
CIFAR
[Figure]
Fig.
6 shows a comparison between the approximated and measured hardware-cost of randomly sampled 100 architectures when being executed on commercial edge devices using the ImageNet and CIFAR-100 datasets, which veriﬁes that our approximation of summing up the performance of the unique blocks is a simple yet quite accurate for providing the hardware-cost for networks in the
FBNet space, consistent with our observation in Table 2.
B C OMPARING THE P ERDITION OF E YERISS U SING A CCELERGY +T IMELOOP AND DNN-C HIP -P REDICTOR
Both Accelergy (Wu et al., 2019)+Timeloop (Parashar et al., 2019) and DNN-Chip Predictor (Zhao et al., 2020) are able to simulate the latency and energy cost of Eyeriss (Chen et al., 2016), a SOTA
ASIC design, when giving the network architectures.
From Table 6, they nearly give the same estimation of the latency and energy cost (i.e. the mean of differences is 6.096%, the standard deviation of the differences is 0.779%, the Pearson correlation coefﬁcient is 0.9998, and the Kendall
Rank correlation coefﬁcient is 0.9633, in term of the average performance) when benchmarked in 3 datasets of NAS-Bench-201.
Therefore, we use the average value of their predictions as the estimated Eyeriss latency and energy in our proposed HW-NAS-Bench .
[Table]
C M INOR M ODIFICATIONS ON FBN ET S EARCH S PACE W HEN B ENCHMARKING ON CIFAR-
We demonstrate our modiﬁcation on FBNet search space when benchmarking on CIFAR-100 (i
[Table]
e., the setting in Section 5) via comparing the marco-architecture before and after such modiﬁcation in
Table 7.
[Table]
D D ETAILS IN THE P IPELINE TO C OLLECT H ARDWARE - COST D. C OLLECT P ERFORMANCE ON E DGE GPU: NVIDIA
Edge GPU Jetson TX2 (Edge GPU) (NVIDIA Inc., a) is a commonly used commercial edge device, consisting of a quad-core Arm Cortex-A57, a dual-core NVIDIA Denver2, a 256-core
Pascal GPU, and a 8GB 128-bit LPDDR4, for various deep learning applications including classiﬁcation (Li et al., 2020), segmentation (Siam et al., 2018), and depth estimation (Wofk et al., 2019), targeting IoT and self-driving environments.
Although widely-used TensorFlow (Abadi et al., 2016) and PyTorch (Paszke et al., 2019) can be used in Edge GPU directly, to achieve faster inference, TensorRT (NVIDIA Inc., b), a C++ library for high-performance inference on NVIDIA GPUs, is more commonly used as the runtime environment in Edge GPU when only benchmarking inference (Wang et al., 2019; NVIDIA Inc., c).
Therefore, we pre-set the Edge GPU in the max-N mode to make full use of the resource on it following (Wofk et al., 2019).
When plugging Edge GPU into the hardware-cost collection pipeline, we ﬁrst compile the PyTorch implementations of network architectures in both NAS-Benchand FBNet spaces to TensorRT format models.
In this way, we can beneﬁt from the optimized inference implementation within TensorRF runtime environment.
And then we can benchmark the architectures in Edge GPU to further measure the energy and latency using the sysfs (Patrick Mochel and Mike Murphy.) of the embedded INA3221 (Texas Instruments Inc.) power rails monitor.
D. C OLLECT P ERFORMANCE ON R ASPI 4:
Raspberry Pi 4 (Raspi 4) (Raspberry Pi Limited.)
is the latest Raspberry Pi device, which is a popular hardware platform for general purpose IoT applications (Zhao et al., 2015; Basu et al., 2020) and is able to support deep learning applications with speciﬁcally framework designs (Google
LLC.
, f; Zhang et al., 2019; Geiger & Team, 2020).
We choose the Raspi 4 type with a Broadcom
BCM2711 SoC and a 4GB LPDDR4 (Raspberry Pi Limited.).
Similar as Edge GPU, Raspi 4 can run architectures in the TensorFlow (Abadi et al., 2016), PyTorch (Paszke et al., 2019) or TensorFlow
Lite (Google LLC., f) runtime environments.
We utilize TensorFlow Lite (Google LLC.
, f) which can further boost the efﬁciency.
To collect hardware-cost operating on Respi 4, an ofﬁcial TensorFlow Lite interpreter is preconﬁgured in the Raspi 4, following the settings in (Google LLC., 2020).
We benchmark the possible architectures in HW-NAS-Bench on Raspi 4 after compiling them to TensorFlow Lite (Abadi et al., 2016) format to measure the resulting latency.
D. C OLLECT P ERFORMANCE ON E DGE TPU:
Edge TPU (Google LLC.
, a) is a series of dedicated ASIC accelerators developed by Google for targeting AI inference at the edge, which can be used for classiﬁcation, pose estimation, and segmentation (Xiong et al., 2020; Google LLC.
, b) with extremely high efﬁciency (e.g., 2 .
32 × more efﬁcient than a single SOTA desktop GPU, GTX
Ti, in terms of number of ﬁxed-point operations per watt (Google LLC.
, d)).
In our proposed collection pipeline, we choose the Dev Board (Google
LLC., a) which provides the most functions among all products.
To collect hardware-cost in Edge TPU, all the architectures to be benchmarked will be converted to TensorFlow Lite (Google LLC.
, f) format ﬁrst from Keras (Chollet et al., 2015) implementation.
After that, an in-house compiler (Google LLC.
, c) will convert the TensorFlow Lite models into a more compressed format.
This pipeline uses the least converting tools to make sure as many operations as possible are supported compared with others (e.g., from PyTorch-ONNX (Bai et al., 2020) implementation).
Only latency is collected for Edge TPU since it lacks accurate embedded power rails monitor.
We do not consider FBNet’ search space for Edge TPU and more details are in
Appendix A.
D. C OLLECT P ERFORMANCE ON P IXEL 3:
Pixel 3 (Google LLC.
, e) is one of the latest Pixel mobile phones that are widely used as the target platform by recent NAS works (Xiong et al., 2020;
Howard et al., 2019; Tan et al., 2019) and machine learning framework benchmark (Google LLC., f).
In our implementation, the Pixel 3 is pre-conﬁgured to use big cores following the setting in (Xiong et al., 2020; Tan et al., 2019).
Similar to the case of Raspi 4, we ﬁrst convert the possible architectures in the search spaces of our proposed
HW-NAS-Bench into TensorFlow Lite format and then use the ofﬁcial benchmark binary ﬁles to measure the latency for each architecture.
D. C OLLECT P ERFORMANCE ON ASIC-E YERISS :
For hardware-cost data colection in ASIC, we consider Eyeriss (ASIC-Eyeriss) a SOTA ASIC accelerator (Chen et al., 2016).
The Eyeriss chip features 168 processing elements (PEs) which are connected through a conﬁgurable dedicated on-chip network into a 2D array.
A 128KB SRAM is shared by all PEs and further divided into multiple banks, each of which can be assigned to ﬁt the input feature maps or partial sums.
Thanks to these conﬁgurable hardware settings, we can deploy the optimal algorithm-to-hardware mappings on Eyeriss to minimize the energy or latency by exploiting data reuse for different layers.
In order to ﬁnd the optimal mappings and evaluate the performance metrics on Eyeriss, we adopt SOTA performance simulators for DNN accelerators (1) Accelergy (Wu et al., 2019)+Timeloop (Parashar et al., 2019) and (2) DNN-Chip Predictor (Zhao et al., 2020).
Both of the simulators can characterize the Eyeriss’ architecture, do mapping exploration, and predict the energy cost and latency metrics.
Given the Eyeriss accelerator and layer information (e.g, layer type, feature map size, kernel size) in NAS-Bench-201 and FBNet, Accelergy+Timeloop reports the energy cost and latency characterization through an integrated mapper that ﬁnds the optimal mapping for such layer when being executed in Eyeriss.
The inputs to DNN-Chip Predictor are the same as
Accelergy+Timeloop, except that we can set the optimization metric as energy/latency/energy-delay product.
DNN-Chip Predictor schedules mapping for the optimization metric and generates the estimated hardware-cost.
We report the average prediction from the two as the estimated hardware-cost of Eyeriss, and more details can be found in Appendix B.
D. C OLLECT P ERFORMANCE ON FPGA:
FPGA is a widely adopted AI acceleration platform which can offer higher ﬂexibility in terms of hardware for accelerating AI applications.
For collecting hardware-cost data in FPGA, we construct a SOTA chunk based pipeline structure (Zhang et al., 2018;
Shen et al., 2017) as our FPGA implementation.
By conﬁguring multiple sub-accelerators (chunks) and assigning different layers to different sub-accelerators(chunks), we can balance the throughput and hardware resource consumption.
To further free up our implantation’s potential to reach the performance frontier across different architectures, we additionally conﬁgure hardware settings such as the number of PEs, interconnection method of PEs, and tiling/scheduling of the operations, which are commonly adopted by FPGA accelerators (Chen et al., 2017; Zhang et al., 2015; Yang et al., 2016).
We then compile all the architectures using standard Vivado HLS toolﬂow (Xilinx Inc., a) and obtain the bottleneck latency, the maximum latency across all sub-accelerators (chunks) of the architectures on a Xilinx
ZC706 development board with Zynq XC7045 SoC (Xilinx Inc., b).
To verify our implementation, we compare our implementation’s performance with SOTA FPGA accelerators (Zhang et al., 2018;
Xiao et al., 2017) given the same architecture and dataset as shown in Table 8.
We can see that our implementation has SOTA performance and thus provides insightful and trusted hardware-cost estimation for the HW-NAS-Bench.
[Table]
[Table]