  HardWare-aware Neural Architecture Search (HW-NAS) has recently gained
tremendous attention by automating the design of DNNs deployed in more
resource-constrained daily life devices. Despite its promising performance,
developing optimal HW-NAS solutions can be prohibitively challenging as it
requires cross-disciplinary knowledge in the algorithm, micro-architecture, and
device-specific compilation. First, to determine the hardware-cost to be
incorporated into the NAS process, existing works mostly adopt either
pre-collected hardware-cost look-up tables or device-specific hardware-cost
models. Both of them limit the development of HW-NAS innovations and impose a
barrier-to-entry to non-hardware experts. Second, similar to generic NAS, it
can be notoriously difficult to benchmark HW-NAS algorithms due to their
significant required computational resources and the differences in adopted
search spaces, hyperparameters, and hardware devices. To this end, we develop
HW-NAS-Bench, the first public dataset for HW-NAS research which aims to
democratize HW-NAS research to non-hardware experts and make HW-NAS research
more reproducible and accessible. To design HW-NAS-Bench, we carefully
collected the measured/estimated hardware performance of all the networks in
the search spaces of both NAS-Bench-201 and FBNet, on six hardware devices that
fall into three categories (i.e., commercial edge devices, FPGA, and ASIC).
Furthermore, we provide a comprehensive analysis of the collected measurements
in HW-NAS-Bench to provide insights for HW-NAS research. Finally, we
demonstrate exemplary user cases to (1) show that HW-NAS-Bench allows
non-hardware experts to perform HW-NAS by simply querying it and (2) verify
that dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost
trade-offs. The codes and all collected data are available at
https://github.com/RICE-EIC/HW-NAS-Bench.
