  Neural Architecture Search (NAS) has enabled the possibility of automated
machine learning by streamlining the manual development of deep neural network
architectures defining a search space, search strategy, and performance
estimation strategy. To solve the need for multi-platform deployment of
Convolutional Neural Network (CNN) models, Once-For-All (OFA) proposed to
decouple Training and Search to deliver a one-shot model of sub-networks that
are constrained to various accuracy-latency tradeoffs. We find that the
performance estimation strategy for OFA's search severely lacks
generalizability of different hardware deployment platforms due to single
hardware latency lookup tables that require significant amount of time and
manual effort to build beforehand. In this work, we demonstrate the framework
for building latency predictors for neural network architectures to address the
need for heterogeneous hardware support and reduce the overhead of lookup
tables altogether. We introduce two generalizability strategies which include
fine-tuning using a base model trained on a specific hardware and NAS search
space, and GPU-generalization which trains a model on GPU hardware parameters
such as Number of Cores, RAM Size, and Memory Bandwidth. With this, we provide
a family of latency prediction models that achieve over 50% lower RMSE loss as
compared to with ProxylessNAS. We also show that the use of these latency
predictors match the NAS performance of the lookup table baseline approach if
not exceeding it in certain cases.
