  Neural architecture search (NAS) typically consists of three main steps:
training a super-network, training and evaluating sampled deep neural networks
(DNNs), and training the discovered DNN. Most of the existing efforts speed up
some steps at the cost of a significant slowdown of other steps or sacrificing
the support of non-differentiable search metrics. The unbalanced reduction in
the time spent per step limits the total search time reduction, and the
inability to support non-differentiable search metrics limits the performance
of discovered DNNs.
  In this paper, we present NetAdaptV2 with three innovations to better balance
the time spent for each step while supporting non-differentiable search
metrics. First, we propose channel-level bypass connections that merge network
depth and layer width into a single search dimension to reduce the time for
training and evaluating sampled DNNs. Second, ordered dropout is proposed to
train multiple DNNs in a single forward-backward pass to decrease the time for
training a super-network. Third, we propose the multi-layer coordinate descent
optimizer that considers the interplay of multiple layers in each iteration of
optimization to improve the performance of discovered DNNs while supporting
non-differentiable search metrics. With these innovations, NetAdaptV2 reduces
the total search time by up to $5.8\times$ on ImageNet and $2.4\times$ on NYU
Depth V2, respectively, and discovers DNNs with better
accuracy-latency/accuracy-MAC trade-offs than state-of-the-art NAS works.
Moreover, the discovered DNN outperforms NAS-discovered MobileNetV3 by 1.8%
higher top-1 accuracy with the same latency. The project website is
http://netadapt.mit.edu.
