Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. 10 4 GPU hours) makes it difficult to directly search the architectures on large-scale tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly w.r.t. candidate set size). As a result, they need to utilize proxy tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These architectures optimized on proxy tasks are not guaranteed to be optimal on target task. In this paper, we present Proxyless-NAS that can directly learn the architectures for large-scale target tasks and target hardware platforms. We address the high memory consumption issue of differentiable NAS and reduce the computational cost (GPU hours and GPU memory) to the same level of regular training while still allowing a large candidate set. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of directness and specialization. On CIFAR-10, our model achieves 2.08% test error with only 5.7M parameters, better than the previous state-of-the-art architecture AmoebaNet-B, while using 6× fewer parameters. On ImageNet, our model achieves 3.1% better top-1 accuracy than MobileNetV2, while being 1.2× faster with measured GPU latency. We also apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design. 1 1 Pretrained models and evaluation code are released at https://github.com/MIT-HAN-LAB/ProxylessNAS. Figure 1 : ProxylessNAS directly optimizes neural network architectures on target task and hardware. Benefit from the directness and specialization, ProxylessNAS can achieve remarkably better results than previous proxy-based approaches. proxy ( Figure 1 ). We also remove the restriction of repeating blocks in previous NAS works Liu et al., 2018c) and allow all of the blocks to be learned and specified. To achieve this, we reduce the computational cost (GPU hours and GPU memory) of architecture search to the same level of regular training in the following ways. GPU hour-wise, inspired by recent works (Liu et al., 2018c; Bender et al., 2018) , we formulate NAS as a path-level pruning process. Specifically, we directly train an over-parameterized network that contains all candidate paths ( Figure 2 ). During training, we explicitly introduce architecture parameters to learn which paths are redundant, while these redundant paths are pruned at the end of training to get a compact optimized architecture. In this way, we only need to train a single network without any meta-controller (or hypernetwork) during architecture search. However, naively including all the candidate paths leads to GPU memory explosion (Liu et al., 2018c; Bender et al., 2018) , as the memory consumption grows linearly w.r.t. the number of choices. Thus, GPU memory-wise, we binarize the architecture parameters (1 or 0) and force only one path to be active at run-time, which reduces the required memory to the same level of training a compact model. We propose a gradient-based approach to train these binarized parameters based on Bina-ryConnect (Courbariaux et al., 2015). Furthermore, to handle non-differentiable hardware objectives (using latency as an example) for learning specialized network architectures on target hardware, we model network latency as a continuous function and optimize it as regularization loss. Additionally, we also present a REINFORCE-based (Williams, 1992) algorithm as an alternative strategy to handle hardware metrics. In our experiments on CIFAR-10 and ImageNet, benefit from the directness and specialization, our method can achieve strong empirical results. On CIFAR-10, our model reaches 2.08% test error with only 5.7M parameters. On ImageNet, our model achieves 75.1% top-1 accuracy which is 3.1% higher than MobileNetV2 (Sandler et al., 2018) while being 1.2× faster. Our contributions can be summarized as follows: • ProxylessNAS is the first NAS algorithm that directly learns architectures on the largescale dataset (e.g. ImageNet) without any proxy while still allowing a large candidate set and removing the restriction of repeating blocks. It effectively enlarged the search space and achieved better performance. • We provide a new path-level pruning perspective for NAS, showing a close connection between NAS and model compression (Han et al., 2016) . We save the memory consumption by one order of magnitude by using path-level binarization. • We propose a novel gradient-based approach (latency regularization loss) for handling hardware objectives (e.g. latency). Given different hardware platforms: CPU/GPU/FPGA/TPU/NPU, ProxylessNAS enables hardware-aware neural network specialization that's exactly optimized for the target hardware. To our best knowledge, it is the first work to study specialized neural network architectures for different hardware architectures. • Extensive experiments showed the advantage of the directness property and the specialization property of ProxylessNAS. It achieved state-of-the-art accuracy performances on CIFAR-10 and ImageNet under latency constraints on different hardware platforms (GPU, CPU and mobile phone). We also analyze the insights of efficient CNN models specialized for different hardware platforms and raise the awareness that specialized neural network architecture is needed on different hardware architectures for efficient inference.
ProxylessNAS REF propose to binarize the architecture parameters and force only one path to be active during running, which reduces the required memory through requires GPU memory management.
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
ArXiv
ArXiv
Mathematics Computer Science