R ank NAS : Efficient Neural Architecture Search by Pairwise Ranking (2021)

Abstract: This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by formulating the task as a ranking problem. Previous methods require numerous training examples to estimate the accurate performance of architectures, although the actual goal is to find the distinction between “good” and “bad” candidates. Here we do not resort to performance predictors. Instead, we propose a performance ranking method (RankNAS) via pairwise ranking. It enables efficient architecture search using much fewer training examples. Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates. Extensive experiments on machine translation and language modeling tasks show that RankNAS can design high-performance architectures while being orders of magnitude faster than state-of-the-art NAS systems.

## introduction neural architecture search (nas) has advanced state-of-the-art on various tasks, such as image classification @xcite @xcite , machine translation @xcite , and language modeling @xcite @xcite . despite the remarkable results, conventional nas methods are computationally expensive, requiring training millions of architectures during search. for instance, obtaining a state-of-the-art machine translation model with an evolutionary algorithm requires more than 250 gpu years @xcite . several techniques have been proposed to improve the search efficiency, such as sharing parameters among all architectures @xcite @xcite , predicting the performance instead of full training (liu et al., 2018; * corresponding author. our method significantly accelerates nas through pairwise ranking and search space pruning. @xcite @xcite , and searching over a continuous space @xcite @xcite . unfortunately, these approaches still suffer from the high cost of predicting the performance of each candidate architecture. an inherent reason for this is that obtaining accurate performance requires training numerous neural networks to convergence, as described in sec. 2.2. however, it is unnecessary to predict the model performance as in previous nas methods. rather, all we need is to distinguish architectures of different quality in nas, say, ranking these architectures. in this paper, we approach the problem by formulating nas as a ranking task. here we propose ranknas, a ranking model for comparing different architectures. one of the key challenges is that directly ranking all architectures in a large search space is still computationally infeasible. therefore, we adopt the pairwise method @xcite , where the ranking problem is reduced to a binary classification problem over architecture pairs. to speed up ranknas further, we develop an architecture selection method that chooses the most promising architectures for evalu-ation according to the importance of features, e.g., the topology of architectures. we test ranknas on well-established machine translation and language modeling benchmarks. experiments show that ranknas is orders of magnitude faster than standard nas systems and can find better architectures. notably, ranknas is generic to different tasks and evaluation metrics. it achieves competitive results on hardware-aware nas tasks and is 10× faster than the hat baseline @xcite . it also discovers new architectures that outperform vanilla transformer by +1.8 bleu points on the iwslt'14 de-en data and +1.5 bleu points on the wmt'14 en-de data, surpassing the evolved transformer @xcite with 150,000× less search cost. ## preliminaries nas generally consists of two steps: 1) sample architectures from the pre-defined search space, and 2) estimate the performance of these samples. this work focuses on the performance estimation step, which is the efficiency bottleneck of nas. ## nas as ranking as mentioned in sec. 2.2, the goal of nas is to find promising architectures that achieve high performance on unseen data. nas requires distinguishing whether the architectures are "good" or "bad" rather than predicting accurate performance. therefore, it is natural to treat nas as a ranking problem, in which the explicit goal is to rank different architectures correctly. ## analysis we analyze both the accuracy and efficiency of our search method and study the effect of different features on model performance. ## related work recent works explored performance prediction based on architectural properties, i.e., the network topology and the model size @xcite @xcite . for instance, hardware-aware transformer (hat) @xcite encoded architectures into feature vectors and predicted the latency with a multilayer perceptron (mlp) for the target hardware. brp-nas @xcite proposed an end-to-end performance predictor based on a graph convolutional network (gcn). although these methods greatly improve the performance estimation efficiency, they still require many samples and train numerous neural networks to converge, thereby increasing the search cost. instead, we are motivated by the fact that nas is expected to distinguish different candidate architectures. thus, nas can be solved by learning pairwise ranking rather than obtaining the accurate performance of architectures. ## conclusion we have presented ranknas, a simple yet efficient nas algorithm for both hardware-aware and highaccuracy architecture search. we have shown that pairwise ranking can significantly improve search efficiency. we also have proposed a search space pruning method to help the ranking model be more efficient during the search. our approach outperforms prior methods in both efficiency and accuracy. ranknas requires 80% less time in ranking model training on the hardware-aware search task and accelerates the overall search process by 11.53 times. also, the architectures discovered by our method outperform state-of-the-art transformer models in terms of efficiency and accuracy.