FENAS : Flexible and Expressive Neural Architecture Search (2020)

Abstract: Architecture search is the automatic process of designing the model or cell structure that is optimal for the given dataset or task. Recently, this approach has shown good improvements in terms of performance (tested on language modeling and image classification) with reasonable training speed using a weight sharing-based approach called Efficient Neural Architecture Search (ENAS). In this work, we propose a novel architecture search algorithm called Flexible and Expressible Neural Architecture Search (FENAS), with more flexible and expressible search space than ENAS, in terms of more activation functions, input edges, and atomic operations. Also, our FENAS approach is able to reproduce the well-known LSTM and GRU architectures (unlike ENAS), and is also able to initialize with them for finding architectures more efficiently. We explore this extended search space via evolutionary search and show that FENAS performs significantly better on several popular text classification tasks and performs similar to ENAS on standard language model benchmark. Further, we present ablations and analyses on our FENAS approach.

## introduction architecture search enables automatic ways of finding the best model architecture and cell structures for the given task or dataset, as opposed to the traditional approach of manually tuning among different architecture choices. recently, this idea has been successfully applied to the tasks of language modeling and image classification @xcite @xcite . the first approach of architecture search involved an rnn controller which samples a model architecture and uses the validation performance of this architecture trained on the given dataset as feedback (or reward) to sample the next architec-1 3 2 h[t] 0 x [t] h [t-1] tanh (1) relu (2) add (3) h [t-1] x [t] x [t] tanh node 1 <@xmath0. however, this process is computationally very expensive, making it infeasible to run on a single gpu in a reasonable amount of time. some recent attempts have made architecture search more computationally feasible @xcite , with further performance improvements by @xcite who introduced efficient neural architecture search (enas) and achieved strong results on language modeling and image classification tasks. in this work, we present a new architecture search approach called flexible and expressible neural architecture search (fenas) with less restrictive and more flexible search space than enas. fenas search space has more number of activation functions (e.g., skip-based tanh, relu) and new atomic-level operations (e.g., addition, element-wise multiplication), as shown in fig. 1 . importantly, unlike enas, fenas can represent previous well-known human-designed architectures such as the gated recurrent unit (gru) and long short-term memory (lstm) in its search space, allowing it to have flexible number of input edges. unlike enas, we do not use weight-sharing strategy during the architecture search, but instead use evolutionary search @xcite and initialize the population with known human-designed rnn architectures to search the space efficiently. we conduct several experiments on a standard language modeling benchmark (ptb) and text classification tasks from glue benchmark @xcite . to the best of our knowledge, we are the first ones to compare nas methods on the full glue benchmark. comparing our fenas approach with the previous nas approaches, fe-nas performs similarly on ptb and significantly better on several downstream glue tasks. finally, we provide various advantages of fenas over enas, and also analyze the learned fenas cell structure for ptb, e.g., learned cell has fewer skip-connections and less network complexity. ## related work neural architecture search (nas) @xcite has been shown to achieve better performance than the human-designed deep networks for image classification @xcite @xcite @xcite @xcite and language modeling @xcite @xcite @xcite . several sampling strategies have been explored for finding the nas optimal cell in the context of reinforcement learning @xcite @xcite , evolutionary algorithms @xcite @xcite @xcite , and performance predictors @xcite . early nas approach @xcite took many days and thousands of gpu hours to train on simple datasets like ptb @xcite and @xcite . recently, a weight-sharing strategy among search space parameters has been proposed by @xcite , which reduced resource requirements to a few gpu days. later, various variations of this approach have been proposed, e.g., @xcite replaced rl with gradient descent. @xcite and @xcite showed that a simple random search approach can give best results. in our work, we propose a new approach with a better search space making nas flexible and expressible in comparison to @xcite . ## fenas method details similar to enas, our method has two stages. in stage-1, we search for an optimal cell, and in stage-2, we train a model using the optimal cell structure. for the rest of this section, we describe our method and the search approach for learning optimal cell. ## conclusion we presented a new architecture search algorithm (fenas) which has more activation functions and more inputs to the computational nodes than the previous best algorithm (enas), thus achieving more flexible and expressible architectures. our fenas approach is also able to reproduce the wellknown lstm and gru architectures, and is also able to initialize with them for finding architectures more efficiently. we also present the first detailed comparison of several nas methods on the full glue benchmark, and achieve significant improvements on several text classification tasks.