Training-free Neural Architecture Search for RNN s and Transformers (2023)

Abstract: Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures, offering an alternative to the laborious process of manually designing complex architectures. However, traditional NAS algorithms are slow and require immense amounts of computing power. Recent research has investigated training-free NAS metrics for image classification architectures, drastically speeding up search algorithms. In this paper, we investigate training-free NAS metrics for recurrent neural network (RNN) and BERT-based transformer architectures, targeted towards language modeling tasks. First, we develop a new training-free metric, named hidden covariance, that predicts the trained performance of an RNN architecture and significantly outperforms existing training-free metrics. We experimentally evaluate the effectiveness of the hidden covariance metric on the NAS-Bench-NLP benchmark. Second, we find that the current search space paradigm for transformer architectures is not optimized for training-free neural architecture search. Instead, a simple qualitative analysis can effectively shrink the search space to the best performing architectures. This conclusion is based on our investigation of existing training-free metrics and new metrics developed from recent transformer pruning literature, evaluated on our own benchmark of trained BERT architectures. Ultimately, our analysis shows that the architecture search space and the training-free metric must be developed together in order to achieve effective results. Our source code is available at https://github.com/aaronserianni/training-free-nas .

## introduction recurrent neural networks (rnns) and bertbased transformer models with self-attention have been extraordinarily successful in achieving stateof-the-art results on a wide variety of language modeling-based natural language processing (nlp) tasks, including question answering, sentence classification, tagging, and natural language inference @xcite @xcite @xcite . however, the manual development of new neural network architectures has become increasingly difficult as models are getting larger and more complicated. neural architecture search (nas) algorithms aim to procedurally design and evaluate new, efficient, and effective architectures within a predesignated search space @xcite . nas algorithms have been extensively used for developing new convolutional neural network (cnn) architectures for image classification, with many surpassing manually-designed architectures and achieving state-of-the-art results on many classification benchmarks @xcite . some research has been conducted on nas for rnns and transformers @xcite @xcite , particularly with bert-based architectures @xcite @xcite @xcite , but nas is not widely used for designing these architectures. while nas algorithms and methods have been successful in developing novel and effective architectures, there are two main problems that current algorithms face. the search space for various architectures is immense, and the amount of time and computational power to run nas algorithms is prohibitively expensive @xcite . because traditional nas algorithms require the evaluation of candidate architectures in order to gauge performance, candidate architectures need to be trained fully, each taking days or weeks to complete. thus, past attempts at nas have been critiqued for being computationally resource-intensive, consuming immense amounts of electricity, and producing large amounts of carbon emissions @xcite . these problems are especially true for transformers and rnns, as they have more parameters and take longer to train when compared to other architectures @xcite . recently, there has been research into trainingfree nas metrics and algorithms, which offer significant performance increases over traditional nas algorithms @xcite @xcite . these metrics aim to partially predict an architecture's trained accuracy from its initial untrained state, given a subset of inputs. however, prior research has focused on developing training-free nas metrics for cnns and vision transformers with image classification tasks. in this work, we apply existing training-free metrics and create our own metrics for rnns and bert-based transformers with language modeling tasks. our main contributions are: ## related work since the development and adoption of neural architecture search, there has been research into identifying well-performing architectures without the costly task of training candidate architectures. ## training-free nas metrics a series of training-free nas metrics have been proposed in recent literature. these metrics look at specific aspects of an architecture, such as parameter gradients, activation correlations, and weight matrix rank. most metrics can be generalized to any type of neural network, but have only been tested on cnn architectures. for transformer architectures, we also adapt various attention parameter pruning metrics as training-free metrics, scoring the entire network. ## experimental results of training-free metrics for the training-free nas metrics presented, we empirically evaluate how well the metric performs in predicting the trained performance of an architecture. we use kendall rank correlation coefficient (kendall τ ) and spearman rank correlation coefficient (spearman ρ) to quantitatively measure the metrics' performance. ## discussion neural architecture search for transformers is a fundamentally different task than neural architecture search for cnns and rnns. almost all search spaces for transformers rely on the same fundamental paradigm of an attention module followed by a feed-forward module within each encoder/decoder layer, connected linearly @xcite @xcite . conversely, most search spaces for cnns and rnns, including nas-bench-201 and nas-bench-nlp, use a cell-based method, typically with an acyclic digraph representing the connections between operations @xcite @xcite , allowing for significantly more flexibility in cell variation. for cnn and rnn search spaces, the connections between operations within a cell have a greater impact on the architecture's performance than number of parameters. in nas-bench-nlp, there is no correlation between number of parameters and model performance (figure 5 ); hence, previous studies did not need to normalize their training-free metrics for number of parameters or features. we hypothesize that for transformer search spaces, the number of parameters in an architecture dominates the model performance, explaining the poor performance for training-free nas metrics. the dependence on number model size for transformer models reveals a significant problem re-garding transformer architecture search: the inflexibility of current transformer search spaces. unless transformer search spaces adopt the variability of connections provided by a cell-based methods, as used by cnn and rnn search spaces, simple heuristics such as number of parameters and features will be the primary training-free predictors of transformer model performance. to our knowledge, only three works have utilized cell-based methods for transformer search spaces, the original transformer architecture search paper, "the evolved transformer" by @xcite , its successor "primer" @xcite , and "autobert-zero" @xcite . some research has been done with cell-based search spaces for conformers @xcite and vision transformers @xcite , but only on the convolution modules of the architectures. ultimately, there is significant opportunity for growth regarding transformer architecture search, and with it training-free nas metric for transformers. ## conclusion in this paper, we presented and evaluated a series of training-free nas metrics for rnn and bertbased transformer architectures, trained on language modeling tasks. we developed new trainingfree metrics targeted towards specific architectures, hidden covariance for rnns, and three metrics based on attention head pruning for transformers. we first verified the training-free metrics on nas-bench-nlp, and found our hidden covariance metric outperforms existing training-free metrics on rnns. we then developed our own nas benchmark for transformers within the flexibert search space, utilizing the electra scheme to significantly speed up pretraining. evaluating the training-free metrics on our benchmark, our proposed attention confidence metric performs the best. however, the current search space paradigm for transformers is not well-suited for training-free metrics, and the number of parameters within a model is the best predictor of transformer performance. our research shows that training-free nas metrics are not universally successful across all architectures, and better transformer search spaces should be developed for training-free metrics to succeed. we hope that our work is a foundation for further research into training-free metrics for rnns and transformers, in order to develop better and more efficient nas techniques.