Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks (2020)

Abstract: Neural Architecture Search (NAS) methods, which automatically learn entire neural model or individual neural cell architectures, have recently achieved competitive or state-of-the-art (SOTA) performance on variety of natural language processing and computer vision tasks, including language modeling, natural language inference, and image classification. In this work, we explore the applicability of a SOTA NAS algorithm, Efficient Neural Architecture Search (ENAS) (Pham et al., 2018) to two sentence pair tasks, paraphrase detection and semantic textual similarity. We use ENAS to perform a micro-level search and learn a task-optimized RNN cell architecture as a drop-in replacement for an LSTM. We explore the effectiveness of ENAS through experiments on three datasets (MRPC, SICK, STS-B), with two different models (ESIM, BiLSTM-Max), and two sets of embeddings (Glove, BERT). In contrast to prior work applying ENAS to NLP tasks, our results are mixed – we find that ENAS architectures sometimes, but not always, outperform LSTMs and perform similarly to random architecture search.

## introduction neural architecture search (nas) methods aim to automatically discover neural architectures that perform well on a given task and dataset. these methods search over a space of possible model architectures, looking for ones that perform well on the task and will generalize to unseen data. there has been substantial prior work on how to define the architecture search space, search over that space, and estimate model performance @xcite . recent works, however, cast doubt on the quality and performance of nas-optimized architectures * work completed while interning at amazon. @xcite , showing that current methods fail to find the best performing architectures for a given task and perform similarly to random architecture search. in this work, we explore applications of a sota nas algorithm, enas @xcite , to two sentence-pair tasks, paraphrase detection (pd) and semantic textual similarity (sts). we conduct a large set of experiments testing the effectiveness of enas-optimized rnn architectures across multiple models (esim, bilstm-max), embeddings (bert, glove) and datasets (mrpc, sick, sts-b). we are the first, to our knowledge, to apply enas to pd and sts, to explore applications across multiple embeddings and traditionally lstm-based nlp models, and to conduct extensive sota hpt across multiple enas-rnn architecture candidates. our experiments suggest that baseline lstm models, with appropriate hyperparameter tuning (hpt), can sometimes match or exceed the performance of models with enas-rnns. we also observe that random architectures sampled from the enas search space offer a strong baseline, and can sometimes outperform enas-rnns. given these observations, we recommend that researchers (i) conduct extensive hpt (preferably using automated methods) across various candidate architectures for the fairest comparisons; (ii) compare the performances of enas-rnns against both standard architectures like lstms and rnn cells randomly sampled from the enas search space; (iii) examine the computational (memory and runtime) requirements of enas methods alongside the gains observed. current sota approaches focus on learning new cell architectures as replacements for lstm or convolutional cells @xcite @xcite @xcite or entire model architectures to replace hand-designed models such as the transformer or densenet @xcite . recently, the superiority of nas to random architecture search and traditional architectures with sota hpt methods has been called into question. @xcite discuss reproducibility issues with current nas methods and find that, on language modeling and image classification tasks, nas algorithms perform similarly to random architecture search. similarly, sciuto et al. (2020) find minimal differences in performance between nas and random search and that the popular weightsharing strategy @xcite decreases performance. with this in perspective, we conduct a study to investigate the value added by enas to two nlp tasks, pd and sts, which, to our knowledge, have not been been explored in previous nas literature. ## neural architecture search for sentence-pair tasks in this work, we explore applications of enas to two sentence-pair tasks, pd and sts. we select enas because prior work @xcite has shown promising results applying it to a closely-related task, nli, with gains of up to 1.3% absolute over lstms and 1.6% over an rnn with a random architecture. through our evaluations on pd and sts, we aim to study whether the enas methods used in prior work for nli are generalizable and whether the results hold when applied to related tasks and datasets. enas models consist of two parts: 1) a search space over model architectures, i.e. child models, and 2) a controller that samples architectures from that search space. the primary contribution of enas is that all child models in the search space share their weights, so each child model does not have to be trained from scratch to evaluate it. train-ing the child models and controller proceeds as follows -first, the controller is fixed, and the child models are trained together for one epoch on the dataset, sampling a new architecture from the controller to use for each minibatch. then, the child model shared parameters are fixed, and the controller is updated -we sample child architectures from its policy and update the controller to maximize the expected reward on the dev set (e.g. dev set accuracy). this two-step process then repeats for a specified number of epochs. after training is complete, a number of child models are sampled from the controller and the best one is trained from scratch and evaluated on the test set. we refer the reader to @xcite for further details on enas. in this work, we follow the setup of @xcite , using standard lstm-based nlp models and replacing the lstms with rnn cells sampled from the enas controller. we leave the rest of the model architecture (e.g. attention, pooling, output layers) the same, so the child model search space consists of every possible enas-rnn architecture with the standard model architecture around it. as with standard enas training, the parameters of the enas-rnns and standard model architecture (e.g. final output layer) are shared across all child models. ## results examining esim, the results are mixed. esim models with enas-rnns in both layers (rows 4, 10) are worse than esim, lstm on 4 of 6 dataset, embedding configurations. the best esim, enas performance is actually achieved using a mix of enas-rnns and lstms across different layers. in fact, the only configurations in which esim, enas outperforms esim, lstm across all three datasets is bert, enas / lstm) (row 5), where we only replace the first lstm layer with an enas-rnn. the gains, however, are modest compared to those of the blm model, improving over esim, lstm by 0.73% on average. further, changing the embeddings to glove glove, enas / lstm) (row 11), esim, enas underperforms esim, lstm across all 3 datasets by nearly 2% on average. since we do not observe similar performance gains with esim as with blm, we hypothesize that optimization of specific rnn architectures might matter less as model complexity (e.g. number of layers) increases. we suggest future work further examine the importance of enas as it relates to model complexity, especially on tasks where an rnn's architecture might have a higher impact on modeling performance. ## conclusion unlike prior work applying enas to nlp, we find that enas-rnns only outperform lstms and random search on some dataset, embedding, model) configurations. our findings parallel recent work @xcite which question the effectiveness of current nas methods and their superiority to random architecture search and sota hpt methods. given our mixed results, we recommend researchers: (i) extensively tune hyperparameters for standard (e.g. lstm) and randomly sampled architectures to create strong baselines; (ii) benchmark enas performance across multiple simple and complex model architectures (e.g. blm & esim); (iii) present computational requirements alongside gains observed with enas methods. hyperparameter search space batch size [16, 32, 64] learning rate 0.0001 -0.01 loss function classification: cross entropy regression: [mae, mse] weight decay 0.001 -0.1 grad norm 0.25 -20.0 hidden dimensions w/ bert: [384, 512, 768, 1152, 1536] w/ glove: [150, 200, 300, 450, 600] dropouts 0.25 -0.75 random seed [0, 1, 2, 3, 4, 5] epochs 75 (with early stopping) rnn architecture choice of 10 unique architectures (only for models with enas-rnns)