Learning Architectures from an Extended Search Space for Language Modeling (2020)

Abstract: Neural architecture search (NAS) has advanced significantly in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell. In this paper, we extend the search space of NAS. In particular, we present a general approach to learn both intra-cell and inter-cell architectures (call it ESS). For a better search result, we design a joint learning method to perform intra-cell and inter-cell NAS simultaneously. We implement our model in a differentiable architecture search system. For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and WikiText data, with a new state-of-the-art on PTB. Moreover, the learned architectures show good transferability to other systems. E.g., they improve state-of-the-art systems on the CoNLL and WNUT named entity recognition (NER) tasks and CoNLL chunking task, indicating a promising line of research on large-scale pre-learned architectures.

## introduction neural models have shown remarkable performance improvements in a wide range of natural language processing (nlp) tasks. systems of this kind can broadly be characterized as following a neural network design: we model the problem via a pre-defined neural architecture, and the resulting network is treated as a black-box family of functions for which we find parameters that can generalize well on test data. this paradigm leads to many successful nlp systems based on well-designed architectures. the earliest of these makes use of recurrent neural networks (rnns) for representation learning @xcite , whereas recent systems have successfully incorporated fully attentive models into language generation and understanding @xcite . in designing such models, careful engineering of the architecture plays a key role for the state-ofthe-art though it is in general extremely difficult to find a good network structure. the next obvious step is toward automatic architecture design. a popular method to do this is neural architecture search (nas). in nas, the common practice is that we first define a search space of neural networks, and then find the most promising candidate in the space by some criteria. previous efforts to make nas more accurate have focused on improving search and network evaluation algorithms. but the search space is still restricted to a particular scope of neural networks. for example, most nas methods are applied to learn the topology in a recurrent or convolutional cell, but the connections between cells are still made in a heuristic manner as usual @xcite . note that the organization of these sub-networks remains important as to the nature of architecture design. for example, the first-order connectivity of cells is essential to capture the recurrent dynamics in rnns. more recently, it has been found that additional connections of rnn cells improve lstm models by accessing longer history on language modeling tasks @xcite . similar results appear in transformer systems. dense connections of distant layers help in learning a deep transformer encoder for machine translation @xcite . a natural question that arises is: can we learn the connectivity of sub-networks for better architecture design? in this paper, we address this issue by enlarging the scope of nas and learning connections among h t-1 sub-networks that are designed in either a handcrafted or automatic way (figure 1 ). we call this the extended search space method for nas (or ess for short). here, we choose differentiable architecture search as the basis of this work because it is efficient and gradient-friendly. we present a general model of differentiable architecture search to handle arbitrary search space of nas, which offers a unified framework of describing intra-cell nas and inter-cell nas. also, we develop a joint approach to learning both high-level and low-level connections simultaneously. this enables the interaction between intra-cell nas and inter-cell nas, and thus the ability of learning the full architecture of a neural network. our ess method is simple for implementation. we experiment with it in an rnn-based system for language modeling. on the ptb and wikitext data, it outperforms a strong baseline significantly by 4.5 and 2.4 perplexity scores. moreover, we test the transferability of the learned architecture on other tasks. again, it shows promising improvements on both ner and chunking benchmarks, and yields new state-of-the-art results on ner tasks. this indicates a promising line of research on largescale pre-learned architectures. more interestingly, it is observed that the inter-cell nas is helpful in modeling rare words. for example, it yields a bigger improvement on the rare entity recognition task (wnut) than that on the standard ner task (conll). despite of great success, previous studies restricted themselves to a small search space of neural networks. for example, most nas systems were designed to find an architecture of recurrent or convolutional cell, but the remaining parts of the network are handcrafted @xcite @xcite . for a larger search space, @xcite optimized the normal cell (i.e., the cell that preserves the dimensionality of the input) and reduction cell (i.e., the cell that reduces the spatial dimension) simultaneously and explored a larger region of the space than the singlecell search. but it is still rare to see studies on the issue of search space though it is an important factor to nas. on the other hand, it has been proven that the additional connections between cells help in rnn or transformer-based models @xcite @xcite . these results motivate us to take a step toward the automatic design of inter-cell connections and thus search in a larger space of neural architectures. ## inter-cell and intra-cell nas in this work we use rnns for description. we choose rnns because of their effectiveness at preserving past inputs for sequential data processing tasks. note that although we will restrict ourselves to rnns for our experiments, the method and discussion here can be applied to other types of models. ## joint learning for architecture search our model is flexible. for architecture search, we can run intra-cell nas, or inter-cell nas, or both of them as needed. however, we found that simply joining intra-cell and inter-cell architectures might not be desirable because both methods were restricted to a particular region of the search space, and the simple combination of them could not guarantee the global optimum. this necessitates the inclusion of interactions between intra-cell and inter-cell architectures into the search process. generally, the optimal inter-cell architecture depends on the intra-cell architecture used in search, and vice versa. a simple method that considers this issue is to learn two models in a joint manner. here, we design a joint search method to make use of the interaction between intra-cell nas and inter-cell nas. figure 4 shows the algorithm. it runs for a number of rounds. in each round, we first learn an optimal intra-cell architecture by fixing the inter-cell architecture, and then learn a new inter-cell architecture by fixing the optimal intra-cell architecture that we find just now. obviously, a single run of intra-cell (or inter-cell) nas is a special case of our joint search method. for example, one can turn off the inter-cell nas part (lines 4-5 in figure 4 ) and learn intra-cell architectures solely. in a sense, the joint nas method extends the search space of individual intra-cell (or inter-cell) nas. both intra-cell and inter-cell nas shift to a new region of the parameter space in a new round. this implicitly explores a larger number of underlying models. as shown in our experiments, joint nas learns intra-cell architectures unlike those of the individual intra-cell nas, which leads to better performance in language modeling and other tasks. ## experiments we experimented with our ess method on penn treebank and wikitext language modeling tasks and applied the learned architecture to ner and chunking tasks to test its transferability. ## conclusions we have proposed the extended search space (ess) method of nas. it learns intra-cell and inter-cell architectures simultaneously. moreover, we present a general model of differentiable architecture search to handle the arbitrary search space. meanwhile, the high-level and low-level sub-networks can be learned in a joint fashion. experiments on two language modeling tasks show that ess yields improvements of 4.5 and 2.4 perplexity scores over a strong rnn-based baseline. more interestingly, it is observed that transferring the pre-learned architectures to other tasks also obtains a promising performance improvement.