In this paper , we study differentiable neural architecture search ( NAS ) methods for natural language processing . In particular , we improve differentiable architecture search by removing the softmax-local constraint . Also , we apply differentiable NAS to named entity recognition ( NER ) . It is the first time that differentiable NAS methods are adopted in NLP tasks other than language modeling . On both the PTB language modeling and CoNLL-2003 English NER data , our method outperforms strong baselines . It achieves a new state-ofthe-art on the NER task . Neural architecture search ( NAS ) has become popular recently in machine learning for their ability to find new models and to free researchers from the hard work of designing network architectures . The earliest of these approaches use reinforcement learning ( RL ) to learn promising architectures in a discrete space ( Zoph and Le , 2016 ) , whereas others have successfully modeled the problem in a continuous manner ( Liu et al . , 2019 ; Xie et al . , 2019b ; Huang and Xiang , 2019 ) . As an instance of the latter , differentiable architecture search ( DARTS ) employs continuous relaxation to architecture representation and makes gradient descent straightforwardly applicable to search . This leads to an efficient search process that is orders of magnitude faster than the RL-based counterparts . Like recent methods in NAS ( Xie and Yuille , 2017 ; Zoph and Le , 2016 ; Baker et al . , 2016 ) , DARTS represents networks as a directed acyclic graph for a given computation cell ( see Figure 1(a ) ) . An edge between nodes performs a predefined operation to transform the input ( i.e. , tail ) Figure 1 : An overview of DARTS cell and our cell to the output ( i.e. , head ) . For a continuous network space , DARTS uses the softmax trick to relax the categorical choice of edges to soft decisions . Then , one can optimize over the graph using standard gradient descent methods . The optimized network is inferred by choosing the edges with maximum weights in softmax . However , DARTS is a " local " model because the softmax-based relaxation is imposed on each bundle of edges between two nodes . This leads to a biased model in that edges coming from different nodes are not comparable . Such a constraint limits the inference space to sub-graphs with one edge between each pair of nodes . Also , the learned network might be redundant because every node has to receive edges from all predecessors no matter they are necessary or not . This problem is similar to the bias problem in other graph-based models where local decisions make the model nonoptimal ( Lafferty et al . , 2001 ; Daphne Koller and Nir Friedman , 2009 ) . Here we present an improvement of DARTS , called I-DARTS , that further relaxes the softmaxlocal constraint . The idea is simple -we consider all incoming edges to a given node in a single softmax . This offers a broader choice of edges and enlarges the space we infer the network from . For example , one can simultaneously select multiple important edges between two nodes and leave some node pairs unlinked ( see Figure 1(b ) ) . I-DARTS outperforms strong baselines on the PTB language modeling and CoNLL named entity recognition ( NER ) tasks . This gives a new stateof-the-art on the NER dataset . To our knowledge , it is the first time to apply differentiable architecture search methods to NLP tasks other than language modeling . More interestingly , we observe that our method is 1.4X faster than DARTS for convergence of architecture search . Also , we provide the architectures learned by I-DARTS , which can be referred for related tasks . We improved the DARTS to address the bias problem by removing the softmax-local constraint . Our method is search efficient and discovers several better architectures for PTB language modeling and CoNLL named entity recognition ( NER ) tasks . We plan to consider the network density problem in search and apply I-DARTS to more tasks in our future study .

Challenge: Softmax-based relaxation used for differentiable architecture search makes edges coming from different nodes incomparable limiting the inference space to sub-graphs.
Approach: They propose removing the softmax constraint from the existing differentiable architecture search method to enable a broader choice of edges and enlarge the search space.
Outcome: The proposed method outperforms an existing method on the CoNLL NER dataset and also achieves 1.4 times faster convergence time.