Faster Discovery of Neural Architectures by Searching for Paths in a Large Model
Anonymous authors
Paper under double-blind review
Abstract
We propose Eﬃcient Neural Architecture Search (ENAS), a faster and less expensive approach to automated model design than previous methods.
In ENAS, a controller learns to discover neural network architectures by searching for an optimal path within a larger model.
The controller is trained with policy gradient to select a path that maximizes the expected reward on the validation set.
Meanwhile the model corresponding to the selected path is trained to minimize the cross entropy loss.
On the Penn
Treebank dataset, ENAS can discover a novel architecture thats achieves a test perplexity of 57 . 8, which is state-of-the-art among automatic model design methods on Penn Treebank.
On the CIFAR-10 dataset, ENAS can design novel architectures that achieve a test error of 2 . 89%, close to the 2 .
65% achieved by standard NAS (Zoph et al., 2017).
Most importantly, our experiments show that ENAS is more than 10x faster and 100x less resource-demanding than NAS.
Introduction
Neural architecture search (NAS) has been applied successfully to design model architectures for image classiﬁcation and language modeling (Zoph & Le, 2017; Baker et al., 2017a;
Bello et al., 2017b; Zoph et al., 2017;
Cai et al., 2017).
NAS however is computationally expensive and time consuming: for example, Zoph et al.
(2017) use 450 GPUs and train for 3days.
Meanwhile, using less resources tends to produce less compelling results (Negrinho &
Gordon, 2017; Baker et al., 2017a).
The main computational bottleneck of NAS is the training of each child model to convergence to measure its accuracy.
We believe that it is very ineﬃcient and wasteful to train every child model and then throw away all the trained weights even though the child models have much in common.
The goal of this work is to remove this ineﬃciency by enabling more sharing between the child models.
This idea is similar to the concept of weight inheritance in neuro-evolution (e.g., Real et al.
To understand our method, we ﬁrst need to understand the standard NAS.
In standard NAS (Zoph & Le, 2017;
Baker et al., 2017a), an RNN controller is trained by policy gradient to search for a good architecture, which is basically a computational graph.
Our observation is that all of the graphs, that NAS has iterated over, can be viewed as sub-graphs of a larger graph.
In other words, we can represent the space of these graphs as a single directed acyclic graph (DAG).
As illustrated in
Figure 1, a neural network architecture can be found by taking a subset of edges in this DAG.
This design is advantageous because it enables sharing parameters among all architectures in the search space.
Our method requires the training of a path within a large model together with the controller.
The controller is trained with policy gradient to select a path that maximizes the expected reward on the validation set.
Meanwhile the model corresponding to the selected path is trained to minimize a cross-entropy loss.
In view of its eﬃciency, we name our method
Eﬃcient Neural Architecture Search (ENAS).
ENAS is both theoretically sound and empirically eﬃcient.
Using Lyapunov functions (Bottou, 1991), we can show that the training of ENAS converges almost surely.
Empirically, on CIFAR-10, ENAS achieves the error rate of 2 . 89%, compared to 2 .
65% by NAS.
On
Penn Treebank, ENAS discovers an architecture that achieves the perplexity of 57 .
8, which outperforms NAS’s perplexity of 62 .
4 (Zoph & Le, 2017).
In all of our experiments, ENAS takes less than 16 hours to train, whilst running on a single Nvidia GTX 1080Ti GPU.
Compared to NAS, our method achieves a 10X reduction of search time and 100X reduction of computational resources.
Related Work
There is growing interest in improving the eﬃciency of neural architecture search.
Concurrent to our work are the promising ideas of using learning curve prediction to skip bad models (Baker et al., 2017b), predicting the accuracies of models before training (Deng et al., 2017), using iterative search method for architectures of growing complexity (Liu et al., 2017a; Elsken et al., 2017), or using hierarchical representation of architectures (Liu et al., 2017b).
Our method is also inspired by the concept of weight inheritance in neuroevolution, which has been demonstrated to have positive eﬀects at scale (Real et al., 2017).
Closely related to our method are other recent approaches that avoid training each architecture from scratch, such as convolutional neural fabrics – ConvFabrics (Saxena & Verbeek, 2016) and SMASH (Brock et al., 2017).
These methods are more computationally eﬃcient than standard NAS.
However, the search space of ConvFabrics is not ﬂexible enough to include novel architectures, e.g. architectures with arbitrary skip connections as in Zoph &
Le (2017).
Meanwhile, SMASH can design interesting architectures but requires a hypernetwork (Ha et al., 2017) to generate the weights, conditional on the architectures.
While a hypernetwork can eﬃciently rank diﬀerent architectures, as shown in the paper, the real performance of each network is diﬀerent from its performance with parameters generated by a hypernetwork.
Such discrepancy in SMASH can cause misleading signals for reinforcement learning.
Even more closely related to our method is PathNet (Fernando et al., 2017), which uses evolution to search for a path inside a large model for transfer learning between
Atari games.
Efficient Neural Architecture Search
In the following, we will ﬁrst present our search space to design recurrent cells for recurrent networks.
We will then explain how to train, and infer with the controller.
Finally, we will explain our search space to design convolutional architectures.
3. Designing Recurrent Cells
To facilitate our discussion of ENAS, we ﬁrst describe how we employ ENAS to design a recurrent cell.
The search space of ENAS is a Directed Acyclic Graph as mentioned in
Section 1.
The DAG has N nodes where the edges represent the ﬂow of information between these N nodes.
Similar to NAS (Zoph & Le, 2017), ENAS has a controller RNN, which decides which edges are activated and which computations are performed at each node.
To create a recurrent cell, the controller RNN samples N blocks of decisions.
Here we illustrate the ENAS mechanism via a simple example for a recurrent cell with N =computational nodes.
Let x t be the input signal for a recurrent cell (e.g., word embedding), and h t − 1 be the output from the previous time step.
The example cell, which we visualize in Figure 2, is sampled as follows.
At node 1: The controller ﬁrst samples an activation function.
In our example in Figure 2, it chooses the tanh activation function, which means that node 1 of the recurrent cell should compute h 1 = tanh ( x t · W ( x ) + h t − 1 · W ( h ) 1 ).
At node 2: The controller then samples a previous index and an activation function.
In our example, it chooses the previous index 1 and the activation function ReLU.
Thus, node 2 of the recurrent cell should compute h 2 = ReLU( h 1 · W ( h ) 2 , 1 ).
At node 3: The controller again samples a previous index and an activation function.
In our example, it chooses the previous index 2 and the activation function ReLU.
Therefore, h 3 = ReLU( h 2 · W ( h ) 3 , 2 ).
At node 4: The controller again samples a previous index and an activation function.
In our example, it chooses the previous index 1 and the activation function tanh, leading to h 4 = tanh ( h 1 · W ( h ) 4 , 1 ).
For the output, we simply average all the loose ends, which are the nodes that are not input to any other nodes.
In our example, since the indices 3 and 4 were never sampled to be the input for any node, the recurrent cell uses their average ( h 3 + h 4 ) / 2 as its output.
In the example above, we note that for each pair of nodes j < ‘ , there is an independent parameter matrix W ( h ) ‘,j .
As shown in the example, the controller decides which parameter matrices are used, by choosing the previous indices.
Therefore, in ENAS, all recurrent cells in a search space share the same set of parameters.
Our search space includes an exponential number of conﬁgurations.
Speciﬁcally, if the recurrent cell has N nodes and we allow 4 activation functions (namely tanh, ReLU, identity, and sigmoid), then the search space has 4 N × N ! conﬁgurations.
In our experiments, N = 12, which means there are approximately 10 15 models in our search space.
3. Training and Inference with ENAS
Our controller network is a two-layer LSTM (Hochreiter & Schmidhuber, 1997) which samples decisions via softmax classiﬁers.
The controller network samples these decisions in an autoregressive fashion: the decision in the previous step is fed as input embedding into the next step.
At the ﬁrst step, the controller network receives an empty embedding as input.
In ENAS, there are two sets of learnable parameters: the parameters of the controller LSTM, denoted by θ , and the shared parameters of the child models, denoted by ω .
The training procedure of ENAS consists of two alternating phases.
The ﬁrst phase trains ω , the shared parameters of the child models, on a whole pass through the training data set, and the second phase trains θ , the parameters of the controller LSTM, for a ﬁxed number of steps.
These two phases are alternated during the training of ENAS.
In our experiments with the
Penn Treebank dataset, for each phase of training ω , we train ω for 450 steps with SGD, each on a batch of 64 examples, where gradients are computed using back-propagation through time, truncated to 35 time steps.
Meanwhile, for each phase of training θ , we train it for 2000 steps with the Adam optimizer and REINFORCE.
More details of their training are as follows.
Training the shared parameters ω of the child models.
In this step, we ﬁx the policy π ( m ; θ ) and perform stochastic gradient descent (SGD) updates on ω to minimize the expected loss function E m ∼ π ( m ; θ )
[ L ( m ; ω )].
Here, L ( m ; ω ) is the standard cross-entropy loss, computed on a minibatch of training data, with a model m sampled from π ( m ; θ ).
The gradient is computed via the Monte Carlo estimate
[Equation]
Note that Eqn 1 provides an unbiased estimate of the gradient ∇ ω
E m ∼ π
( m ; θ )
[ L ( m ; ω )].
Therefore, according to Chapter 3.3.2 of Bottou (1991), with an appropriate learning schedule, the SGD updates of ω converge almost surely.
While convergence is guaranteed, these updates on ω have an inherently larger variance than SGD performed on a ﬁxed model m .
Nevertheless, we ﬁnd that M = 1 works just ﬁne, i.e. we can update ω using the gradient from any single model m sampled from π ( m ; θ ).
As mentioned, we train ω for a whole pass through the training data set.
Training the parameters of θ of the controller LSTM.
In this step, we ﬁx ω and update the policy parameters θ , aiming to maximize the expected reward E m ∼ π ( m ; θ )
[ R ( m , ω )].
We employ the Adam optimizer (Kingma & Ba, 2015), for which the gradient is computed using REINFORCE (Williams, 1992), with a moving average baseline to reduce variance.
We compute the reward R ( m , ω ) on the validation set , rather than the training set to encourage ENAS, to select the model that generalizes well rather than the model that overﬁts the training set well.
In our experiments with language modeling, the reward function is c/ valid ppl, where the perplexity is computed on a minibatch, also sampled from the validation set.
Later, in our experiments with image classiﬁcation, the reward function is the classiﬁcation accuracy on a minibatch of images sampled from the validation set.
Inference.
We discuss how to derive novel architectures from a trained ENAS model.
Following Zoph & Le (2017), we sample several models from the trained policy π ( m , θ ).
For each sampled model, we compute its reward on a single minibatch sampled from the validation set.
We then take the model with the highest reward to re-train from scratch.
3. Designing Convolutional Models
We now discuss the search space for convolutional architectures.
Recall that in the search space of the recurrent cell, the controller RNN samples two decisions at each decision block: 1) what previous node to connect to and 2) what activation function to use.
In the search space for convolutional models, the controller RNN also samples two sets of decisions at each decision block: 1) what previous node to connect to and 2) what computation operation to use.
The decision of what previous node to connect to allows the model to form skip connections (He et al., 2016; Zoph & Le, 2017); whereas the decision of what computation operation to use sets a particular layer into convolution or average pooling or max pooing.
These decisions help construct a layer in the convolutional model.
To be even more ﬂexible, instead of deciding if a particular layer is convolution or average pooling or max pooling, we change the search space to blend all these choices together.
To achieve this, we treat all operations such as convolution, average pooling, and max pooling as channels, and allow the controller to select a mask over these channels.
For example, in our experiments, we allow the controller to choose a mask over how many conv1x1, conv3x3, conv5x5, conv7x7, average pooling, max pooling operations to use (see Figure 3).
Each operation at each layer in our network has its own convolutional parameters.
During each data pass, only the parameters corresponding to the active channels are used and updated.
For the skip connections, at layer k , up to k − 1 mutually distinct previous indices are sampled.
For example, at layer k = 5, let’s suppose the controller samples { 2 , 4 } , then the outputs of layer 2 and layer 4 will be concatenated and sent to layer 5 via skip connections.
In our experiments with the CIFAR-10 dataset, the training of the controller LSTM and the child models are also alternating.
For each phase of training θ , we train it forsteps with the Adam optimizer and REINFORCE.
Meanwhile, for each phase of training the shared parameter ω , we train it for 450 minibatches, each has 100 images, using Nesterov
Momentum (Nesterov, 1983).
3. Designing Convolutional Cells
An alternative to designing the entire convolutional network is to design smaller modules and then ﬁt repeats of them together (Zoph et al., 2017; Zhong et al., 2017).
Figureillustrates this approach, where the convolutional cell and reduction cell architectures are to be designed.
We now discuss how to use ENAS to search for the architecture of these cells. Following Zoph et al. (2017), we sample both our convolutional cell and our reduction cell using an RNN controller. Speciﬁcally, at each decision block, the controller RNN samples for two sets of decisions: 1) two previous nodes to be used as inputs to the current block and 2) two operations to respectively apply to the two sampled nodes.
We allow the followingoperations: identity, separable convolution with kernel size 3x3 and 5x5, and average pooling and max pooling with kernel size 3x3.
Note that each cell receives two input nodes, indexed by node 1 and node 2, corresponding to the outputs of the two previous cells in the entire network.
Figure 5 depicts an example run of our controller in the search space with 4 nodes.
As with our other search spaces, each operation in each cell has its own parameters.
During each data pass, only the relevant parameters are used and trained with their corresponding gradients.
In the following, we will show our experimental results with ENAS to design recurrent cells on the Penn Treebank dataset and to design convolutional architectures on the CIFARdataset.
We then present an ablation study which shows the role of ENAS in discovering novel architectures, as well as details regarding the eﬃciency of ENAS.
4. Language Model with Penn Treebank
Dataset.
We ﬁrst apply ENAS to the task of language modeling, whose goal is predict next words in a text given a history of previous words, by ﬁtting a probabilistic model over sentences.
We use the Penn Treebank dataset (PTB) (Marcus et al., 1994), a well-studied language modeling benchmark.
In this experiment, we show that ENAS discovers a novel recurrent cell which, without extensive hyper-parameters tuning, can outperform models with the same or more parameters.
Training details.
Our controller is trained using Adam (Kingma & Ba, 2015), with a learning rate of 0 . 001.
We use a tanh constant of 2 . 5 and a temperature of 5 . 0 for the sampling logits (Bello et al., 2017a).
We also add the controller’s sample entropy to the reward, with a weight of 10 − 4 .
Additionally, we augment the simple transformations between the constructed recurrent cell’s nodes with highway connections (Zilly et al., 2017).
For instance, instead of having h 2 = ReLU( h 1 · W ( h ) 2 , 1 ) as shown in the example from Section 3.1, we have h [-
-]2 =
[Equation]
⊗ ReLU( h [Equation] W ( h ) 2 , 1 ) +
[Equation] ) ⊗ h 1 , where c 2 = sigmoid( h 1 · W ( c ) 2 , 1 ).
More details can be found in Appendix A. A novel RNN cell found in our search space is shown in Figure 6.
The shared parameters of the child models ω are trained using SGD with a learning rate of 0 . 2, decayed by a factor of 0 . 9 after every 3 epochs starting at epoch 15, for a total of 150 epochs.
During the architecture search process, following Melis et al.
(2017), we randomly reset the starting state with probability of 0 .
We also tie the model’s word embeddings matrix with its softmax matrix (Inan et al., 2017).
When retraining the architecture recommended by the controller, however, we use variational dropout (Gal & Ghahramani, 2016), an ‘ 2 regularization with weight decay of 10 − 7 , and a state slowness regularization of 0 .
(Merity et al., 2017).
Results.
Running on a single Nvidia GTX 1080Ti GPU, ENAS ﬁnds the recurrent cell in less than 10 hours.
This cell is depicted in Figure 6.
Table 1 presents our results in comparison with other methods.
The ENAS cell, with 24M parameters, outperforms the NAS cell and has a similar performance to the LSTM model that uses extensive hyperparameters tuning (Melis et al., 2017), which we did not do.
Our ENAS cell has a few interesting properties.
First, all non-linearities in the cell are either ReLU or tanh, even though the search space also has two other functions: identity and sigmoid.
We suspect this cell is a local optimum, similar to the observations by Zoph &
Le (2017).
When we randomly pick some nodes and switch the non-linearity into identity or sigmoid, the perplexity increases up to 8 points.
When we randomly, swith some ReLU nodes into tanh or vice versa, the perplexity also increases, but only up to 3 points. Method [Table]
LSTM+Vanilla Dropout (Zaremba et al., 2014)
[Table]
LSTM+VD+MC (Gal & Ghahramani, 2016)
[Table]
LSTM+WT (Inan et al., 2017)
[Table]
Recurrent Highway Network (Zilly et al., 2017)
[Table]
LSTM+Hyper-parameters Search (Melis et al., 2017)
[Table]
4. Image Classification on CIFAR-
Dataset.
The CIFAR-10 dataset (Krizhevsky, 2009) consists of 50,000 training images and 10,000 test images.
We use the standard data pre-processing and augmentation techniques, i.e., subtracting the mean and dividing the standard deviation from each channel computed on the training images, centrally padding the training images to 40 × 40 and randomly cropping them back to 32 × 32, and randomly ﬂipping them horizontally.
Search spaces.
In Section 3.3, we present a search space for convolutional architectures in which the controller can make decisions over skip connections and the mask over the channels.
To improve our results, we additionally explore two restricted versions of this search space: one where the controller only needs to make decisions of the mask over the channels and one where the controller only needs to make decisions of the skip connections.
More details are available in Appendix B.
Searching for the masks over channels.
We ﬁx a pattern of skip connections and search for the masks at each branch and each layer in a 12-layer network.
The pattern that we use is the dense pattern (Huang et al., 2016).
Searching for skip connections.
We force all the convolutions to have the ﬁlter size of 3 × 3, and only search for the skip connections. Searching for convolutional and reduction cells. We search
for both
cells as discussed in Section 3.4.
Training details.
The shared parameters ω are trained with Nesterov momentum (Nesterov, 1983), where the learning rate follows the cosine schedule with l max = 0 .
05, l min = 0 .
001, T 0 = 10 and T mul =
(Loshchilov & Hutter, 2017).
Each architecture search is run for 10+20+40+80+160 = 310 epochs.
The parameter ω is initialized from a scaled Gaussian as described in He et al.
We also apply an ‘ 2 weight decay of 10 − 4 .
The same settings are employed to train the architectures recommended by the controller.
The policy parameters θ are initialized uniformly in [ − 0 . 1 , 0 . 1], and trained with the Adam optimizer at a learning rate of 10 − 3 .
We additionally utilize three techniques to prevent the premature convergence of REINFORCE.
First, we apply a temperature τ = 5 .
and a tanh constant c = 2 .
5 to the controller’s logits.
Second, we add to the reward the entropy term of the controller’s samples weighted by λ ent = 0 .
1, which discourages convergence (Williams &
Peng, 1991).
Lastly, we enforce the sparsity in the skip connections by adding to the reward the Kullback-Leibler divergence between: 1) the skip connection probability between any two layers and 2) our chosen probability ρ = 0 . 4, which represents the prior belief of a skip connection being formed.
The KL divergence term is weighted by λ kl = 0 .
Tricks to stabilize and improve training.
We ﬁnd the following tricks crucial for achieving good performance with ENAS.
• Structure of Convolutional Layers.
Each convolutional operation in our method is followed by a batch normalization (Ioﬀe & Szegedy, 2015) and then a ReLU layer.
We ﬁnd the alternate setting of batch norm-conv-ReLU (Zoph et al., 2017) to have worse results.
• Stabilizing Stochastic Skip Connections.
If a layer receives skip connections from multiple layers before it, then these layers’ outputs are concatenated in their depth dimension, and then a convolution of ﬁlter size 1 × 1 (followed by a batch normalization layer and a ReLU layer) is performed to ensure the number of output channels is still equal to C .
• Global Average Pooling.
After the ﬁnal convolutional layer, we average all the activations of each channel and then pass them to the Softmax layer.
This trick was introduced by Lin et al.
(2013), with the purpose of reducing the number of parameters in the dense connection to the Softmax layer to avoid overﬁtting.
The last two tricks are extremely important, since the gradient updates of the shared parameters ω , as described in Eqn 1, have a very high variance.
In fact, we ﬁnd that without these last two tricks, the training of ENAS is very unstable.
Results.
Table 2 summarizes the test errors of ENAS and other approaches.
As can be seen from the table, ENAS successfully found several architectures that outperform other automatic model designing approaches with the same usage of computing resource.
In particular, in the general search space, ENAS takes 15 .
6 hours to ﬁnd a model that achieves 4 .
23% error rate on CIFAR-10.
This model outperforms all but one model reported by Zoph & Le (2017), while taking 30x less time and using 800x less computing resource to discover.
In the restricted search space over the masks, ENAS takes 11 .
6 hours to ﬁnd a model that achieves the test error of 4 .
The resulting model, depicted in Figure 7- Top Left , almost always has 64 or 96 channels at each branch and each layer, indicating that the controller does not choose to activate all blocks.
This is the desired behavior, as activating all channels would over-parametrize the model and result in overﬁtting.
Moreover, the fact that the model found in a restricted search space has similar performance to the model found in the general search space indicates that ENAS can discover skip connection patterns that are comparable to the dense pattern, which is the state-of-the-art human-designed architecture on CIFAR-10 (Huang et al., 2016).
In the restricted search space over skip connections, ENAS takes 12 .
4 hours to discover the pattern of skip connections depicted in Figure 7- Top Right .
This pattern has the property [Table] the state-of-the-art models, all of which are designed by human experts. The second block presents various approaches that design the entire network.
ENAS outperforms all these methods but NAS, which requires much more computing resource and time.
The last block presents techniques that design modular cells which are used to build a large model.
ENAS outperforms MicroNAS, which uses 32 GPUs to search, and achieves similar performance with NASNet-A. that skip connections are formed much more densely at higher layers than at lower layers, where most connections are only between consecutive layers.
The model has the test error of 5 . 04%, which is slightly worse than the one found in the restricted search space over masks.
However, if we increase the number of output channels from 256 to 512, then the network achieves the test error of 3 .
In the search space over cells, ENAS takes 11 .
5 hours to discover the convolution cell and the reduction cell, which are shown in Figure 7- Bottom .
With the convolutional cell replicated for 6 times, ENAS achieves 3 .
54% test error, which is on par with the 3 .
41% error of NASNetA (Zoph et al., 2017). With CutOut (DeVries & Taylor, 2017), ENAS’s error decreases to 2 . 89%, compared to 2 . 65% by NASNet-A.
However, as discussed in Appendix B, our ENAS cells are trained for only 310 epochs, compared to 600 epochs by NAS.
4. The Importance and Efficiency of ENAS
Sanity Check and Ablation Study.
To understand the role of ENAS, we carry out two control experiments.
In the ﬁrst study, we uniformly at random pick a conﬁguration of channels and skip connections and just train a model.
As a result, about half of the channels and skip connections are selected, resulting in a model with 47 .
1M parameters and an error rate of 5 . 86%.
This error rate is signiﬁcantly worse than the models designed by ENASand has many more parameters.
In the second study, we only train ω and do not update the controller.
The eﬀect is similar to dropout with a rate of 0 .
5 on both the channels and the skip connections.
At convergence, the model has the error rate of 11 .
On the validation set, the ensemble of 250 Monte Carlo conﬁgurations of the trained model could only reach 8 . 99% test error rate.
We therefore conclude that the appropriate training of the ENAS controller is crucial for good performance.
Eﬃciency of ENAS.
Table 3 further compares of the time and resources needed for
ENAS to discover good architectures on CIFAR-10 with other methods.
In all of our experiments, ENAS takes less than 15 hours to train, whilst running on a single Nvidia GTX 1080Ti GPU.
Compared to NAS by Zoph & Le (2017); Zoph et al.
(2017), our method achieves a 10X reduction of search time and 100X reduction of computational resources. [Table]
Conclusion
Neural Architecture Search (NAS) is an important advance that allows faster architecture design for neural networks.
However, the computational expense of NAS prevents it from being widely adopted. In this paper
[Table] we presented ENAS [Table] an alternative method to NAS, that requires three orders of magnitude less resources × time.
The key insight of our method is to share parameters across child models during architecture search.
This insight is implemented by having NAS search for a path within a larger model.
We demonstrate empirically that the method works well on both CIFAR-10 and Penn Treebank datasets [Table]
Appendices A Details on Parameter Sharing in an RNN Cell
We think of the cell at time step t as a block of N computational nodes, indexed by h ( t ) 1 , h ( t ) 2 , ...
h ( t ) N .
Node h ( t ) 1 receives two inputs: (1) the RNN signal x ( t ) at its current time step and (2) the outputs h ( t − 1) D of the cell from the previous time step.
Then, the following computations are performed:
[Equation]
where f 1 is an activation function that the controller will decide.
For ‘ = 2 , 3 , ..., N , node h ‘ receives its input from a layer j ‘ ∈ { h 1 , ..., h ‘ − 1 } , which is speciﬁed by the controller, and then performs the following computations:
[Equation]
The shared parameters ω between diﬀerent recurrent cells thus consist of all the matrices
W ( x , c ) , W ( x , h ) , W ( c ) ‘,j , and W ( h ) ‘,j .
The controller decides the connection j
‘ and the activation function f ‘ , for each ‘ ∈ { 2 , 3 , ..., N } .
The layers that are never selected by any subsequent layers are averaged and sent to a softmax head, or to higher recurrent layers.
As in the case of convolutional models, to stabilize the training of ω , we add a batch normalization layer after the average of the layers that are not selected.
B Details for CIFAR-10 Search Spaces B. Details on Search Space 1: Channels
We use a block size of S = 32, resulting in C/S = 256 / 32 = 8 blocks per branch per layer.
Each branch conﬁguration has its own embedding and softmax head.
To elaborate, this means that a time step in the controller RNN that predicts the conﬁguration for any branch should have a softmax matrix of size
H ×
(2 C/S − 1), where H = 64 is the hidden dimension of the RNN, and 2 C/S − 1 = 255 is the number of possible binary masks for that branch.
Each branch also has an embedding matrix of size (2 C/S − 1) × H , from which the row corresponding to the sampled binary mask is selected and sent to the next time step.
Layers 4 and 8 of our 12-layer network are max pooling layers with a kernel size of 2 ×and a stride of 2, and reduce each spatial dimension of the layers’ outputs by a factor of 2.
Within each group of 3 layers where the spatial dimensions of the layers remain constant, we connect each layer to all layers before it (Huang et al., 2016).
B. Details on Search Space 2: Connections
We use 3 × 3 convolutions with 48 output channels at all layers.
The controller RNN for this search space has the same form as the controller RNN that is depicted in Figure 3, with two modiﬁcations.
First, each block has only 2 time steps (as opposed to 7 in the general search space): (1) the time step that predicts the mask for the convolution of ﬁlter size 3 × 3, which we force to always activate all channels; and (2) the anchor time step, which we allow to sample multiple previous indices.
Second, our controller is thus allowed to form skip connections between arbitrary layers, but forming such connections between layers with diﬀerent spatial dimensions would result in compilation failures.
To circumvent, after each max pooling in the network, we centrally pad the output so that its spatial dimensions remain unchanged.
B. Details on Search Space 3: Convolution and Reduction Cells
We perform a 3 × 3 convolution on the image to create the outputs for the ﬁrst convolutional cell.
After that, in the ﬁrst 6 convolution cells, each separable convolution has 32 output channels.
Each reduction cell is applied in the same way as the convolutional cell, with the only modiﬁcation being each operation is applied with a stride of 2.
When the cells are found, the ﬁnal models (both with and without CutOut) are trained for 310 epochs, using the cosine learning schedule, where the reset cycle is originally set to 10 and then doubled after each reset.
Additionally, we insert an auxiliary head at the layer immediately before the second application of the reduction cell (Szegedy et al., 2016).