S IMPLE AND EFFICIENT ARCHITECTURE SEARCH FOR C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural networks have had a lot of success in recent years for variety of tasks.
However, designing a well performing neural network architecture is still an exhausting trial-and-error process done by experts.
We propose a new method for automatically searching for well performing CNN architectures based on a simple hill climbing procedure, network morphisms, and an anytime performance optimization stemming from a cosine based learning rate scheduler.
Our method yields competitive results, whereas the resource requirements lie - in contrast to others - in the same order of magnitude as training a single network.
On CIFAR10, our method designs and trains networks with an error rate below 6% in only 12 hours on a single GPU.
After one day the error is reduced to almost 5%.
I NTRODUCTION
Neural networks have become more and more popular over the last few years due to their success in a variety of tasks such as image recognition (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012) or machine translation (Bahdanau et al., 2015).
Designing these neural networks is still done by hand in most cases, which is an exhausting, time-consuming process.
Additionally, the vast amount of conﬁgurations when designing a neural network requires an expert who can restrict the search to a subset of conﬁgurations.
For these reasons it seems natural to apply some optimization algorithm which automatically searches for well-performing neural network architectures.
However, most classic optimization algorithms fail to solve this problem, mainly due to two reasons: Firstly, evaluating an architecture is costly as it requires training the network, i.e., methods that rely on training a vast amount of networks are impractical for most applicants/applications.
Secondly, the architecture search space is discrete (e.g., number of layer, type of layer) and conditional (e.g., the number of parameters deﬁning a layer depends on the layer type).
Thus methods that rely on, e.g., differentiability or independent parameters are not applicable.
This led to a growing interest in using evolutionary algorithms (Real et al., 2017; Suganuma et al., 2017) or reinforcement learning (Baker et al., 2016;
Cai et al., 2017; Zoph & Le, 2017) for automatically designing CNN architectures.
Unfortunately most proposed methods are either very costly (training for days with hundreds of GPUs) or yield non-competitive performance.
In this work, we aim to dramatically reduce these computational costs while still achieving competitive performance.
Our contributions are as follows:
• We randomly generate networks and evaluate their performance.
In our experiments, they achieve 6%-7% test error on CIFAR-10 when trained with SGDR (Loshchilov & Hutter, 2017).
Future architecture search methods should considerably decrease error rates compared to this baseline, which is not the case at the moment for some methods.
This also indicates that CIFAR-10 may be too easy and hopefully encourages researchers to consider other data sets in the future.
• We formalize and extend the work on network morphisms (Chen et al., 2015; Wei et al., 2016;
Cai et al., 2017) in order to support popular network building blocks such as skip connections or Batch Normalization.
• We show that architecture search does not have to be expensive - our method ﬁnds competitive architecture, whereas the computational costs lie in the same order of magnitude as training a single network.
In more detail, the cost reduction is achieved by exploiting network morphisms/transformations (Chen et al., 2015; Wei et al., 2016) as well as exploiting the good anytime-performance learning rate scheduler from Loshchilov & Hutter (2017).
We show that these two ideas incorporated in a simple hill climbing algorithm yield similar performance to other proposed methods.
On CIFAR-10, our method ﬁnds and trains CNNs with an error rate below 6 % in roughly 12 hours on a single GPU.
After one day the error is reduced to almost 5%.
Models from different stages of our algorithm can be combined to achieve an error of 4.7 % within two days on a single GPU.
On CIFAR-100, we achieve an error below 24% in one day and get close to 20% after two days.
• Our method is easy to use and easy to extend, so it hopefully can serve as a basis for future work.
We discuss related work in Section 2.
The concept of network morphisms is formalized in Section 3.1.
Our method is then proposed in Section 3.2. and evaluated in Section 4.
We conclude our work in Section 5.
R ELATED WORK
Automated architecture search: Due to the success of neural networks, there has been a growing interest in automated architecture search during the last years.
One popular approach is to train a reinforcement learning agent with the objective of designing well-performing convolutional neural networks (Baker et al., 2016; Zoph & Le, 2017; Cai et al., 2017).
In Baker et al.
(2016), a RL agent is trained to sequentially choose the type of layers (convolution, pooling, fully connected) and their parameters.
The RL agent from Zoph & Le (2017) also sequentially generates a string representing the network architecture.
In both works, the generated networks are then trained and their performance is evaluated on a validation set, which is then used as a reward signal to update the RL agent.
In a follow-up work (Zoph et al., 2017), a RL agent learned to build cells, which are then used as building blocks for a neural network.
The network itself however had a ﬁxed architecture (given the cells).
Unfortunately the training a RL agents with the objective of designing architecture is quite expensive as it requires training a vast amount of neural networks: Baker et al.
(2016)’s experiment on CIFAR-10 required 10 GPUs for 8-10 days.
Zoph & Le (2017) did their computations on 800 GPUs.
To overcome this drawback, Cai et al.
(2017) proposed to apply the concept of network transformations/ network morphisms.
The basic idea is to generate new networks based on already trained ones with the help of these transformation in order to avoid training all networks from scratch.
Their experiments on CIFAR-10 were conducted on 5 GPUs
(2017) and Suganuma et al.
(2017) utilized evolutionary algorithms to generate powerful networks iteratively from a small network.
Operations like inserting a layer, modifying the parameters of a layer or adding skip connections serve as ”mutations” in their framework of evolution.
Whereas the computational resources spent in Real et al.
(2017) were also enormous (250 GPUs, 10 days), Suganuma et al.
(2017) were restricted to relatively small networks due to handling a population of networks.
In contrast to the previous methods where network capacity increases over time, Saxena & Verbeek (2016) start with training a large network.
In fact, they train an ensemble of neural networks, dubbed ”convolution neural fabric”.
As they use weight sharing across networks within their fabric, training cost scales linearly with the fabric size though the number of networks within the fabric grows exponentially.
In the very end, the fabric is pruned; networks with almost just zero weights are thrown out of the fabric.
Very recently, Brock et al.
(2017) used hypernetworks (Ha et al., 2017) to generate the weights for a randomly sampled network architecture with the goal of eliminating the costly process of training a vast amount of networks.
However, their approaches requires training the hyper network, which is in their case a 26 layer DenseNet (Huang et al., 2016).
Network morphism/ transformation: Network transformations were (to our knowledge) ﬁrst introduced by Chen et al.
(2015) in the context of transfer learning.
The authors described a function preserving operation to make a network deeper (dubbed ”Net2Deeper”) or wider (”Net2Wider”) with the goal of speeding up training and exploring network architectures.
Wei et al.
(2016) proposed additional operations, e.g., for handling non-idempotent activation functions or altering the kernel size and introduced the term network morphism.
As mentioned above, Cai et al.
(2017) used network morphisms for architecture search, though they just employ the Net2Deeper and Net2Wider operators from Chen et al.
(2015) as well as altering the kernel size, i.e., they limit their search space to simple architectures without, e.g., skip connections.
M ETHODS
In what follows, we formalize the concept of network morphisms, which is afterwards incorporated in a hill climbing algorithm for architecture search.
3. N ETWORK MORPHISM
Let N ( X ) denote a set of neural networks deﬁned on X ⊂ R n .
A network morphism is a mapping
M : N ( X ) × R k → N ( X ) × R j from a neural network f w ∈ N ( X ) with parameters w ∈ R k to another neural network g ˜ w ∈ N ( X ) with parameters ˜ w ∈ R j so that
[Equation]
for every x ∈ X .
[Equation]
In the following we give a few examples of network morphisms and how standard operations for building neural networks (e.g., adding a convolutional layer) can be expressed as a network morphism.
For this, let f w i
i
( x ) be some part of a NN f w ( x ) , e.g., a layer or a subnetwork.
Network morphism Type I.
We replace f w i i by
[Equation]
with ˜ w i
=
( w i , A, b ) 1 .
Equation (1) obviously holds for A = 1 , b = 0 .
This morphism can be used to add a fully-connected or convolutional layer, as these layers are simply linear mappings.
Chen et al.
dubbed this morphism ”Net2DeeperNet”.
Alternatively to the above replacement, one could also choose
[Equation]
with ˜ w i
=
( w i , C, d ) .
A, b are ﬁxed, non-learnable.
In this case network morphism Equation (1) holds if C =
A − 1 , d = − Cb .
A Batch Normalization layer (or other normalization layers) can be written in the above form: A, b represent the batch statistics and C, d the learnable scaling and shifting.
Network morphism Type II.
Assume
f
w
i
i has the form
f w i
i ( x ) =
Ah w h ( x ) + b for an arbitrary function h .
We replace f w i
i , w i = ( w h , A, b ) , by
[Equation]
with an arbitrary function ˜
h w ˜ h
( x ) .
The new parameters are ˜ w i =
( w i , w ˜ h , ˜ A ) .
Again, Equation (1) can trivially be satisﬁed by setting ˜ A = 0 .
We think of two modiﬁcations of a NN which can be expressed by this morphism.
Firstly, a layer can be widened (i.e., increasing the number of units in a fully connected layer or the number of channels in a CNN - the Net2WiderNet transformation from Chen et al. (2015)).
Think of h ( x ) as the layer to be widened.
For example, we can then set ˜ h
= h to simply double the width.
Secondly, skip-connections by concatenation as used by Huang et al.
(2016) can be formulated as a network morphism.
If h ( x ) itself is a sequence of layers, h ( x ) = h n ( x ) ◦ · · · ◦ h 0 ( x ) , then one could choose ˜ h
( x ) = x to realize a skip from h 0 to the layer subsequent to h n .
Network morphism Type III.
By deﬁnition, every idempotent function f w i
i can simply be replaced by
[Equation]
(5) with the initialization ˜ w i = w i .
This trivially also holds for idempotent function without weights, e.g., Relu.
Network morphism Type IV.
Every layer f
w i i is replaceable by
[Equation]
with an arbitrary function h and Equation (1) holds if λ is initialized as 1 .
This morphism can be used to incorporate any function, especially any non-linearities.
For example, Wei et al.
(2016) use a special case of this operator to deal with non-linear, non-idempotent activation functions.
Another example would be the insertion of an additive skip connection, which were proposed by He et al.
(2016) to simplify training: If f w i i itself is a sequence of layers, f w i i = f w in i n ◦ · · · ◦ f w i 0 i 0 , then one could choose h ( x ) = x to realize a skip from f w i
ito the layer subsequent to f w in i n .
Note that every combinations of the network morphisms again yields a morphism.
So one could for example insert a block ”Conv-BatchNorm-Relu” subsequent to a Relu layer by using equations (2), (3) and (5).
3. A RCHITECTURE S EARCH BY N ETWORK MORPHISMS
Our proposed algorithm is a simple hill climbing strategy (Russell & Norvig, 2009): we start with a small, (possibly) pretrained network.
Afterwards we apply network morphisms to this initial network to generate larger, possibly better performing one when further trained.
These new networks can be seen as neighbors of the initial network in the space of network architectures.
Due to the network morphism Equation (1), the new networks start at the same performance as the old one.
In essence, network morphisms can be seen as a way to initialize new networks at a high performing level and we do not have to train them from scratch.
They can then be further trained to exploit the additional capacity obtained by the network morphism and eventually improve.
Note that in principle it could also be possible that the new network is stuck in a poor performing region, which it can never escape.
We are going to investigate whether this is true in our experiment section later on.
The described procedure can be repeated a few times until performance on a validation set saturates.
We summarize the proposed method in Algorithm 1.
In our implementation, the function
ApplyNetMorph ( model, n ) (line 15) applies n times one of three operations, uniformly sampled:
• Make the network deeper, i.e., add a ”Conv-BatchNorm-Relu” block as described in the end of Section 3.1.
The position where to add the block as well as the kernel size ( ∈ { 3 , 5 } ) are uniformly sampled.
The number of channels is chosen to be equal to he number of channels of the closest preceding convolution.
• Make the network wider, i.e., increase the number of channels by using the network morphism type II.
The conv layer to be widened is uniformly sampled as well as the widening factor ( ∈ { 2 , 4 } ).
• Add a skip connection from layer i to layer j
(either by concatenation or addition - uniformly sampled) by using the network morphism type II or IV, respectively.
i,j are also uniformly sampled.
Note that the current best model is also considered as a neighbor, i.e. our algorithm is not forced to select a new model but rather can also keep the old one.
It is important for our method that the performance of models trained for just a few epochs 2 (line 17) correlates with its performance when trained to convergence.
Hence, an optimization algorithm with good anytime performance is required.
We chose the cosine annealing strategy from Loshchilov &
Hutter (2017), whereas the learning rate is implicitly restarted: the training in line 17 always starts with a learning rate λ start which is annealed to λ end after epoch neigh epochs.
We use the same learning rate scheduler in the ﬁnal training (aside from a different number of epochs).
Our algorithm could also be interpreted as a very simple evolutionary algorithm with a population size of n neigh and network morphisms as mutations.
1: function NASH( model 0 , n steps , n neigh , n NM , epoch neigh , epoch final , λ end , λ start ) 2: 3: # model 0 (cid:44) model to start with , n steps (cid:44) number of hill climbining steps 4:
# n neigh (cid:44) number of neighbours , n NM (cid:44) number of net.
morph.
applied 5: # epoch neigh (cid:44) number of epochs for training every neighbour 6: # epoch final (cid:44) number of epochs for training 7: # initial LR λ start is annealed to λ end during SGDR training 8: 9: # set current best model to start model 10: model best ← model11: # start hill climbing 12: for i ← 1 , . . .
, n steps do 13: #get n neigh neighbors of model 0 by applying n NM network morphisms to model best 14: for j ← 1 , . . .
, n neigh − 1 do 15: model j ← ApplyNetMorphs ( model best , n NM ) 16: # train for a few epochs on training set with SGDR 17: model j ← SGDRtrain ( model j , epoch neigh , λ start , λ end ) 18: end for 19: # in fact, last neighbor is always just the current best 20: model n neigh ← SGDRtrain ( model best , epoch neigh , λ start , λ end ) 21: # get best model on validation set 22: model best ← arg max j =1 ,...,n neigh { performance vali ( model j ) } 23: end for 24: # train the ﬁnal model on training and validation set 25: model best ← SGDRtrain ( model best , epoch final , λ start , λ end ) 26: return model best 27: end function
E XPERIMENTS
We evaluate our method on CIFAR-10 and CIFAR-100.
First, we investigate whether our considerations from the previous chapter coincide with empirical results.
We also check if the interplay of [Table] modifying and training networks harms their eventual performance.
Finally, we compare our proposed method with other automated architecture algorithms as well as hand crafted architectures.
We use the standard data augmentation scheme for both CIFAR datasets as described by Loshchilov & Hutter (2017) in all of the following experiments.
The training set (50.000 samples) is split up in training (40.000) and validation (10.000) set for the purpose of architecture search.
Eventually the performance is evaluated on the test set.
All experiments where run on Nvidia Titan X (Maxwell) GPUs.
Code is implemented in Keras (Chollet et al., 2015) with TensorFlow (Abadi et al., 2015) backend.
4. E XPERIMENTS ON CIFAR4.1. B ASELINES
Before comparing our method to others, we ﬁrst run some baseline experiments to see whether our considerations from the previous chapter coincide with empirical data.
First, we investigate whether the simple hill climbing strategy is able to distinguish between good and bad-performing models.
For this, we run our algorithm with n neigh = 8 as well as n neigh = 1 , i.e., there is no model selection and we simply generate a single, random network and train it.
All other parameters are the same in this experiment, namely n steps = 5 , n NM = 5 , epoch neigh = 17 , epoch final = 100 .
We choose λ start = 0 .
05 , λ end = 0 .
as done in Loshchilov & Hutter (2017).
model 0 was a simple conv net: Conv-MaxPool-Conv-MaxPool-Conv-FC-Softmax 3 , which is pretrained for 20 epochs, achieving ≈ 75% validation accuracy (up to 91% when trained till convergence), see Figure 3 in the appendix.
If our algorithm is able to identify better networks, one would expect to get better results with the setting n neigh = 8 .
As a second experiment we turn off the cosine annealing with restarts (SGDR) during the hill climbing stage, i.e., the training in line 17 of Algorithm 1 is done with a constant learning rate, namely the starting learning rate of SGDR λ start = 0 .
Note that we still use use the cosine decay for the ﬁnal training.
The results for these two experiments are summarized in Table 1 .
The hill climbing strategy actually is able to identify better performing models ( 5 . 7% vs. 6 . 5% ).
Notice how hill climbing prefers larger models ( 5 . 7 million parameters on average vs. 4 . 4 million).
Also, SGDR plays an important role.
The resulting models chosen by our algorithm when training is done with a constant learning rate perform similarly to the models without any model selection strategy ( 6 . 8% and 6 .
5% , respectively ) , which indicates that the performance after a few epochs on the validation set when trained without SGDR correlates less with the ﬁnal performance on the test set as it is the case for training with SGDR.
With the constant learning rate, the few epochs spent are not sufﬁcient to improve the performance of the model.
Figure 2 shows the progress while running our algorithm for two (randomly picked) examples - one when trained with and one when trained without SGDR.
4.1. R ETRAINING FROM SCRATCH .
In the this experiment we investigate whether the ”weight inheritance” due to the network morphisms used in our algorithm harms the ﬁnal performance of the ﬁnal model.
This weight inheritance can be seen as a strong prior on the weights and one could suspect that the new, larger model may not be able to overcome a possibly poor prior.
Additionally we were interested in measuring the overhead of the architecture search process, so we compared the times for generating and training a model with the time needed when training the ﬁnal model from scratch.
The retraining from scratch is done for the same number of epochs as the total number of epochs spent to train the model returned [Table] by our algorithm 4 .
We list our ﬁndings in Table 2.
The only major difference can be observed for model 3.
In all other cases
[Table] the difference is below 0.5% - sometimes in favor of our algorithm [Table] sometimes in favor of the retraining from scratch.
This experiments indicates that our algorithm does not harm the ﬁnal performance of a model.
Regarding the runtime, the overhead for ﬁrst having to search the architecture is roughly a factor 3.
We think this is a big advantage of our method and shows that architecture search can be done in the same order of magnitude as a single training
[Table]
4.1. C OMPARISON TO HAND CRAFTED AND OTHER AUTOMATICALLY GENERATED ARCHITECTURES
We now compare our algorithm against the popular wide residual networks (Zagoruyko & Komodakis, 2016), the state of the art model from Gastaldi (2017) as well as other automated architecture search methods.
Beside our results for n steps = 5 from the previous section, we also tried n steps = 8 to generate larger models.
For further improving the results [Table] we take snapshots of the best models from every iteration while running our algorithm following the idea of Huang et al.
(2017) when using SGDR (Loshchilov & Hutter, 2017) for training.
However different from Huang et al.
(2017), we do not immediately get fully trained models for free, as our snapshots are not yet trained on the validation set but rather only on the training set.
Hence we spent some additional resources and train the snapshots on both sets. [Table] [Table] neigh
[Table]
(snapshot ensemble
[Table] (ensemble across runs) [Table] Afterwards the ensemble model is build by combining the snapshot models with uniform weights.
Lastly, we also build an ensemble from the models returned by our algorithm across all runs.
Results are listed in Table 3.
The proposed method is able to generate competitive network architectures in only 12 hours.
By spending another 12 hours, it outperforms most automated architecture search methods although all of them require (partially far) more time and GPUs.
We do not reach the performance of the two handcrafted architectures as well as the ones found by Zoph & Le (2017) and Brock et al.
However note that Zoph & Le (2017) spent by far more resources than we did.
Unsurprisingly, the ensemble models perform better.
It is a simple and cheap way to improve results which everyone can consider when the number of parameters is not relevant.
4. E XPERIMENTS ON CIFAR-
We repeat the previous experiment on CIFAR-100; hyperparameters were not changed.
The results are listed in Table 4.
Unfortunately most automated architecture methods did not consider CIFAR100.
Our method is on a par with Real et al.
(2017) after one day with a single GPU.
The snapshot ensemble performs similar to Brock et al.
(2017) and an ensemble model build from the 5 runs can compete with the hand crafted WRN 28-10.
The performance of the Shake-Shake network (Gastaldi, 2017) is again not reached.
C ONCLUSION
We proposed a simple and fast method for automated architecture search based on a hill climbing strategy, network morphisms, and training via SGDR.
Experiments on CIFAR-10 as well as CIFAR-100 showed that our method yields competitive results while requiring considerably less computational resources than most alternative approaches.
Our algorithm is easily extendable, e.g., by other network morphisms, evolutionary approaches for generating new models, other methods for performance evaluation, e.g., learning curve prediction (Klein et al., 2017), or better resource handling strategies, e.g., hyperband (Li et al., 2016) ).
In this sense, we hope that our algorithm can serve as a basis for more sophisticated methods.