Paper:
We propose Efficient Neural Architecture Search (ENAS), a faster and less expensive approach to automated model design than previous methods. In ENAS, a controller learns to discover neural network architectures by searching for an optimal path within a larger model. The controller is trained with policy gradient to select a path that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected path is trained to minimize the cross entropy loss. On the Penn Treebank dataset, ENAS can discover a novel architecture thats achieves a test perplexity of 57.8, which is state-of-the-art among automatic model design methods on Penn Treebank. On the CIFAR-10 dataset, ENAS can design novel architectures that achieve a test error of 2.89%, close to the 2.65% achieved by standard NAS . Most importantly, our experiments show that ENAS is more than 10x faster and 100x less resource-demanding than NAS.

Review:
Summary: 
The paper presents a method for learning certain aspects of a neural network architecture, specifically the number of output maps in certain connections and the existence of skip connections. The method is relatively efficient, since it searches in a space of similar architectures, and uses weights sharing between the tested models to avoid optimization of each model from scratch. Results are presented for image classification on Cifar 10 and for language modeling.
Page 3: “for each channel, we only predict C/S binary masks”   -> this seems to be a mistake. Probably “for each operation, we only predict C/S binary masks” is the right wording
Page 4: Stabilizing Stochastic Skip Connections: it seems that the suggested configuration does not enable an identity path, which was found very beneficial in (He. et al., 2016). Identity path does not exist since layers are concatenated and go through 1*1 conv, which does not enable plain identity unless learned by the 1*1 conv.
Page 5: 
-	The last paragraph in section 4.2 is not clear to me. What does a compilation failure mean in this context and why does it occur? And: if each layer is connected to all its previous layers by skip connections, what remains to be learned w.r.t the model structure? Isn’t the pattern of skip connection the thing we would like to learn?
-	Some  details of the policy LSTM network are also not clear to me:
o	How is the integer mask (output of the B channel steps) encoded? Using 1-hot encoding over 2^{C/S} output neurons? Or maybe C/S output neurons, used for sampling the C/S bits of the mask? this should be reported in some detail.
o	How is the mask converted to an input embedding for the next step? Is it by linear multiplication with a matrix? Something more complicated? And are there different matrices used/trained for each mask embedding (one for 1*1 conv, one for 3*3 conv, etc..)?
o	What is the motivation for using equation 5 for the sampling of skip connection flags? What is the motivation for averaging the winning anchors as the average embedding for the next stage (to let it ‘know’ who is connected to the previous?). Is anchor j also added to the average?
o	How is equation 5 normalized? That is: the probability is stated to be proportional to an exponent of an inner product, but it is not clear what the constant is and how sampling is done.
Page 6: 
-	  Section 4.4: what is the fixed policy used for generating models in the stage of training the shared W parameters? (this is answered at page 7
Experiments:
-	The accuracy figures obtained are impressive, but I’m not convinced the ENAS learning is the important ingredient in obtaining them (rather than a very good baseline)
-	Specifically, in the Cifar -10 example it does not seem that the networks chooses the number of maps in a way which is diverse or different from layer to layer. Therefore we do not have any evidence that the LSTM controller has learnt any interesting rule regarding block type, or relation between block type width and layer index. All we see is that the model does not chose too many maps, thus avoid significant overfit. The relevant baseline here is a model with 64 or 96 maps on each block, each layer.Such a model is likely to do as well as the ENAS model, and can be obtained easily with slight parameter tuning of a single parameter.
-	 Similarly, I’m not convinced the  skip connection pattern found for Cifar-10 is superior to standard denseNet or Resnet pattern. The found configuration was not compared to these baselines. So again we do not have evidence showing the merit of keeping and tuning many parameters with the RINFORCE
-	The experiments with Penn Treebank are described with too less details: for example, what exactly is the task considered (in terms on input-output mapping), what is the dataset size, etc..
-	Also, for the Penn treebank experiments no baseline is given, so it is not possible to understand if the structure learning here is beneficial. Comparison of the results to an architecture with all skip connections, and with a single skip connection per layer is required to estimate if useful structure is being learnt.
Overall:
-	Pro: the method gives high accuracy results 
-	Cons: 
o	It is not clear if the ENAS search is responsible to the results, or just the strong baseline. The advantage of ENAS over plain hyper parameter choosing was not sufficiently established.
o	The controller was not presented in a clear enough manner. Many of its details stay obscure.
o	The method does not seem to be general. It seems to be limited to choosing a specific set of parameters in very specific scenario (scenario which enable parameter sharing between model. The conditions for this to happen seems to be rather strict, and where not elaborated).
After revision:
The controller is now better presented.
However, the main points were not changed:
   - ENAS seems to be limited to a specific architecture and search space, in which probably the search is already exhausted. For example for the image processing network, it is determining the number of skip connections and structure of a single layer as a combination of several function types. We already know the answers to these search problems (denser skip connection pattern works better, more functions types  in a layer in parallel do better, the number of maps should be adjusted to the complexity and data size to avoid overfit). ENAS does not reveal a new surprising architectures, and it seems that instead of searching in the large space it suggests, one can just tune a 1-2 parameters  (for the image network, it is the number of maps in a layer).
  - Results comparing ENAS results to the simple baseline of just tuning 1-2 hyper parameters were not shown. I hence believe the strong empirical results of ENAS are a property of the search space (the architecture used) and not of the search algorithm.