Paper:
LEARNABLE EMBEDDING SPACE FOR EFFICIENT NEURAL ARCHITECTURE COMPRESSION
We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search. Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation. We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018). The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShuffleNet (Zhang et al., 2018). We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training.

Review:
================
Post-Rebuttal
================
I thank the authors for the larger amount of additional work they put into the rebuttal. Since the authors addressed my main concerns, i e. comparison to existing methods,  clarifications of the proposed approach, adding references to related work, I will  increase my score and suggest to accept the paper.

The paper describes a new neural architecture search strategy based on Bayesian optimization to find a compressed version of a teacher network. The main contribution of the paper is to learn an embedding that maps from a discrete encoding of an architecture to a continuous latent vector such that standard Bayesian optimization can be applied. 
The new proposed method improves in terms of compressing the teacher network with just a small drop in accuracy upon an existing neural architecture search method based on reinforcement learning and random sampling.
Overall, the paper presents an interesting idea to use Bayesian optimization on high dimensional discrete problems such as neural architecture search. I think a particular strength of this methods is that the embedding is fairly general and can be combined with various recent advances in Bayesian optimization, such as, for instance, multi-fidelity modelling.
It also shows on some compression experiments superior performance to other state-of-the-art methods.
However, in its current state I do not think that the paper is read for acceptance:
- Since the problem is basically just a high dimensional, discrete optimization problem, the paper misses comparison to other existing Bayesian optimization methods such as TPE [1] / SMAC [2] that can also handle these kind of input spaces. Both of these methods have been applied to neural architecture search [3][4] before. Furthermore, since the method is highly related to NASBOT [5], it would be great to also see a comparison to it.
- I assume that in order to learn a good embedding, similar architectures need to be mapped to latent vector that are close in euclidean space, such that the Gaussian process kernel can model any correlation[7]. How do you make sure that the LSTM learns a meaningful embedding space? It is also a bit unclear why the performance f is not used directly instead of p(f|D). Using f instead of p(f|D) would probably also make continual training of the LSTM easier, since function values do not change.
- The experiment section misses some details:
  - Do the tables report mean performances or the performance of single runs? It would also be more convincing if the table contains error bars on the reported numbers.
  - How are the hyperparameters of the Gaussian process treated?
  
- The related work section misses some references to Lu et al.[6] and Gomez-Bombarelli et al.[7] which are highly related.
- What do you mean with the sentence  "works on BO for NAS can only tune feed-forward structures" in the related work section? There is no reason why other Bayesian optimization should not be able to also optimize recurrent architectures (see for instance Snoek et al.[8]). 
- Section 3.3 is a bit confusing and to be honest I do not get the motivation for the usage of multiple kernels. Why do the first architectures biasing the LSTM? Since Bayesian optimization with expected improvement samples around the global optimum, should not later evaluated, well-performing architectures more present in the training dataset for the LSTM?
[1] Algorithms for Hyper-Parameter Optimization
    J. Bergstra and R. Bardenet and Y. Bengio and B. Kegl
    Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NIPS'11)
[2] Sequential Model-Based Optimization for General Algorithm Configuration
    F. Hutter and H. Hoos and K. Leyton-Brown
    Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION'11)
[3] Towards Automatically-Tuned Neural Networks
    H. Mendoza and A. Klein and M. Feurer and J. Springenberg and F. Hutter
    ICML 2016 AutoML Workshop
[4] Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures
    J. Bergstra and D. Yamins and D. Cox
    Proceedings of the 30th International Conference on Machine Learning (ICML'13)
[5] Neural Architecture Search with Bayesian Optimisation and Optimal Transport
    K. Kandasamy and W. Neiswanger and J. Schneider and B. P{\'{o}}czos and E. Xing
    abs/1802.07191
[6] Structured Variationally Auto-encoded Optimization
    X. Lu and J. Gonzalez and Z. Dai and N. Lawrence
    Proceedings of the 35th International Conference on Machine Learning
[7] Automatic chemical design using a data-driven continuous representation of molecules
    R. Gómez-Bombarelli and J. Wei and D. Duvenaud and J. Hernández-Lobato and B. Sánchez-Lengeling and D. Sheberla and J. Aguilera-Iparraguirre and T. Hirzel. and R. Adams and A. Aspuru-Guzik
    American Chemical Society Central Science
[8] Scalable {B}ayesian Optimization Using Deep Neural Networks
    J. Snoek and O. Rippel and K. Swersky and R. Kiros and N. Satish and N. Sundaram and M. Patwary and Prabhat and R. Adams
    Proceedings of the 32nd International Conference on Machine Learning (ICML'15)