Paper:
Neural Architecture Search with Bayesian Optimisation and Optimal Transport
Bayesian Optimisation (BO) refers to a class of methods for global optimisation of a function f which is only accessible via point evaluations. It is typically used in settings where f is expensive to evaluate. A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model. Conventional BO methods have focused on Euclidean and categorical domains, which, in the context of model selection, only permits tuning scalar hyper-parameters of machine learning algorithms. However, with the surge of interest in deep learning, there is an increasing demand to tune neural network architectures. In this work, we develop NASBOT, a Gaussian process based BO framework for neural architecture search. To accomplish this, we develop a distance metric in the space of neural network architectures which can be computed efficiently via an optimal transport program. This distance might be of independent interest to the deep learning community as it may find applications outside of BO. We demonstrate that NASBOT outperforms other alternatives for architecture search in several cross validation based model selection tasks on multi-layer perceptrons and convolutional neural networks.

Review:
The paper describes a new distance metric for neural networks based on optimal transport. Based on this distance, the authors introduce a new Bayesian optimization strategy to optimize the architecture of neural networks.   Overall I think the methods is interesting and that the proposed distance might be also of interest for other applications than Bayesian optimization. However, it seems that computing the distance also requires some manual tuning such as defining the cost matrix which in turn requires expert knowledge. My only point of criticism is that this might hinder its success in automated machine learning applications, where human interaction should be reduced to a minimum.  Having said that, the method is well explained and the authors provide some meaningful empirical insights, such that the paper represents an important contribution for Bayesian optimization.  The following points need some further clarification:  1) How many actual function evaluation did every method perform in Figure 2. i.e how many architectures were trained? In the figure it seems that every method returns something after a discrete time step?  2) What do you mean with at least 5 independent runs for each method in the caption of Figure 2? Why did you perform less runs for some methods?  3) How are pending jobs treated in the parallel set up described in section 5?  4) In table 3, it says CIFAR-10 has 1k features instead of 32*32*3=3072. I guess that this is a typo?    --- Post Rebuttal ---  I thank the authors for taking their time to write the rebuttal and clarifying my points.