Paper:
Automating Bayesian optimization with Bayesian optimization
Bayesian optimization is a powerful tool for global optimization of expensive functions. One of its key components is the underlying probabilistic model used for the objective function f . In practice, however, it is often unclear how one should appropriately choose a model, especially when gathering data is expensive. We introduce a novel automated Bayesian optimization approach that dynamically selects promising models for explaining the observed data using Bayesian optimization in model space. Crucially, we account for the uncertainty in the choice of model; our method is capable of using multiple models to represent its current belief about f and subsequently using this information for decision making. We argue, and demonstrate empirically, that our approach automatically finds suitable models for the objective function, which ultimately results in more-efficient optimization.

Review:
PAPER SUMMARY:  This paper investigates a new Bayesian Optimization (BO) approach that automatically learns the correlation structure (i.e., kernel function) of the GP model of the black-box function while collecting observation to optimize it.  The key idea is to run a second BO on the model space (i.e., a collection of complex kernels generated by summing and multiplying base kernels) at each iteration to update the list of model candidates before choosing the next observation.   This 2nd BO routine casts the model evidence as a random function distributed by another GP. The implementation is adopted from a previous work (BOMS) on model selection via BO.  SIGNIFICANCE & NOVELTY:  This is an interesting problem & the approach taken is fairly new to me. I do, however, have the following concerns regarding the significance & technical soundness of the proposed solution:  The authors propose to treat the data collected at earlier stages as noisier observations to avoid recomputing the model evidence (as new data arrives) & claim that it would make the search better informed. This is, however, a strange intuition: adding (implicitly) more noise to earlier observations only increases the prediction variance & while it might help preventing the model from being overly confident in its prediction, it does not make the model more informed because the quality of information remains the same if the model evidence is not recomputed.  Section 4 (line 150-151) seems to suggest that the noise variance of a sample candidate depends on the previously collected samples. If this is indeed the case, the integration in line 179 is no longer analytically tractable. The description of Algorithm 1 is somewhat confusing: the update model sub-routine is said to recompute the corresponding model evidence before running the 2nd BO sub-routine on the model space but I thought the entire point here is to exploit heterogeneous noise scheme and avoid recomputing them? What exactly does the update model sub-routine do?  How do we set the model prior p(M)? At the beginning of a new iteration, the pool of model candidate is expanded & the prior will need to be changed. This does not seem to be discussed in the main text. Also, another issue is the large no. of approximation layers involved in computing the acquisition function for the main BO routine (see Section 4.1). This might affect the solution quality and I am raising this concern because I think the experiments are not very conclusive in this regard (please see my comments below).  EXPERIMENT:  If I understand correctly, the whole point of adding an extra model-searching BO sub-routine is to boost the performance of the main BO process. To demonstrate this conclusively, besides the experiments that the authors have presented, the proposed approach should also be compared with other state-of-the-art BO algorithms (e.g., predictive entropy search (PES) [1], upper-confidence-bound (UCB) ant etc., [2]) implementing the best model among a set of pre-selected candidates.   However, in this regard, the proposed approach is only compared with a baseline including a standard EI implementing the simple non-isotropic SE kernel. Is this SE model among the proposed approach’s pre-selected pool of candidates & is it the best among those? Also, the experiment description seems to suggest that this baseline is subject to the same amount of approximations in Section 4 (see lines 185-186). This is somewhat strange because EI is analytically tractable and such approximations are not necessary and only serve to weaken the baseline.  For a more conclusive demonstration, I would suggest the authors to compare their method with other state-of-the-art BO algorithms (without model selection) implementing (a) the worst and (b) the best model among those in the pre-selected pool. If the model selection machinery is efficient, we should at least see ABO outperforms (a) and approaching (or surpassing) (b) & this would resolve the last issue that I mentioned in the previous section (regarding the stable quality of the resulting solution after being subjected to multiple approximations).  CLARITY:  The paper is clearly written.  REVIEW SUMMARY:   I find the problem investigated in this paper novel & interesting but I have a few concerns and doubts regarding the proposed solution as well as its empirical evaluations, which are not very conclusive.  ---  [1] https://arxiv.org/abs/1406.2541 [2] https://arxiv.org/pdf/1206.2944  ---  Thank you for the rebuttal.   My first comment in the significance & novelty section is about the inner BO routine (not the outer BO routine as interpreted in your response). I do understand that by adding noises to the evidence observation to discourage poor model & at the same time, acknowledge that some of the old model evaluations might be out of date.   This however does not really explain how this would make the inner BO routine more informed in model search (e.g., how would a previously discarded model get reconsidered in light of new data? If the model search only uses its model evaluation + artificial noise, how could the predicted score of the discarded model be improved to make the search more informed?).   Secondly, part of my concern is whether the SE model is competitive against other model candidates in the pool. If it is less competitive than the others then the benefit from considering more model is not shown conclusively. I still think that the authors should compare with other models in the pool to demonstrate the efficiency of the developed model selection machinery.