Paper:
Gaussian Process Bandit Optimisation with Multi-fidelity Evaluations
In many scientific and engineering applications, we are tasked with the optimisation of an expensive to evaluate black box function f . Traditional methods for this problem assume just the availability of this single function. However, in many cases, cheap approximations to f may be obtainable. For example, the expensive real world behaviour of a robot can be approximated by a cheap computer simulation. We can use these approximations to eliminate low function value regions cheaply and use the expensive evaluations of f in a small but promising region and speedily identify the optimum. We formalise this task as a multi-fidelity bandit problem where the target function and its approximations are sampled from a Gaussian process. We develop MF-GP-UCB, a novel method based on upper confidence bound techniques. In our theoretical analysis we demonstrate that it exhibits precisely the above behaviour, and achieves better regret than strategies which ignore multi-fidelity information. MF-GP-UCB outperforms such naive strategies and other multi-fidelity methods on several synthetic and real experiments.

Review:
The authors considered Bayesian optimization framework. The main difference of the considered problem statement consists in that multi-fidelity black-boxes are available, realizing the same function. The data source with the highest fidelity at the same time is the most expensive to evaluate. It is assumed that costs of evaluation of data sources with different fidelities are given.  The authors proposed multi-fidelity bandit problem statement, they formalized the notion of regret when explicit costs of different fidelity observations are given. Then the authors proposed an approach for Bayesian optimization, which explicitly takes into account costs of observations and relationships between black-boxes with different fidelities. The approach is based on Gaussian process upper confidence bound algorithm. Asymptotics of the corresponding regret is provided w.r.t. increasing capital of a resource, available for black-boxes evaluations. Extensive computational experiments provide evidence about efficiency of the proposed approach.
Comments: - line 88. It seems that the comma should be placed before the word «where», not after this word. - line 101. In the formula for a Gaussian distribution (at the end of the line) there is a redundant power «2». It seems that after the word «with» the comma should not be used. - line 105. The definition of the regret depends on B, which is defined in A3. In practice, if I would like to monitor the dependence of the regret on iterations number, how should I set a value of B? - In formula (2) the authors introduce variables $\lambda^{(m)}$. However, starting from line 119 there is no any explanation what $\lambda^{(m)}$ stands for (although it is quite easy to understand, but formally it should be written). - Will the considered regret bound and theoretical results about it be still valid if for the case of lower fidelity functions we set the instantaneous rewards to be  equal not to -B, but  to some set of arbitrary numbers? - Formula (3). At the end of the formula the dot should be placed. - line 252. «...if the algorithm, is stuck at fidelity m for too long then …». What is «too long» here? Any possibility to make a decision automatically? Or a human intervention is needed? - Are there any recommendations how to set values of $\lambda^{(m)}$? Of course, I understand, that these values are defined by to which extent a high fidelity model is more expensive to evaluate then low fidelity models. However, this information is not always precisely known. At the same time values of $\lambda^{(m)}$ influence the outcomes of the proposed algorithm. That is why I am asking about the corresponding authors' experience. - line 313, Type Ia Supernovae. M = 3 fidelity functions are considered. In previous examples these functions were defined e.g. by the sample size, used to estimate the functions values. How is it done in this test case? - line after line 396 (Appendix, A.1 Table of notations), the seventh row from below. There is some problem with a reference to some equation. - line 491. «Let $\sigma_{t-1}$ denote the posterior…» -> «Let $\sigma_{t-1}^2$ denote the posterior..." -  line 532. «vol(vol(…»-> «vol(…» - In the proof of theorem 10 I would like to ask the authors to explicitly: a) write where in the proof assumption A2 (line 105) is used, b) comment on whether we consider the statement of the theorem conditional only on those GP trajectories, which fulfill assumptions A2 and A3 (line 105), or not.  Conclusions: - The topic of the paper is very important. E.g. in engineering practice it is a typical situation when black-boxes of different fidelities are available. - As far as I know this is the first time when a notion of a cumulative regret for the multi-fidelity case is rigorously introduced. - The proofs seem correct. Results of experiments confirm theoretical conclusions. The paper is well written. - Therefore, I would recommend to publish this paper in the NIPS proceedings.