Paper:
Truncated Variance Reduction: A Unified Approach to Bayesian Optimization and Level-Set Estimation
We present a new algorithm, truncated variance reduction (TRUVAR), that treats Bayesian optimization (BO) and level-set estimation (LSE) with Gaussian processes in a unified fashion. The algorithm greedily shrinks a sum of truncated variances within a set of potential maximizers (BO) or unclassified points (LSE), which is updated based on confidence bounds. TRUVAR is effective in several important settings that are typically non-trivial to incorporate into myopic algorithms, including pointwise costs and heteroscedastic noise. We provide a general theoretical guarantee for TRUVAR covering these aspects, and use it to recover and strengthen existing results on BO and LSE. Moreover, we provide a new result for a setting where one can select from a number of noise levels having associated costs. We demonstrate the effectiveness of the algorithm on both synthetic and real-world data sets.

Review:
This manuscripts introduces a new algorithm called "truncated variance reduction" (TruVaR; I will write TV here) that is appropriate both for level set estimation and Bayesian optimization on nonlinear functions. The idea is to maintain a set of points that are either possible maximizers of the function (for Bayesian optimization) or points near the level set threshold (for LSE). We then greedily evaluate the function so as to minimize the total truncated variance of points in this set. When the maximum uncertainty in this set drops below the current truncation value, we decrease that value and continue. The authors provide a theoretical guarantee for the performance of this algorithm, providing a unified analysis for both LSE and Bayesian optimization in terms of a notion they call "epsilon-accuracy." The authors also consider the cases where the observation noise is heteroskedastic and when there is a known cost of observing a given point, providing simple extensions of the algorithm for these cases and proving a result in the nonuniform cost case. Finally, a series of experiments investigates the performance of the proposed method on the LSE and Bayesian optimization tasks.
This manuscript has some interesting ideas. The algorithm is simple, easy-to-understand, and the fact that it works is intuitive. The joint analysis of Bayesian optimization and LSE via "epsilon-accuracy" is nice, although it's not terribly surprising that this is possible. The analysis is somewhat straightforward and predictable, using many tools that are now common in algorithms like these (confidence bounds, maximum mutual information, submodularity). That said, the theoretical contributions certainly strengthen the proposed method.  That said, I have a few complaints about this paper. The joint analysis of LSE and Bayesian optimization is interesting and perhaps even insightful. I think the LSE problem is interesting and deserving of more attention than it gets, which seems to at the rate of a paper every few years. Where this paper fails, however, is in purporting that these problems are of equal practical importance. This is not done so explicitly, but rather implicitly by certain choices made, e.g., in the experimental setup (see below). Although LSE is a compelling problem, and indeed has some applications, e.g., in environmental monitoring, it has a rather niche role in the scheme of things. This somewhat limits the potential impact of this paper. Bayesian optimization, on the other hand, has many natural applications across a range of domains. I say this not because Bayesian optimization is my pet problem (it's not), but rather out of simple pragmatism. A strong algorithmic contribution to Bayesian optimization can have a large impact indeed.  Unfortunately, I don't think this paper does a sufficient job of selling the proposed algorithm to that audience, due to the experimental treatment.  I believe the experiments are rather weak in comparison to the theoretical contributions here. LSE is investigated first, on an interesting real-world environmental monitoring dataset. The authors investigate the performance of TV against VAR (uncertainty sampling), a previous LSE algorithm, and the "straddle" algorithm from Bryan and Schneider, with TV appearing to work well but perhaps not quite as well as straddle. They turn their attention to the incorporation of cost (in this case, defined with respect to the movement of the autonomous vehicle), and show that TV outperforms LSE when measuring performance vs cost. This is not surprising due to the fact that LSE is agnostic of cost. The authors point out that their algorithm often causes the boat to remain stationary, instead choosing to measure at the same location but at a different depth. Even then, the boat moves around quite a bit along the transect. I can't help but ask how a simple grid design would perform, which could be realized with very little movement cost.  The authors then turn their attention to Bayesian optimization, the more important of the two problems, and in my opinion give it short shrift experimentally. The test functions are synthetic draws from Gaussian processes and a single hyperparameter tuning experiment (where the results are inconclusive). I do not judge these to be sufficient to convince the Bayesian optimization community that "yet another acquisition function" is needed in an already crowded space.  Varying costs of observation are also ignored, with the authors claiming that the LSE experiments "already demonstrated the effects of costs." I disagree. The LSE experiments show only that TV outperforms the cost-unaware LSE algorithm. Moreover, varying observation cost has received much attention in Bayesian optimization, and offering a better solution to that problem would significantly raise the impact of this contribution. Instead, the authors sweep it under the rug. A comparison against, e.g., EI / cost would go a long way here.  I do not understand the sentence "Since the domain is discrete, we compare only against EI and GP-UCB." Entropy search is certainly tractable here, if only via sampling and rank-one updates.  Is 24 * 14 * 4 (= 1344) a typo? 25 * 14 * 4 (= 1400)?  Finally, the experiments are riddled with magic constants, some of which are not really motivated (although citations are given for some). For example, why the exact set of costs {15, 10, 2}? Why not some other set? Reducing the noise variance from 0.05 to 1e-6 is a 223-fold increase in fidelity. Why is that only 7.5 times more expensive?  The manuscript is relatively well written and easy to read. However, many of the figures are difficult to read due to tiny text and/or small markers/details (e.g. Figure 1, Figures 2(e) and 2(f)). Reading the paper, I wondered whether there might be a nontrivial connection between TV and entropy search. It seems like TV will by construction be reducing the entropy of p(x* | D), if only indirectly. A bit of discussion on that might be interesting. For example, could it be possible that TV is approximately performing entropy search, using a cheap surrogate? If so, is it possible this could be made precise?