Paper: A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR-2024)

Reviewer comment (Experiments):
I agree with the authors that the time might not be relevant when the function evaluations are expensive, still it is important to create an experiment assuming fast function evaluations and see whether the ranking holds.

Author response:
In response to your comment on runtime, we have now updated Appendix D.11 with a new experiment comparing the performance of the surrogate models within a time budget, under the assumption of very fast function queries. We find that in this setting, GPs and I-BNNs outperform other BNNs which have slower inference times. However, we would like to re-emphasize that this scenario is often not representative of many common Bayesian optimization problems, which instead have very expensive function queries which would dominate the time budget.

We hope we have addressed your concerns, and please let us know if you have any remaining questions.

Reviewer comment (Experiments):
Although the dataset collection is diverse, the study is performed on a very small amount of datasets. There is no guarantee that these findings extrapolate easily to new datasets.

Author response:
We respectfully disagree that we consider a "very small number of datasets". We benchmarked our results on 15 different datasets, which is more than many other related Bayesian optimization papers:
- 8 datasets: Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization, Kristiadi et al. 2023
- 5 datasets: Scalable Global Optimization via Local Bayesian Optimization, Eriksson et al. 2019
- 8 datasets: Scalable Bayesian Optimization Using Deep Neural Networks, Snoek et al. 2015
- 3 datasets: Maximizing Acquisition Functions for Bayesian Optimization, Wilson et al. 2018
- 9 datasets: Bayesian Optimization with High-Dimensional Outputs, Maddox et al. 2021
- 11 datasets: Bayesian Optimization with Robust Bayesian Neural Networks, Stefan et al. 2016

Reviewer comment (Novelty):
Although the insights are interesting (some of them are not surprising), the effective impact is questionable. How could these insights lead to SotA in drug discovery algorithms, active learning, material science or hyperparameter optimization?

Author response:
The choice of surrogate model in Bayesian optimization is a fundamental design decision that has been largely overlooked in the literature, and many papers default to using standard Gaussian process surrogates (e.g., with Matern or RBF kernels). Our paper is the first to provide a comprehensive study of BNN surrogates, and the wide range of neural network-based surrogates explored in our submission allows us to evaluate the role of representation learning, non-stationarity, and stochasticity in modeling Bayesian optimization objectives. These findings offer pragmatic insights for BO practitioners by providing guidance on selecting appropriate surrogate models.

Our paper also has many novel and important findings, in some cases even prompting us to re-think conventional wisdom in BO. Surprising and unexpected findings include the following: (1) There are limited empirical benefits of stochasticity in the surrogate, given the competitive performance of DKL to BNN models; (2) the performance of deep ensemble surrogates is surprisingly poor, despite their success in other non-BO contexts; (3) I-BNN demonstrate compelling performance on high dimensional benchmarks; (4) Different BO objectives exhibit highly different salient structures (e.g., in contrast to many vision and NLP problems). We additionally note that we are, to the best of our knowledge, the first to consider infinite BNN and HMC surrogates, and I-BNNs have strikingly good results in the high dimensional settings. While not every result would be expected to be surprising, many of the results in our submission are both surprising and informative.

In this paper, we do not try to develop new methods for specific Bayesian optimization applications and note that in order to reach state-of-the-art performance in drug discovery, materials engineering, or some applied problem of that nature, the paper would need to be entirely about that single applied problem and the idiosyncrasies associated with it. Our focus is instead on a general foundational scientific study on the effectiveness of BNN surrogate models on a wide range of problems, covering a wide array of standard benchmarks, with various different properties. We have further outlined the contributions of the paper in the separate general post.

Reviewer comment (Experiments):
No discussion on the regularization effect on the surrogates. What happens if I regularize the DKL or the Bayesian neural networks via some useful prior?

Author response:
We explored this question in the submitted manuscript. In Appendix D.7, we include results for DKL as suggested in [2], showing the performance of optimizing the parameters with the marginal likelihood (ML) compared to the conditional marginal likelihood (CML). We see that there is no clear preference for using the ML or the CML, and the behavior of DKL with ML and CML is very similar across many of the objectives.

Reviewer comment (Experiments):
How is the performance of DKL affected by regularization approaches such as [2], [3]?

Author response:
We explored this question in the submitted manuscript. In Appendix D.7, we include results for DKL as suggested in [2], showing the performance of optimizing the parameters with the marginal likelihood (ML) compared to the conditional marginal likelihood (CML). We see that there is no clear preference for using the ML or the CML, and the behavior of DKL with ML and CML is very similar across many of the objectives.

Reviewer comment (Experiments):
For an empirical paper whose conclusions rest solely on the experimental results, 5 trials for each experimental setup is too little, as evidenced by multiple plots having heavily overlapping confidence intervals, Figure 6 in particular.

Author response:
> 5 trials is too little

With five trials we do see sufficient evidence to draw many informative conclusions. However, we agree more trials would be helpful and will run all of our experiments for a minimum of ten trials in the final paper to further differentiate the performance between surrogate models.

Although Figure 6 has heavily overlapping confidence intervals, running more trials may not decrease the width of the intervals. This figure plots the distribution of the relative performance of each surrogate model across all trials and all objective functions. For example, we see that for at least 25% of the trials, the relative performance of GPs is less than 0.5, meaning the maximum reward found by GPs for that trial was closer to the reward found by the worst surrogate than the reward found by the best surrogate. We would expect this ratio to stay the same even as we increase the number of trials.

Thank you again for the thoughtful feedback, which we believe has improved the paper. We have added substantial new experiments inspired by your questions, and would appreciate if you could consider raising your score in light of our response.

Reviewer comment (Evaluation):
From Figure 6, GP and I-BNN are the top 2 performing models on all problems, and I-BNN dominates by far on high dimensional problems. However, I-BNNs are equivalent to (and implemented as) GPs with a specific neural-network based kernel. One may reach the conclusion that GPs still reign supreme in BO: use a standard Matern kernel for low dimensional problems, and the I-BNN kernel for high dimensional problems, and ignore BNNs that are not also GPs. Would you say this is a fair alternate conclusion?

Author response:
> Question 2: Do GPs still reign supreme?

Our results show that GPs with a standard Matern kernel works well for lower-dimensional problems and GPs with the I-BNN kernel work well in higher dimensions.

Indeed, GPs encompass a broad class of models. GPs with I-BNN kernels exhibit very different behavior from standard GPs with a Matern or RBF kernel. The infinite-width BNN has a non-Euclidean and non-stationary similarity metric, which enables its relative success in high dimensions. DKL based GPs are also significantly different, with their own properties, such as representation learning.

For this reason, we have been careful to distinguish "standard GPs"---GPs with the kernels almost always used in Bayesian optimization---from other model classes, as GPs allowing for any kernel are so general it becomes difficult to make informative conclusions or comparisons. Indeed, the line can get blurry---I-BNNs are a particular type of GP derived from a BNN.

We also note that we have several nuanced take-aways, rather than a single conclusion, and are careful not to proclaim standard GPs or BNNs as overall winners or losers. There is no desire to show that BNNs are generally superior, and we are in fact careful to highlight the broad appeal of standard GP surrogates. Our goal is to understand how various properties of the surrogate (stochasticity, representation learning, etc.) interact with performance in different settings, which transcend the GP vs. BNN distinction, though many of our considered surrogates are NN-inspired (HMC, SGHMC, Deep Ensembles, I-BNN, DKL). If we define BNNs narrowly, and GPs broadly, it may be reasonable to say that GPs have an edge, but we are most concerned with the low-level property distinctions of the surrogates.

Reviewer comment (Novelty):
I think it would make the study more complete if another relevant line of works is discussed: using (non-Bayesian) neural networks as the surrogate in BO and using neural tangent kernel for exploration. The recent line of work on neural bandits has made it possible to use (non-Bayesian) neural networks as the surrogate in BO while still preserving the regret guarantee of BO by using the theory of the NTK. The relevance of neural bandits in BO has been shown by [1] below, and you can also refer to [1] to find the related works on neural bandits. In fact, I think the findings in [1] can be used to corroborate some of the findings in this work. For example, [1] also found that deep ensemble doesn't work well and explains it by arguing that deep ensemble cannot do principled exploration. I think this is in fact consistent with what's observed in this work, because the performance of deep ensemble plateaus at low objective values because its subpar exploration ability makes it unable to find the region containing the global optimum. The paper [2] below also did an empirical study of neural bandit methods, so the findings in [2] may also be compared/combined with those in this paper to potentially get more insights. For example, [2] also found that neural bandits tend to work better when the objective function is complicated. In fact, the connection between BNN-surrogate BO and neural bandits has also been discussed by the concurrent work of Kristiadi et al. (2023) in the context of linearized-Laplace approximation. The recent work of [3] has also shown the potential of using NTK in kernel regression, which may also be an empirical justification for the potential of BO with NTK based surrogate.

Author response:
Thank you for the references! While our paper focuses on Bayesian neural network surrogates for Bayesian optimization, for a thorough investigation, we recognize that there are other works focusing on non-Bayesian methods such as neural bandits and NTK. We have updated our related works section with the suggested papers, and we have also updated Appendix D.3 to include a discussion of the performance of deep ensembles in other works.

Reviewer comment (Presentation):
Another minor point which can make the paper easier to read is that when referring to the appendix, it'll make it easier for the reader if the specific subsection is referred to instead of just "Appendix D".

Author response:
Thank you for the suggestion! We have updated our draft accordingly.