Paper: A General Framework for User-Guided Bayesian Optimization (ICLR-2024)

Reviewer comment (Theory):
The mathematical derivation is too informal in some places, affecting clarity. I believe this needs to be improved. For instance, $\pi$ here represents a belief over functions. Then why is Eq. (4) a function that receives a point $x \in \mathcal{X}$? Shouldn't it be a probability distribution over functions? Also, Def 3.1 introduces a conditioning on $\pi$. I found this quite confusing. Is $\pi$ a density? a function? or a random variable?

Author response:
We see that the confusion is warranted, and have consequently re-hashed section 3.1. Hopefully, this improves the understanding and overall experience. $\pi (f)$ is replaced by $\rho (f)$. Moreover, the domains, co-domains and variable properties (i.e. randomness of $\rho$ is communicated throughout. 

The idea of Eq. 4 was to demonstrate how $\pi(x)$ can be expressed as a function not of the optimizer $x^*$, but as a function of $f$. We subsequently introduce notation where $\rho$ is a function of $f$ for some arbitrary function property, e.g. $x_*$ and $f_*$. We have now re-done the notation to clarify the inputs to each quantity and hope that this improves the overall clarity.

Reviewer comment (Experiments):
There are concerns about the fairness of the experiments. The authors state that "ColaBO and πBO are initialized with the mode of the prior followed by 2 Sobol samples, whereas LogEI and MES are conventionally initialized with D + 1 Sobol samples." Shouldn't LogEI and MES also have been initialized on the mode of the prior? If we assume that such prior information is present in user-guided BO methods, it is fair to assume that conventional BO methods also have access to this information.

Author response:
1. Added five additional HPO tasks (6D) from LCBench. Hopefully, this tackles the reviewer's concerns regarding the experimental evaluation. Moreover, all methods now utilize the same $\text{Mode} + 2$ initialization, on all tasks. Lastly, we have added an illustrative example of how priors over preference relations can be used in ColaBO in a newly added Fig. 1 in the intro.

Reviewer comment (Experiments):
The evaluations are okay but not extensive (especially compared to the piBO paper). I think this is an important point, given that the utility of user-guided BO methods can only be judged empirically on a case-study basis. This connects with my next concern.

Author response:
1. Added five additional HPO tasks (6D) from LCBench. Hopefully, this tackles the reviewer's concerns regarding the experimental evaluation. Moreover, all methods now utilize the same $\text{Mode} + 2$ initialization, on all tasks. Lastly, we have added an illustrative example of how priors over preference relations can be used in ColaBO in a newly added Fig. 1 in the intro.

Reviewer comment (Experiments):
The paper lists a couple of different types of function priors in Section 3.1 but only seems to evaluate one type of such prior. It is, therefore, hard to judge how general this framework is for incorporating prior knowledge. Especially since the users state in the contributions that one can "incorporate arbitrary user knowledge."

Author response:
1. Added five additional HPO tasks (6D) from LCBench. Hopefully, this tackles the reviewer's concerns regarding the experimental evaluation. Moreover, all methods now utilize the same $\text{Mode} + 2$ initialization, on all tasks. Lastly, we have added an illustrative example of how priors over preference relations can be used in ColaBO in a newly added Fig. 1 in the intro.

Reviewer comment (Novelty):
The contribution statement is too general except for item 2. For instance, 1 and 3 can be applied to any user-guided BO method. I expect some more technical details about what this method offers to the field of user-guided BO.

Author response:
3. Specified contributions to better distinguish ColaBO from other user-guided BO.

Reviewer comment (Theory):
The derivation of importance sampling in Eq. (10) is unclear. $\pi(f) p(f | D)$ is an unnormalized density, but the integral is taken over it. Then, the equality with Eq. (9) doesn't hold since the expectation is not normalized.

Author response:
The derivation of importance sampling in Eq. (10) is unclear. is an unnormalized density, but the integral is taken over it. Then, the equality with Eq. (9), doesn't hold since the expectation is not normalized.

Thanks for catching this typo. We missed a normalizing constant on $\pi$, which will not generally be tractable. We have added this normalizing constant and clarified accordingly. We have added a breif comment on the sampling, but will cover it in more detail in the CR when space permits.

Reviewer comment (Theory):
In Section 3.3, the authors introduce a tempering scheme based on the number of datapoints. Furthermore, they draw connections with generalized Bayesian inference (GBI). I think the authors should make it clear that the connection is very loose. A major characteristic of GBI is to find a statistically principled way to come up with the temperature, which is not the case here. It is particularly misleading as, in the last sentence of Section 3.3, the paper states that tempering is done so in a "principled Bayesian manner."

Author response:
We agree with the reviewer, and that section is now entirely removed. In later experiments, we found that the tempering did not substantially improve performance, and since its "principled" nature was debatable, we removed it to obtain an overall cleaner approach. In practice, the data is sufficient to decrease the impact of a poor user belief.

Reviewer comment (Writing):
Couldn't the name collaborative Bayesian optimization be misleading as to making people to think this method involves multiple users?

Author response:
Couldn't the name collaborative Bayesian optimization be misleading as to making people to think this method involves multiple users?

That is fair. We very much liked the name since the user can collaborate with BO by expressing any type of prior they have. But given your feedback, we will reconsider the name. We have not found a more appealing one as of now, though. Would you have a suggestion? If so, please let us know.

Reviewer comment (Writing):
Section 1 first sentence: Please cite classic papers that introduced BO for historical context.

Author response:
Section 1 first sentence: Please cite classic papers that introduced BO for historical context.
Thanks, we added Mockus, Jones et al and Snoek et al in the very beginning of the intro.

Reviewer comment (Presentation):
Figures 9-12: The boundary of the error bands makes it hard to distinguish between the plots. Please consider improving the visibility here.

Author response:
Figure 9-12: The boundary of the error bands makes it hard to distinguish between the plots. Please consider improving the visibility here.
We thank the reviewer for taking the time to go through the appendix. As the tempering is no longer part of the paper, these plots are removed. We do, however, agree with the reviewer's point.

Reviewer comment (Theory):
Section 2.1 equation: Generally, we aim to minimize the expectation of f or the regret. Saying that we minimize f, in the presence of noise, is too informal.

Author response:
Section 2.1 equation: Generally, we aim to minimize the expectation of f or the regret. Saying that we minimize f, in the presence of noise, is too informal.

We believe we are in line with convention using this problem formulation. Can the reviewer to point to an example in the literature paper that they believe is suitable for the noisy problem setting?

Reviewer comment (Writing):
Section 2.2 second from the last sentence: I think replacing "standard method" with "classic method" would be better here since we have an influx of more efficient methods.

Author response:
Section 2.2 second from the last sentence: I think replacing "standard method" with "classic method" would be better here since we have an influx of more efficient methods.
We agree, it has been changed.

Reviewer comment (Writing):
Eq (2): Has the "equivalent in distribution" sign been defined anywhere?

Author response:
Eq (2): Has the "equivalent in distribution" sign been defined anywhere?
That is correct, it has not. We have not added it as of now, and have not seen it defined e.g. in Wilson et. al (2020). Nonetheless, we will add it in the camera ready if the reviewer believes it is necessary (and spacing permits).

Reviewer comment (Writing):
Section 3.1 second paragraph second sentence: the sentence is incomplete.

Author response:
Section 3.1 second paragraph second sentence: the sentence is incomplete.
We agree. Section 3 is rewritten, so that sentence is no longer present.

Reviewer comment (Experiments):
Given that the authors rely on GP posterior sampling, an obvious thing that could have been tried is Thompson sampling. Have the authors tried to include it?

Author response:
The TS exclusion was an oversight, but it is now added (the decoupled variant, without user prior resampling, which it could be used in conjunction with). As there are many methods included who are relatively close (Fig. 6 in particular), we will do what we can to improve visibility in the CR.

Reviewer comment (Experiments):
The empirical section can be further enhanced by testing the algorithms on more challenging tasks, for example, higher-dimensional problems. The paper primarily focuses on the low-dimensional tasks, which might not provide a strong basis for demonstrating the algorithm’s performance in realistic problems, where the dimensions might be much higher.

Author response:
We have added more HPO tasks (6D) from LCBench. Hopefully, this gives a more complete view of performance of ColaBO. They can now be found in the newly added Appendix A, but will go into Sec. 4 when space permits.

Reviewer comment (Writing):
The presentation of the work can be improved. Initially, the authors define $\pi(\cdot)$ in Eq. 3 with input from $\mathcal{X}$, but it later becomes $\pi(f)$. Moreover, the notations become messier after Eq. 4, making it more challenging to follow. Also, while rejection sampling seems to be an important step in the model, the way it works is not introduced in the paper. Similar for RFF.

Author response:
We outline RFFs in the first paragraph of 2.3 and couldn’t find space to expand on it further. The exclusion of various sampling approaches (rejection, importance) but will be added to a camera-ready along with RFFs when space will permit it. 

We have made our best attempt at addressing your broader concern regarding clarity. Section 3.1 is now presented using slightly altered notation, where each property is properly distinguished and defined.

Reviewer comment (Experiments):
Given the authors’ claim that this is a general framework for BO to incorporate prior, the paper should present concrete examples to demonstrate its generality. For example, the authors could show how the examples in the second paragraph of the introduction section can be solved effectively within the proposed framework.

Author response:
We agree that the conventional prior over the optimum is over-emphasized in the experiments we had presented. To the reviewer’s point, we have added Fig. 1 to highlight examples of how each type of prior is incorporated into ColaBO. Hopefully, this serves to highlight the generality of ColaBO, as requested by the reviewer.

Moreover, there are max-value prior tasks in App. C.2, Fig. 10. We will try and highlight these types of tasks in the paper by adding yet another one.

Reviewer comment (Writing):
What is the difference between $f^*$ and $f_*$? What is $x_*$?

Author response:
Thanks for pointing this out, we have defined these objects more clearly in the new pdf. $f_*$ is the optimal value, $x_*$ is the optimal location —  $f^*$ is a typo which we have now fixed. Thank you for catching this! We used subscripts (as opposed to superscripts) to avoid double superscripts.

Reviewer comment (Experiments):
Is there any reason the authors in favor of rejection sampling compared to other sampling methods? What is the efficiency of using rejection sampling here?

Author response:
We initially pursued importance- and rejection sampling, but quickly found that rejection sampling yielded similar or better performance with substantially smaller memory usage, since many importance weights would become very small. Our method is, however, compatible with importance sampling.

##### What is the efficiency of using rejection sampling here? 
This depends on the dimensionality and strength of the prior, but in our experiments on the order of one in a thousand at worst, for the higher-dimensional problems. The procedure of acquiring samples takes between 1 and 5 seconds depending on the problem.

While this is certainly a drawback, the worst case only happens if the user specifies a narrow prior over each added dimension. We would reasonably expect the user to have a sound belief over a handful (<5) dimensions, and typically put a uniform prior on the rest. For this number of non-uniform dimensions _on the prior_, the sampling step does not bottleneck runtime.

Reviewer comment (Evaluation):
In Figure 5, even though ColaBO-MES is injected with a poorly located priors, it still performs better than MES, how do the authors interpret that?

Author response:
We posit that this is due to the RFF and MC approximations inducing additional randomness in the queries (caused by approximation errors), which assists in Stybtang specifically, where most acquisitions get stuck. As such, we do not believe ColaBO-MES is generally superior, and the outperformance is circumstantial.

Reviewer comment (Theory):
The authors show in Eq. 5 that $p(f|D, \pi) \propto \pi(f)p(f|D) $. Does this proportionality still hold in Eq. 6?

Author response:
Eq. 6 follows from Eq. 5 and Matheron's rule, where the realization of a random variable (whether $p(\hat{f})$ or $p(\hat{f}|\pi)$) is updated deterministically using the update described in 6. Since $(\hat{f}|\pi) \sim p(\hat{f})\pi(\hat{f})$, this holds.

Reviewer comment (Theory):
To indicate where the optimum is located within the prior, couldn’t we achieve it by simply defining the prior mean function in GP properly? For instance, assigning higher values to points where the users believe to be good and low values to the rest.

Author response:
Certainly - This has been done in Snoek et. al. (2015) for similar purposes. However, _it is unclear how a given mean function translates to a desired distribution over the optimum_, so one would have to properly parametrize and tune the mean function to get the desired distribution over the optimum. Moreover, the induced distribution over the optimum would be affected by both prior mean and lengthscales (and thus change when these estimates change) whereas ColaBO always retain the distribution over the optimum that has been specified, regardless of the hyperparameters of the model due to the orthogonality of these priors.

Specifically, the approximated GP mean in Fig. 1 (bottom left, dark blue) is a result of the prior (green). The reviewer's approach does this process in the reverse order - the prior over the optimum would follow from the (parameterizable) mean, but the mean would need updating if other HPs change.

Lastly, the approach would be orthogonal to ColaBO, in the sense that one could define _both_ a non-constant mean _and_ a prior over the optimum for the same problem. By aligning them, one would increase the efficiency of sampling (since more samples would have their optima at the mode of the user prior) but would introduce additional hyperparameters.

Reviewer comment (Theory):
Why do the authors consider log EI instead of EI?

Author response:
Apart from LogEI being superior from an acquisition optimization perspective (Ament et. al. 2023), smooth acquisition functions such as LogEI or MES are easier to approximate than non-smooth ones.

Reviewer comment (Experiments):
Test functions used to evaluate the proposed framework were quite limited, only a restricted set of test functions was employed.

Author response:
We thank the reviewer for their positive review.
#### __Test functions used to evaluate the proposed framework were quite limited, only a restricted set of test functions was employed.__
We have added 5 6D HPO tasks from the LCBench HPO benchmarking suite.Hopefully, this alleviates the reviewer's concern and further demonstrates the empirical performance and flexibility of ColaBO.