Paper: Enhanced Bayesian Optimization via Preferential Modeling of Abstract Properties (ICLR-2024)

Reviewer comment (Evaluation):
There is no notion of human query cost compared to the black-box function evaluation cost. For expensive black-box functions, it is conceivable that querying the human is cheaper, but if many comparisons are needed, it is difficult to assess the benefit of the method. BOAP provides a way to integrate expert preferences, yielding an improvement for relevant preferences, but this assumes a null human query cost, and perhaps results would be different otherwise.

Author response:
- **BOAP assumes null human query cost**_
    - Bayesian optimization (BO) is the workhorse of scientific experimentation and product design. It provides a rich platform for human-AI collaboration because the nature of BO applications ensures that there would be an expert running the optimization process. In BOAP, the expert inputs considered are qualitative and available as pairwise design preferences
based on some abstract properties. Eliciting such pairwise rankings from the human expert should not add significant overhead, in contrast to specifying the complex explicit knowledge. 
    -  BOAP allows an expert to intervene only when they are willing to provide any preferential input thereby allowing them to evolve their knowledge over the course of optimization. Therefore, we have not considered any human query costs in the current version of the manuscript. However, it is an interesting direction to pursue that makes our proposed BOAP framework even more robust. In our experiments, at each iteration, we randomly choose $n/2$ datapoints to generate preferential data with the newly suggested datapoint.

Reviewer comment (Experiments):
Q1: During experiments, the process begins with $p = (t' 2)$ preferences, which gets updated in every iteration of the optimization process. Clarification is requested: if beginning with 5 initial observations gives 10 preferences, is the newly acquired sample in each iteration compared with all previous ones (e.g., the 6th sample compared with 5 previous ones), leading to a growing number of comparisons (15 preferences, then 21, and so on)? If so, this raises again the question of human query cost relative to black-box function evaluation cost.

Author response:
- **BOAP assumes null human query cost**_
    - Bayesian optimization (BO) is the workhorse of scientific experimentation and product design. It provides a rich platform for human-AI collaboration because the nature of BO applications ensures that there would be an expert running the optimization process. In BOAP, the expert inputs considered are qualitative and available as pairwise design preferences
based on some abstract properties. Eliciting such pairwise rankings from the human expert should not add significant overhead, in contrast to specifying the complex explicit knowledge. 
    -  BOAP allows an expert to intervene only when they are willing to provide any preferential input thereby allowing them to evolve their knowledge over the course of optimization. Therefore, we have not considered any human query costs in the current version of the manuscript. However, it is an interesting direction to pursue that makes our proposed BOAP framework even more robust. In our experiments, at each iteration, we randomly choose $n/2$ datapoints to generate preferential data with the newly suggested datapoint.

Reviewer comment (Experiments):
Q2: In all experiments, the expert is simulated and considered to reason over high-level features which are then used for comparisons and to learn the preferential Gaussian process whose posterior mean is added as additional input dimensions to the BO surrogate. How would this compare with using the high-level features directly as additional dimensions (e.g., for Benchmark-1d, input space $[x, \exp(2-x)^2, \frac{1}{x^2}]$)? Given that preferences are identifiable only up to a monotonous transformation, performance differences could be investigated to see if they relate to this identifiability issue. This could also be tested in "real-world experiments" as high-level features are also derived there.

Author response:
- **How would this method compare with actually using the high-level features directly as additional dimensions?**_
    - We would like to clarify to our reviewers that high-level features are not known directly to be used as an additional dimension, instead in practice an expert would reason about a system only qualitatively. However, we have conducted additional experiments to directly use high-level features as additional dimensions (BOAP-AD) and we have found that it outperforms BOAP by a small margin we believe this phenomenon is due to the fact that explicit measurements are always better than approximate qualitative measurements.

     -----------------------------------------------------
      |                 |      BOAP-AD     |       BOAP     |
     -----------------------------------------------------
      | Benchmark - 1D  |     0.01±0.002   |     0.05±0.01  | 
      | Rosenbrock - 3D |     3.16±1.19    |     5.32±0.12  | 
     -----------------------------------------------------

Reviewer comment (Writing):
Simple regret and Bayes Regret are never formally defined in the paper; adding definitions in the supplementary would be valuable.

Author response:
- _**Formal definitions: Bayes regret, simple regret; Search space for the synthetic experiments**_
    - We thank our reviewer for the suggestions, and we will provide them in the updated version of the manuscript.

Reviewer comment (Experiments):
The search space for the synthetic experiments is not mentioned anywhere in the text.

Author response:
- _**Formal definitions: Bayes regret, simple regret; Search space for the synthetic experiments**_
    - We thank our reviewer for the suggestions, and we will provide them in the updated version of the manuscript.

Reviewer comment (Theory):
The concept of abstract properties introduced in the paper appears measurable, making them potentially amenable to multi-objective or multi-fidelity optimization. Since they are referred to as "abstract," the authors should further motivate why they are considered abstract (i.e., immeasurable) and explain why existing Bayesian optimization approaches are unsuitable.

Author response:
- _**Why properties are abstract (i.e. immeasurable) and why existing BO approaches are ill-suited**_  
    - In numerous scenarios, experts often reason about a system based on their approximate or qualitative knowledge of the system at different abstraction levels. They reason using intermediate physical properties that may not be measurable directly. Using such high-level abstractions the expert compares designs and the reason why a design is better than another.
    - The existing BO approaches aim at capturing the expert preferential input directly on the objective function, in contrast to the preferences based on the aforesaid intermediate properties. Therefore we propose a human-AI collaborative framework to incorporate such expert preferential data about unmeasured abstract properties into BO.

Reviewer comment (Experiments):
(in 3.1) “To do this, we utilize an Automatic Relevance Determination (ARD) kernel where we set the lengthscales of the augmented input dimensions in proportion to the rank GP uncertainties” — why this instead of learning the parameters as one would typically do with GP models? Have ablation studies been run on how exactly the scale parameter \alpha is chosen? Is it inverse std? inverse var? or some other notion of uncertainty? The author has provided some explanation for this choice, but it is not convincing that this is the best choice without additional evidence.

Author response:
- _**Why this instead of learning the parameters as one would typically do with GP models?**_  
     - Tuning the human augmented GP ($\mathcal{GP}_{h}$) using the traditional loglikelihood/evidence maximization does not account for the expert uncertainties for the augmented features. Incorporating expert uncertainties from the individual rank GPs can be crucial for modelling as the expert feedback may be inaccurate and thus can adversely affect the overall optimization performance.  

- _**The choice of scale parameter $\alpha$**_  
    - The predictive variance ($\sigma^{2}$) in the rank GP is a measure of uncertainty in the expert model. In our proposed approach, the scale at which this uncertainty is to be considered in modelling the overall objective function is dictated by the scale parameter $\alpha$ and $\alpha\times\sigma$ will be the underlying lengthscale. We tune the scale parameter for each of the augmented dimensions by maximizing the log-likelihood. Alternatively, we could treat $\alpha$ as an overall "average" lengthscale, such that $\alpha$ is the underlying lengthscale when there is no uncertainty and with expert uncertainty the overall lengthscale is $\alpha+\sigma$.

Reviewer comment (Novelty):
If we are able to model abstract properties with preference models that are believed to be compositional parts (or contain important information) of the function of interest f, why don’t we explicitly exploit this compositional structure as done in Astudillo and Frazier (2020)?

Author response:
- _**why don't we explicitly exploit this compositional structure as done in Astudillo and Frazier (2020)?**_  
    - Our idea is fundamentally different from Astudillo and Frazier (2020) as we try to incorporate expert preferential feedback about the immeasurable abstract properties in modelling the given objective function $f$. To the best of our knowledge, Astudillo and Frazier (2020) capture expert preferences (along with their uncertainties) directly on $f$ and fail to account for auxiliary abstract properties that can be crucial for improving the optimization performance.

Reviewer comment (Experiments):
Human mistakes are mentioned multiple times throughout the paper but the author does not investigate how different kinds/scales of human errors could affect the performance of BOAP. Similarly, different human preference querying strategies are not explored. Astudillo and Frazier (2020) and Lin et al. (2022) have shown that querying the DM different questions (e.g., using the EUBO acquisition function) can have a significant impact on downstream model performance.

Author response:
- _**Different kinds of human errors can affect the performance of BOAP}**_  
    - We would like to refer our reviewer to the additional experiments mentioned in the supplementary material (Section A.4). We have conducted experiments to account for the human expert errors in providing preferential feedback. We have done this ablation study in two-fold. First, we show the robustness of our proposed approach against human errors in the selection of higher-order abstract properties. Second, we account for the human errors in providing the expert preferential feedback by flipping the preference between two input designs with some probability $\delta=0.3$. The ablation results and the experimental details are provided in the supplementary material (A.4.2).

Reviewer comment (Writing):
“Although we anticipate that experts will provide accurate preferences on abstract properties, the expert preferential knowledge can sometimes be misleading” — what does misleading mean here?

Author response:
- _**Expert preferential knowledge can sometimes be misleading**_  
    - We expect that the human expert preferential input is accurate and thus likely to accelerate BO. But, in some cases, this preferential feedback may be inaccurate due to the wrong/incorrect assumptions on the given search space. Therefore, such misguided preferential data given by an expert could potentially slow down the optimization. On the other hand, our method has some immunity to such inaccurate feedback.

Reviewer comment (Reproducibility):
(in 3.2) “To handle different scaling levels in rank GPs, we normalize its output in the interval [0, 1], such that μ(ω_i(x)) ∈ [0, 1]” — how is the normalization done?

Author response:
- _**Normalization in Rank GPs**_  
    - The individual rank GPs (${\mathcal{GP_{\omega_i}}}$) operate at different output scales (as dictated by MAP) and thus different scaling levels in their corresponding predictions. Therefore, before we augment the main GP with the mean predictions ($\mu_{\omega_{i}}(\mathbf{x})$) of the individual rank GPs, we min-max scale their outputs to be normalized in the interval $[0,1]$.

Reviewer comment (Theory):
The concatenation of different GP models, using some as inputs to others, is not rigorously investigated. What happens when kernel lengthscales are dynamically set based on a different model at different inputs? Is this provably a positive definite kernel still?

Author response:
- -**Dynamically setting kernel lengthscales based on a different model at different inputs? Is this provably a p.d kernel still?**_
    - Spatially varying lengthscales have already been explored in the past literature. [2] proved the positive definiteness of the spatially
varying kernel by showing that it has the required property of spatial variation of the lengthscales by suitably replacing the constant lengthscale with an arbitrary parameterized function of x. We refer our reviewer to \textquotedblleft Non-stationary Covariance Functions\textquotedblright{} Section (2.3.2) of [2] for the detailed discussion on spatially varying lengthscales and the resultant positive definite kernels.

    - [2] Gibbs, Mark N. "Bayesian Gaussian processes for regression and classification", PhD diss., University of Cambridge, 1998.

Reviewer comment (Theory):
The approach is based on modelling both the objective functions and the user-derived feedback given $\mathbf{x}$ where neither are yet observed. What assumptions tell us that predicting the $\omega$ properties first and then predicting $y$ is easier than predicting $y$ directly? In the synthetic experiments this is clear – the functional relationship between $y$ and the $\omega$ is somehow simpler than the relationship between $y$ and $\mathbf{x}$ directly. It would be nice to clarify and understand these assumptions more carefully.

Author response:
- _**Why predicting the properties fìrst and then predicting $y$ is easier than predicting directly?**_
    - The abstract properties can be simple physical properties or even a combination of multiple physical properties that an expert uses
to reason about why one design is better than another. A well-chosen abstract property captures the input space of a system sufficiently
enough that enable the domain expert to better reason about the output of the system in terms of such higher-order abstract properties. The main objective of augmenting abstract properties into the input space is to capture the inherent transformations of the function space $\mathcal{F}$ that significantly reduce the complexity and thereby simplify the
task of GP modelling, thus a better overall optimization performance. 
    - Furthermore, human expert feedback is not available always, thus we cannot simply use expert-derived feedback as input. However we model this (intermittent) expert feedback using a GP, and that model will act as a proxy for the expert in experiments where no expert feedback is given.

Reviewer comment (Theory):
The theory quoted in this paper from Russo & Van Roy, 2014 may not be directly applicable. The theory assumes the model is a GP, which is not guaranteed with this input-dependent lengthscale. The theory further assumes the GP model itself is fixed, whereas in the paper both GP hyperparameter optimization and updates to the $\omega$-GPs are applied at each step. (Cf. Section 6.3 of the Russo & Van Roy paper)

Author response:
- _**The theory quoted in this paper from Russo \& Van Roy, 2014 may not be directly applicable?**_

    - At every step of a typical BO, the kernel hyperparameters are updated, so technically speaking the model is different at each step. Nevertheless, the results of papers like Russo and Van Roy, while not precisely matched to reality, are considered \textbf{indicative} of performance, typically in big-O form (no one expects exact predictions, just an indication of expected performance). The only real distinction in this case is that our kernel - which in effect includes the posterior mean/variance of the expert models in the kernel definition - varies more radically in the early stages of optimization. We discuss Russo and Van Roy in this sense - a guide to what we might expect, not a limiting framework.

Reviewer comment (Experiments):
The real-world experiments synthesise human feedback by converting some columns of real data into unobserved rank data. No actual human-in-the-loop experiment is conducted.

Author response:
- _**No actual human-in-the-loop experiment is conducted**_
   - Level of expertise required to conduct these experiments is high and access to such real experts is very difficult. Due to the cost and access limitations of experimental facilities, we rely on numerical simulations and simulated data.

Reviewer comment (Novelty):
What existing work has been done on using GP-predicted values as inputs to another GP? Has anyone studied this model? Do we know if it is actually a (mathematical) GP?

Author response:
- _**GP-predicted values as inputs to another GP? Has anyone studied this model?**_

    - Using GP predictive distribution to model another GP has been explored in the past. [1] proposed a two-layered hierarchical model for GP regression that partitions the input data points such that the upper layer is responsible for coarse modelling and the lower layer is responsible for fine modelling. 
    - [1] Park, Sunho, and Seungjin Choi. \textquotedbl Hierarchical Gaussian process regression.\textquotedbl{} In Proceedings of 2nd Asian Conference on Machine Learning, pp. 95-110. JMLR Workshop and Conference Proceedings, 2010.