Paper: A Unified Framework for Bayesian Optimization under Contextual Uncertainty (ICLR-2024)

Reviewer comment (Experiments):
The authors can consider inspecting why in various cases (in the Infection problem for DRO), TS-BOCU has almost linear regret. It would be interesting to see if there are types of problems that the policy tends to perform badly on. (Perhaps this is connected to the insight that the algorithm resulting from setting $\alpha, \beta > 0$ tends to be more robust?)

Author response:
**On the performance of TS-BOCU on the DRO uncertainty objective in Infection**: While the performance seems not great, we still observe the sublinear property towards the final iterations as the growth of the curve slows down. If we were to run the experiments for more iterations, this sublinear property would be more clear. Also, it may simply be the case that the baselines are very strong in this setting. UCB-BOCU is explicitly designed for the DRO uncertainty objective, whereas a possible reason that UCB-RO performs well is that the RO optimal decision coincides with the DRO optimal decision with this objective function. In any case, the sublinear property still assures us that our algorithm is able to handle the DRO objective as expected, although the relative performance between baselines will certainly vary depending on specific objective function.

Reviewer comment (Theory):
I am a bit confused about the assumption that $\mathcal{X}$ and $\mathcal{C}$ are finite: Is it necessary for the theoretical guarantee (since Section 4.1 mentions that the result can be extended to infinite sets)? In the experiments (for example, with the Hartmann functions), are you constraining the search spaces to be finite? My understanding is that the algorithm can run on continuous search spaces too.

Author response:
**On the finite decision and context spaces**: We have formalized the extension to continuous, infinite decision sets in the new uploaded revision. The regret bound maintains the same order with respect to $T$, and the conditions necessary for sublinear regret remain the same as explained in the body of the paper. Please see our global response titled "Revision for infinite decision sets and intractability of infinite context sets" (https://openreview.net/forum?id=oMNkj4ER7V&noteId=EjvMtBW1sK) for more details. In that post, we have also defended the assumption of a finite context set. To summarize the salient points, DRO with infinite context sets is intractable with general convex distribution distances $d$, and this difficulty leads to the assumption of a finite context set being standard in the DRBO literature. Our experiment results are still from an implementation with a finite decision set, but we imagine that the results would not be too different if the optimization were performed over an infinite decision set instead.

Reviewer comment (Reproducibility):
I think the paper can benefit from extending the discussion at the end of Section 3.1 and offer guidance for setting $\alpha$, $\beta$, and $\epsilon$ in practice.

Author response:
**On guidance for setting** $\alpha$, $\beta$**, and** $\epsilon$ **in practice**: We agree with you that such a discussion would be interesting and useful. However, we also fear that such a discussion would become more subjective and philosophical than technical, and may depend too much on the individual use case. For example, what arguments could be made for a risk-averse learner to optimize the value-at-risk versus the conditional value-at-risk versus the mean-variance tradeoff? We decided in the end that an extended discussion on this topic would be out of the scope of this paper. We leave the choice of uncertainty objective to the practitioner; our goal was to show that the practitioner can choose between a large diversity of uncertainty objectives with a single mental framework and a single algorithm.

Reviewer comment (Theory):
Instead of having both $\alpha$ and $\beta$, can't we follow the formulation of mean-risk tradeoff and only vary the weight for the $\delta(\mathbf{x})$ term, unless, for example, $\alpha = \beta = 1$ gives a different objective than $\alpha = \beta = 0.5$ (assuming the same $\epsilon$)?

Author response:
**On the necessity of both** $\alpha$ **and** $\beta$: Both parameters are necessary, as robust satisficing and worst-case sensitivity have $\alpha=0$ and do not involve the $v(\mathbf x)$ term at all. If only $\beta$ were present, we would not be able to set the weight on the $v(\mathbf x)$ term to $0$ to recover these objectives.

Reviewer comment (Theory):
As I understand, if $\epsilon_t = d(\mathbf{p}_t, \mathbf{p}^*)$ does not approach $0$ with probability $1$, we don't obtain the sublinear regret result. How does this might affect performance in practice? Can we still perform well if $\mathbf{p}_t$ is sufficiently different from $\mathbf{p}^*$? Are there situations where $\mathbf{p}^*$ is very hard to learn?

Author response:
**On what happens if** $d(\mathbf p_t, \mathbf p^*)$ **does not approach** $0$: In practice, even if $\mathbf p_t$ does not approach $\mathbf p^*$, we are still able to get good performance. In our experiments, for simplicity, we keep $\mathbf p_t$, $\mathbf p^*$, and $\epsilon_t > 0$ constants throughout, and we see that TS-BOCU still performs well and displays the sublinear property. The necessary condition for good performance is nuanced: informally speaking, we can perform well as long as the learner is able to reduce the uncertainty at all 'relevant' decision-context points, where 'relevance' is determined by the supports of the distributions within the uncertainty set $\mathcal U_t$. Notice that $\mathbf p^*$ only governs the distribution of contexts observed, and is not a part of the uncertainty objective: only the learner-chosen parameters $\mathbf p_t$ and $\epsilon_t$ are. This means that, if the true underlying function $f$ were completely known to the learner, it does not matter what $\mathbf p^*$ is, the learner would be able to solve the problem exactly. $\epsilon_t \rightarrow 0$ (with the constraint $d(\mathbf p_t, \mathbf p^*) \leq \epsilon_t$) is a sufficient but not necessary condition for good performance. $\mathbf p^*$ gets more difficult to learn as $|\mathcal C|$ increases.

Reviewer comment (Novelty):
The significance is of question to me - though it is good to have a unified form for multiple previously proposed objectives, the unification achieved in this paper seems straightforward (adding two parameters) and the additional technical challenge (e.g., in algorithm design or analysis) is unclear, and it is not clear whether those new objectives are really of significance for practitioners.

Author response:
1. **On the significance of the unification and the analysis**: We believe that the significance of the unification goes beyond adding two parameters, as the right-derivative of the DRO objective at any $\epsilon$ was not known in the BOCU literature to be an uncertainty objective of interest. Tay et. al. (2022) introduced the idea of worst-case sensitivity to DRBO, but the worst-case sensitivity is only the right-derivative at $\epsilon = 0$. Our work is the first to establish the general usefulness of the right-derivative by proving in Proposition 3.1 that the right-derivative at $\epsilon > 0$ is of theoretical interest as well through the robust satisficing objective. We believe that this is a non-trivial and elegant extension of DRBO.  As you have pointed out, it is good to have a unified form for previously unrelated objectives, but the unification was anything but straightforward. 

    Recognizing and formally proving the relevance of the right-derivative was only the first part: the novel analysis of the regret bound for Thompson sampling when the right-derivative is part of the objective was not a straightforward undertaking. As we mention in the paper, the derivative term presents significant challenges: the limit term causes difficulties without a closed form, and the inner term is a difference of worst-case expected values which, informally speaking, `erases' the dependency on the uncertainty set and precludes bounding it in terms of the margin $\epsilon$ as was done for the DRO objective. We encourage you to explore the supplementary material, in particular Lemma A.3 and its auxiliary lemmas Lemma A.4, A.5, A.11, A.12, and A.13, to see the work involved in bounding the DRO first-derivative regret. For rigor, we also prove with Proposition 4.2 that the upper bound sequence $U_t$ used in the proof of the regret bound is well-defined.

Reviewer comment (Novelty):
Is the first-derivative objective only previously proposed for GP, or is also applicable in other areas? Is 3.1 only novel in GP literature, or for the first time also in other areas?

Author response:
2. **On the first-derivative objective**: Sections 2 and 3 on the general framework are general enough to be applicable outside of BO as well; note that the sequential learning setting and GPs are only introduced in Section 4. Proposition 3.1 is an original proposition that, to the best of our knowledge, has not been published anywhere else. It is entirely plausible that a similar result exists in the operations research literature that we are unaware of, and that we have simply re-discovered it. However, given that robust satisficing was only introduced this year (Long et. al., 2023), it seems unlikely.

Reviewer comment (Theory):
Why finite context and action space? These read very limited. Are they also required by prior works?

Author response:
3. **On the finite decision and context spaces**: We have **extended our results to continuous, infinite decision sets in the new uploaded revision**. The regret bound maintains the same order with respect to $T$, and the conditions necessary for sublinear regret remain the same as explained in the body of the paper. Please see our global response titled "Revision for infinite decision sets and intractability of infinite context sets" (https://openreview.net/forum?id=oMNkj4ER7V&noteId=EjvMtBW1sK) for more details. In that post, we have also defended the assumption of a finite context set. To summarize the salient points, DRO with infinite context sets is intractable with general convex distribution distances $d$, and this difficulty leads to the assumption of a finite context set being standard in the DRBO literature.

Reviewer comment (Theory):
Some assumptions are too strong. For example, the assumptions on finite sets of $\mathcal{X}$ and $\mathcal{C}$ are not practical. Moreover, it is unclear why $\boldsymbol p$ is known at the beginning of the optimization. This assumption is not practically meaningful.

Author response:
**On the finite decision and context spaces**: We have **extended our results to continuous, infinite decision sets** in the new uploaded revision. The regret bound maintains the same order with respect to $T$, and the conditions necessary for sublinear regret remain the same as explained in the body of the paper. Please see our global response titled "Revision for infinite decision sets and intractability of infinite context sets" (https://openreview.net/forum?id=oMNkj4ER7V&noteId=EjvMtBW1sK) for more details. In that post, we have also defended the assumption of a finite context set. To summarize the salient points, DRO with infinite context sets is intractable with general convex distribution distances $d$, and this difficulty leads to the **assumption of a finite context set being standard in the DRBO literature**. This should address your concerns about the strengths of these assumptions and the corresponding usefulness of our theoretical results.  
**On the reference distribution $\mathbf p$**: We wish to emphasize that the reference distribution $\mathbf p$ **is a problem parameter chosen by the learner and captures the learner's prior belief of the context distribution, and is not the true distribution** $\mathbf p^*$. It is very possible in real life to obtain a reasonable prior distribution for random contexts, such as from expert knowledge or estimation from historical data. For instance, in the introduction's example of sunlight in farming, the amount of solar radiation is a quantity that has been tracked by meteorological organizations for decades (e.g., https://nsrdb.nrel.gov/). A reasonable reference distribution of solar radiation can be estimated from this data. If the learner has no expert knowledge or historical data, they can simply set the reference distribution at iteration 1 $\mathbf p_1$ to be the uniform distribution with a large margin $\epsilon_1$ to capture the uncertainty, then update their reference distributions $\mathbf p_t$ for $t>1$ based on the observed contexts over the learning procedure. This is the 'data-driven' setting in Kirschner et. al. (2020) described under 'Conditions for sublinear regret' in Sec. 4.  

From a more technical perspective, it is not possible to even define the DRO problem without a reference distribution. Every problem definition under the BOCU framework needs some reference distribution to say anything useful, including BO for expected values (Toscano-Palmarin \& Frazier, 2022), risk-averse BO (Cakmak et al., 2020; Nguyen et al., 2021a;b), and prior work in distributionally robust BO (Kirschner et al., 2020; Nguyen et al., 2020; Tay et al., 2022).

Reviewer comment (Theory):
Since theoretical results are built on the assumptions on finite sets of $\mathcal{X}$ and $\mathcal{C}$, they are limited.

Author response:
**On the finite decision and context spaces**: We have **extended our results to continuous, infinite decision sets** in the new uploaded revision. The regret bound maintains the same order with respect to $T$, and the conditions necessary for sublinear regret remain the same as explained in the body of the paper. Please see our global response titled "Revision for infinite decision sets and intractability of infinite context sets" (https://openreview.net/forum?id=oMNkj4ER7V&noteId=EjvMtBW1sK) for more details. In that post, we have also defended the assumption of a finite context set. To summarize the salient points, DRO with infinite context sets is intractable with general convex distribution distances $d$, and this difficulty leads to the **assumption of a finite context set being standard in the DRBO literature**. This should address your concerns about the strengths of these assumptions and the corresponding usefulness of our theoretical results.

Reviewer comment (Evaluation):
Reasoning and justification behind experimental results are not appropriately provided.

Author response:
**On the "reasoning and justification behind experimental results"**. We have to kindly ask you to elaborate on your concerns. What exactly about the design of the experiments do you not understand or agree with? Is it the choice of 1) objective functions, 2) uncertainty objectives, 3) distribution distances, 4) problem parameters, or 5) baselines? We believe we have made a reasonable effort to compare our proposed algorithm to suitable baselines across a diversity of objective functions, uncertainty objectives, and distribution distances. If there is anything that does not make sense to you, please specify them so that we can discuss them and improve our paper.

Reviewer comment (Presentation):
Table 1 can have a column for the corresponding references. It would help understand and compare diverse algorithms.

Author response:
Your style suggestions have been noted. In the latest revision, we have moved the caption of Table 1 to the top. Due to ICLR's citation style that requires the name and date, including the references in Table 1 would cause it to be too cluttered. We have instead included equation numbers beside each uncertainty objective; the relevant references are always right before each equation.

Reviewer comment (Writing):
Could you explain the intuition and meaning of knowing $\boldsymbol p$ at the beginning?

Author response:
**On the reference distribution $\mathbf p$**: We wish to emphasize that the reference distribution $\mathbf p$ **is a problem parameter chosen by the learner and captures the learner's prior belief of the context distribution, and is not the true distribution** $\mathbf p^*$. It is very possible in real life to obtain a reasonable prior distribution for random contexts, such as from expert knowledge or estimation from historical data. For instance, in the introduction's example of sunlight in farming, the amount of solar radiation is a quantity that has been tracked by meteorological organizations for decades (e.g., https://nsrdb.nrel.gov/). A reasonable reference distribution of solar radiation can be estimated from this data. If the learner has no expert knowledge or historical data, they can simply set the reference distribution at iteration 1 $\mathbf p_1$ to be the uniform distribution with a large margin $\epsilon_1$ to capture the uncertainty, then update their reference distributions $\mathbf p_t$ for $t>1$ based on the observed contexts over the learning procedure. This is the 'data-driven' setting in Kirschner et. al. (2020) described under 'Conditions for sublinear regret' in Sec. 4.  

From a more technical perspective, it is not possible to even define the DRO problem without a reference distribution. Every problem definition under the BOCU framework needs some reference distribution to say anything useful, including BO for expected values (Toscano-Palmarin \& Frazier, 2022), risk-averse BO (Cakmak et al., 2020; Nguyen et al., 2021a;b), and prior work in distributionally robust BO (Kirschner et al., 2020; Nguyen et al., 2020; Tay et al., 2022).

Reviewer comment (Presentation):
It is uncertain whether the ICLR paper format allows it, but the table captions should be located on top of the tables.

Author response:
Your style suggestions have been noted. In the latest revision, we have moved the caption of Table 1 to the top. Due to ICLR's citation style that requires the name and date, including the references in Table 1 would cause it to be too cluttered. We have instead included equation numbers beside each uncertainty objective; the relevant references are always right before each equation.

Reviewer comment (Evaluation):
For the COVID infection problem, I found the results of DRO are not too good. I’m just wondering what are the issues of these cases?

Author response:
3. **On the performance of TS-BOCU on the DRO uncertainty objective in Infection**: While the performance seems not great, we still observe the sublinear property towards the final iterations as the growth of the curve slows down. If we were to run the experiments for more iterations, this sublinear property would be more clear. Also, it may simply be the case that the baselines are very strong in this setting. UCB-BOCU is explicitly designed for the DRO uncertainty objective, whereas a possible reason that UCB-RO performs well is that the RO optimal decision coincides with the DRO optimal decision with this objective function. In any case, the sublinear property still assures us that our algorithm is able to handle the DRO objective as expected, although the relative performance between baselines will certainly vary depending on specific objective function.