Paper: Meta-VBO: Utilizing Prior Tasks in Optimizing Risk Measures with Gaussian Processes (ICLR-2024)

Reviewer comment (Reproducibility):
The distribution of Z is supposed to be known. Is it a parameterized distribution that is fitted in practice?

Author response:
We adopt this assumption from previous works on BO of risk measures (Cakmak et al., 2020; Nguyen et al., 2021a;b). In practical applications, we believe one should parameterize and fit the distribution of Z to historical data.

Reviewer comment (Reproducibility):
The definition of the lacing values is unclear. If I understand well, this means that at each iteration the authors pick a value for z (corresponding to the lacing value) and draw an observation \( y_t = f(x_t, z) + \varepsilon \). Is this correct? How are those lacing values computed in practice? I think a paragraph is necessary to explain exactly what is done here. Would it be possible to provide a pseudo-code summarizing the main steps of the algorithm?

Author response:
Your interpretation regarding the selection of z as a lacing value is accurate. Following your suggestion, we have included the pseudocode of the algorithm in Appendix G of the revised paper to facilitate its implementation for the reader. Additionally, a concise discussion on the definition of lacing values is provided in Appendix C.

Reviewer comment (Experiments):
It would have been interesting to see an illustration of the set \(\mathcal{Q}_t\) in the numerical experiments and the computed importance weights, and also to see the evolution of this query set over time. Figure 1 is not easily readable in my opinion.

Author response:
We have followed your suggestion to include an illustration of the set $\mathcal{Q}_t$ and the computed importance weights over several iterations in Appendix H.1 of the revised paper.

Reviewer comment (Experiments):
The authors write that "we want to maximize the size of the query sets." I understand this point since it enables exploiting all the previous tasks. However, it seems that there might exist some tasks that do not bring useful information or bring harmful information. In such case, wouldn't it be more interesting to discard these tasks from the query set? From a computational perspective, it would make more sense to consider fewer previous tasks. Is this correct?

Author response:
Identifying and excluding harmful tasks to improve computational efficiency, especially when dealing with a large number of harmful tasks, is an interesting direction for future exploration. However, this is a challenging task given our current understanding. The difficulty arises from the absence of a concrete measure of a task's harmfulness. Two risk measures may exhibit similarity in certain local regions while diverging in others, a nuance captured by our assignment of priorities to inputs in $\mathcal{Q}_t$.

  On the other hand, the set $\mathcal{Q}_t$ plays a vital role in ensuring the algorithm's robustness against harmful tasks. As illustrated in Figure 6 of the revised paper, where previous task 1 differs from the current task, the reduction in the size of $\mathcal{Q}_t$ diminishes the negative impact of previous task 1 on the optimization of the current task.

Reviewer comment (Evaluation):
Could you comment on the computational cost of the approach?

Author response:
We have included a discussion on the computational complexity of the proposed algorithm, comparing it to V-UCB (BO of risk measures without utilizing any previous tasks), in Appendix G of the revised paper. In summary, our algorithm introduces only an additional linear dependence on the number of previous tasks.

Reviewer comment (Presentation):
Titles of subsection 2.2 and Section 3 are the same.

Author response:
We have revised the titles of Subsection 2.2 and Section 3 to be 'Meta-BO of Risk Measures' and 'Our Proposed Solution to Meta-BO of Risk Measures', respectively.

Reviewer comment (Reproducibility):
Is the code publicly available?

Author response:
We have attached the code in the supplementary materials.

Reviewer comment (Theory):
It is unclear why the $\lambda$ and $\eta$ are proposed to trade off the goals of NQS explicitly. The algorithm, by default, sets $\lambda=0$ and $\eta=1$ as discussed following the theorem 3.7.

Author response:
To assist readers in grasping the essence of our technical approach, we begin by presenting it in a way that starts with basic conceptual ideas accessible to layman readers and then gradually progresses to a rigorous mathematical formulation in Section 3.1. The parameters $\lambda$ and $\eta$ play a crucial role in translating conditions (C1) and (C2) into overlapping and uncertainty conditions. Subsequently, we elucidate the rationale behind choosing $\lambda = 0$ and $\eta = 1$ after a more in-depth discussion on the regret incurred by our proposed algorithm. In contrast, setting them directly and specifically to $\lambda = 0$ and $\eta = 1$ right at the beginning would potentially raise questions regarding alternative formulations of the overlapping and uncertainty conditions (with $\lambda \neq 0$ and $\eta \neq 1$). Hence, we chose to discuss their roles and trade-off first before setting them to specific values. Furthermore, it could be challenging to discuss the trade-off between exploiting previous tasks and exploring current tasks and the size of NQS without parameters $\lambda$ and $\eta$.

Reviewer comment (Experiments):
Figure 3 demonstrates the effectiveness of the priority mechanism. Yet since the priority function relies on counting, it is unclear whether the proposed method is robust to the increase of the portion of harmful tasks in $\tau$.

Author response:
The robustness does not come from the priority assignment; rather, it mainly arises from the construction of NQS and the theoretical guarantee that choosing any input from NQS leads to a no-regret algorithm, which is a unique contribution of our work. Additionally, we have included a visualization of the algorithm in the scenario where the set of previous tasks consists solely of harmful tasks, as illustrated in Figure 6 of the revised paper.

   In Figure 2, note that the sets $\\mathcal{T}\_{\\texttt{harmful-neg-scale}}$ and $\\mathcal{T}\_{\\texttt{harmful-hshift}}$ of previous tasks consist of all harmful tasks (i.e., no useful previous tasks). Hence, we have empirically demonstrated the performance of our algorithm when the portion of harmful tasks is $100\%$ apart from the sublinear cumulative regret proof. This is highlighted in the last paragraph of Section 4: "In the extreme case when $\mathcal{T}$ contains only harmful tasks ...".

Reviewer comment (Theory):
It seems that the construction of NQS basically relies on the upper and lower confidence bounds of the risk measure $\rho_f$ in eq (3). In ordinary BO, the UCB and LCB of the unknown objective function f are the direct equivalence of these confidence bounds for risk measure when the goal is optimized f. Then, an NQS for ordinary meta-BO could be constructed in the same way and could incur a similar regret guarantee. Could the author comment on this extension of NQS in classic meta-BO?

Author response:
This is a very insightful observation! As outlined by the reviewer, we think there is potential to extend NQS to classic meta-BO by substituting the upper and lower confidence bounds of the risk measure with UCB and LCB of the unknown objective function. We will provide a comprehensive discussion of this extension in the revised paper to enhance the impact of our work within the BO community. In the context of risk measures, where a solution for meta-BO incorporating risk measures is lacking, we consider it innovative and practical to position and define the scope of our work as being the first work in the domain of meta-BO focused on risk measures.

Reviewer comment (Presentation):
The layout is intense and could be more reader-friendly. Could the author reduce the discussion over literature in sections 1 and 2 and leave more room for the essential equations, especially those for the assumptions?

Author response:
Given that our work is situated within the domains of BO, meta-BO, and BO of risk measures, we agree with the reviewer's observation about the extensive discussion on the literature and background. However, we also believe that it is crucial to present this background information to ensure readers can fully grasp the equations and assumptions. If the reviewer has suggestions for improving the discussion of any specific assumptions or equations in mind, we would greatly appreciate them. We are committed to enhancing the clarity and coherence of these sections in the revised paper based on any feedback provided.

Reviewer comment (Experiments):
Empirical results are on extremely toy examples.

Author response:
We have incorporated experiments involving 9-dimensional inputs in Appendix H.2 of the revised paper. The outcomes for both VaR and CVaR risk measures, utilizing various sets of previous tasks (including harmful and useful tasks), are presented in Figure 8 of the revised paper. These results are consistent with the findings from other experiments detailed in the paper.

We would like to highlight our experiments with the yacht hydrodynamics and portfolio optimization datasets which makes our experiments section comparable with that of existing works on BO of risk measures.
The difficulty in finding real-world experiments arises the absence of existing meta-BO research that considers risk measures. This results in a lack of previous real-world tasks within the meta-BO literature that incorporate real-world environmental random variables.

Additionally, we enhance the overall comprehensiveness of our experiments section by including the CVaR experimental results in Appendix H of the revised paper.

It is worth mentioning that research on high-dimensional BO is ongoing. One feasible approach to address this challenge is the use of simple techniques, such as random projection, to effectively reduce the input dimension of the optimization problem.

Reviewer comment (Novelty):
The key contribution seems to be the introduction of the idea of a no-regret-query set (NQS). Maybe I missed it, but it feels like it is more generally applicable than the exact problem being considered in this work. If not, perhaps it would be beneficial to know what makes NQS restricted to the setting of risk measures? Also, seems like various other side-information can be used to prioritize samples within NQS.

Author response:
This is an excellent observation! Our original aim was to tackle the unexplored domain of meta-learning for BO of risk measures. This led to the introduction of the novel NQS. From the insightful comment of the reviewer, we acknowledge that this approach proves to be more versatile than initially anticipated. At present, we recognize the potential application of the NQS technique to various existing BO variants within the realm of meta-learning, including:

+ Contextual Gaussian process bandit optimization (Krause et al., 2011).
+ Adversarially robust optimization with Gaussian processes (Bogunovic et al., 2018).

We will incorporate this expanded discussion in the revised version of the paper.

Reviewer comment (Writing):
While I understand that the requirement of noise distribution is similar to assumptions made in the prior work and is not directly related to the proposed method, however, since the paper currently is phrased as a solution for optimizing risk measures, it would be useful for the readers to have the assumption formally stated in Section 2.

Author response:
To enhance clarity, we plan to restate the assumption outlined in the introduction when we delve into Section 2.

Reviewer comment (Experiments):
Can the authors provide some experiments on the scalability of the proposed approach? The toy examples considered all have dimensions less than 5 or 6.

Author response:
We have incorporated experiments involving 9-dimensional inputs in Appendix H.2 of the revised paper. The outcomes for both VaR and CVaR risk measures, utilizing various sets of previous tasks (including harmful and useful tasks), are presented in Figure 8 of the revised paper. These results are consistent with the findings from other experiments detailed in the paper.

We would like to highlight our experiments with the yacht hydrodynamics and portfolio optimization datasets which makes our experiments section comparable with that of existing works on BO of risk measures.
The difficulty in finding real-world experiments arises the absence of existing meta-BO research that considers risk measures. This results in a lack of previous real-world tasks within the meta-BO literature that incorporate real-world environmental random variables.

Additionally, we enhance the overall comprehensiveness of our experiments section by including the CVaR experimental results in Appendix H of the revised paper.

It is worth mentioning that research on high-dimensional BO is ongoing. One feasible approach to address this challenge is the use of simple techniques, such as random projection, to effectively reduce the input dimension of the optimization problem.

Reviewer comment (Presentation):
I think the introduction can be compressed significantly. Currently, the main contribution of the work starts at the end of page 5.

Author response:
As our research is positioned in the domains of BO, meta-BO, and BO of risk measures, we agree with the reviewer about the comprehensive introduction which aims to discuss the literature and background in these 3 domains. Although the technical solution is on page 5, we have attempted to summarize the contribution in the last paragraph of the introduction (Section 1). This serves to provide readers with a preview of the paper's essence before delving into the background, the problem statement, and the technical solution.

Reviewer comment (Experiments):
If \lambda goes near 0, then \eta can go near infinity and the regret guarantee becomes vacuous. This results in two important hyper-parameters of the problem: \eta and \lambda, that would essentially control exploration-exploitation. While 3.3 gives one recommendation, how sensitive are the practical results to the choice of their values?

Author response:
We would like to thank the reviewer for the insightful question. While $\eta$ can go near infinity when $\lambda$ goes near $0$, from equation (7), note that the LHS is bounded from below by $\min_{x' \in \mathcal{X}} \rho_{u_t}(x';\alpha) - \rho_{l_t}(x';\alpha) > 0$ (since the posterior standard deviation is strictly positive). Therefore, we will revise the upper bound of $\eta$ as $\eta \le \min(1/\lambda, (\rho_{u_t}(x_t^+;\alpha) - \rho_{l_t}(x_t^-;\alpha)) / (\min_{x' \in \mathcal{X}} \rho_{u_t}(x';\alpha) - \rho_{l_t}(x';\alpha)))$. As a result, when $\lambda$ goes to $0$, $\eta$ is still finite and the regret guarantee is still meaningful.

In practical applications, we suggest setting $\lambda$ to 0 and $\eta$ to 1 when practitioners believe that previous tasks are useful for the current task. Conversely, if practitioners are risk-averse and believe that previous tasks are predominantly harmful to the current task, they can set $\lambda$ close to 1 and $\eta$ to 1. This adjustment makes the algorithm more akin to GP-UCB, which reduces the use of information from previous tasks. This corresponds to our discussion on the exploration-exploitation trade-off in the paragraph following Lemma 3.1.

Reviewer comment (Theory):
I do not understand the Lacing value choice of the noise variable, and why it is important to obtain the desired result. Given that Theorem 3.2 is the core contribution of the work, I think it would be imperative to explain this assumption in more detail.

Author response:
Given our focus on meta-BO for risk measures, it is essential, at each BO iteration, to query both $x_t$ and $z_t$ as detailed in Section 2.1, an approach that is adopted from existing literature on BO of risk measures. The transfer of information from BO solutions of prior tasks to the current one primarily occurs through the selection of $x_t$. Selecting for specific values for $z_t$ ensures the no-regret property, akin to previous studies on BO of risk measures. The absence of a specific choice for $z_t$ could compromise the guarantee of sublinear regret, regardless of the chosen $x_t$.