From: auai.org/UAI/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: Meta Review: The paper develops a very interesting idea of meta Bayesian optimisation where optimisation over one problem helps to solve another. Reviewers were very positive about this work but also made a number of suggestions. Please take these into account for the final version.

From: auai.org/UAI/2022/Conference/Paper226/Reviewer_vGXz
Q1 Summary and contributions: The authors propose a method for meta-learning Bayesian Optimization (meta-BO), that exploits data available from similar tasks while ensuring robustness against potentially harmful dissimilar tasks. Two approaches are proposed: robust meta-Gaussian process-upper confidence bound (RM-GP-UCB) and RM-GP-Thompson sampling (RM-GP-TS). The theory for both algorithms shows asymptotically no regret. RM-GP-UCB enjoys better theoretical robustness than RM-GP-TS.  


Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: - Clearly and well-written paper.
- Significant theoretical results for both  RM-GP-UCB and RM-GP-TS.
- The authors can optimize weights assigned to previous tasks to reduce the impact of dissimilar tasks.
- Exhaustive experimental validation.
Q4 Main weakness: - Proposed method does not account for the scenario where the meta-functions are shifted or scaled versions of the target function.
Q5 Detailed comments to the authors: - The plots in the paper are too small to be read clearly. They should be made bigger before the paper can be accepted.
Q6 Overall score: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
Q7 Justification for your score: This is a significant contribution as shown by the exhaustive experiments. The paper is clearly  written and the theoretical
results justify the proposed method.
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper226/Reviewer_82m4
Q1 Summary and contributions: This paper proposed two novel Bayesian Optimization (BO) problems that are capable of leverage previous evaluations of potentially related functions and thus lead to potentially better optimization results. For target function f with bounded norm induced by the kernel of the GP prior, high probability finite time regret upper-bounds are established. The proposed algorithms outperform a number of existing BO algorithms in extensive real-world experiments. 
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 4: Excellent: The experimental evaluation is comprehensive and the results are compelling.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: This paper gives novel meta-BO algorithms with provable robust performance guarantee against harmful dissimilar tasks, which is a technically significant result compared to previous works. The experiment results of the new algorithms look satisfactory on a large variety of real-world hyper-parameter tuning tasks, which suggests they are competitive in practice. 

Q4 Main weakness: Overall this paper is satisfactory to me. Some minor questions are listed in Q5.
Q5 Detailed comments to the authors: I have some concerns on whether to emphasize the scalability of the new algorithms. In Theorem 1, N_i’s and d_i’s are assumed to be constants independent with T, then the O(MN^2 + T^2) inference cost is asymptotically not far from O((MN + T)^2) in the single GP case. If we allow N_i’’s to be parameterized in T, then we need additional assumptions on d_i’s, and must take the order of N_i and d_i into consideration when picking nu_t.
Q6 Overall score: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: I have read the main paper and skimmed the proof in the appendix. The score is mainly based on the main paper while the proof also looks okay to me.
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper226/Reviewer_1u3v
Q1 Summary and contributions: Proposes and tests transfer-learning (or multi-task learning) method in Bayesian optimization (BO).  Demonstrates theoretically "no regret" property (of a close approximation of the algorithm used) and compares algorithm favorably to other BO and meta-BO algorithms.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 4: Excellent: The experimental evaluation is comprehensive and the results are compelling.
Q2(5) Reproducibility: 4: Excellent: Key resources (e.g., proofs, code, data) are available and key details (e.g., proof sketches, experimental setup) are comprehensively described for competent researchers to confidently and easily reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: + Proposed method shown to work well on a variety of tasks and against a fair set of "competing" methods
+ No-regret proof
+ Well presented and written
Q4 Main weakness: - Bound used to optimize the weight of each "prior task," yet direct optimization of this bound would result in 0 weight to these tasks.  Suggests bound does not capture necessary characteristics for this purpose.
- No-regret is not surprising given form of the acquisition function (essentially, it converges to a single-task acquisition function with known no-regret properties as T -> infinity)
Q5 Detailed comments to the authors: My main concern, as mentioned above, is that the bound is used to pick w and v (nu?).  However, truly optimizing the bound wrt to these would lead to no weight being placed on the prior problems.  This bound only captures "differences" of the current problem from the prior problems, not "similarity" --- that is how much they differ rather than what information might be transferable.  So, why should we expect it should be used for setting w/v?  The path taken in this paper has to force the solution not to allow 1) all weight to be placed on a single prior problem and 2) that weight to be set all the way to zero arbitrarily quickly.

Given the assumption that N_i = N (for all i), I would expect an experiment in which this was not true (to test the importance of that assumption).  I didn't see one.  This would be a good addition.

Minor points:
The diabetes problem could use a bit more detail (from the appendix -- say the last sentence plus a bit) on how it was transformed into a meta-learning problem so that the results can be gauged better.

I'm uncertain why you get N^{3/2} instead of N in Equation 7.  I've looked this over multiple times and I feel I must be missing something, but I cannot find it.  I see an approximation of N(sqrt(log(N)) + d) which I might make Nd, but not N^{3/2}d.  Feel free to ignore this comment if I'm just missing something obvious.

Multi-fidelity BO is a closely related concept.  I was surprised one of those acquisition functions was not tried (pinning the "fidelity" to be the current task).   At least it would be useful for the reader new to BO to have a mention of how it differs in one of the related works sections.

Theorem 1 mentions lambda.  I did not see it defined prior to this theorem.  Again, ignore this if I just failed my reading comprehension test.

Given that Equations (2) and (3) use the same notation for the acquisition function (and that the method names look very similar), it was confusing to me at a quick first pass.

Q6 Overall score: 8: Strong Accept: Technically strong paper with, with novel ideas, excellent impact on at least one area of AI or high to excellent impact on multiple areas of AI, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: A solid paper on meta-BO.
Q8 Confidence in your score: 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper226/Reviewer_13HA
Q1 Summary and contributions: Conventional BO can only handle each task independently. However, when providing a set of tasks and there exist correlations among the tasks, treating them independently are waste of computing resources and result in inferior performance. Powered by the idea of meta-learning, meta-BO is a relatively new topic. The key idea is how to utilize the correlations of tasks to accelerate the performance of new BO tasks. The authors proposed two provable no-regret meta-BO algorithms.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: Meta-BO is a relatively new topic. The study given in this paper is novel for meta-BO. First, the authors gave theoretical justifications for the no-regret guarantee of their two proposed meta-BO algorithms. Second, the previous meta-BO methods either brutally use the dataset distance to transfer the task or evaluate the dataset distributions. In this word, the authors also provide a principled way to infer the task weight. 
Q4 Main weakness: The material presented in this paper is self-contained and the study given is comprehensive enough. However, though the main surrogate used for BO is GP, there are lots of existing works that use BNN as the surrogate. I am sure that it is hard to provide theoretical guarantees for BNN based BO. But it will be better to include other meta-BO baselines that might use BNN as the surrogate. 
Q5 Detailed comments to the authors: I have no further comment or criticism on this work. I think the authors have done good work. The experiment parts are also detailed, not only including synthetic functions, but also real-world hyper-parameter tunning applications.
Q6 Overall score: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: Meta-BO is an important topic due to the BO itself is a very expensive procedure. Inferring prior knowledge from similar tasks can accelerate the current task is crucial. Previous work did not address or formulate the meta-BO problem in a principled and provable way. This is the main strength of this paper. 
Q8 Confidence in your score: 5: Very confident. I have checked all points of the paper carefully. I am certain I did not miss any aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.