On Provably Robust Meta-Bayesian Optimization
Abstract
Bayesian optimization (BO) has become popular for sequential optimization of black-box functions.
When BO is used to optimize a target function, we often have access to previous evaluations of potentially related functions.
This begs the question as to whether we can leverage these previous experiences to accelerate the current BO task through meta-learning (meta-BO), while ensuring robustness against potentially harmful dissimilar tasks that could sabotage the convergence of BO.
This paper introduces two scalable and provably robust meta-BO algorithms: robust meta-Gaussian process-upper conﬁdence bound (RM-GP-UCB) and RM-GP-Thompson sampling (RM-GP-TS).
We prove that both algorithms are asymptotically no-regret even when some or all previous tasks are dissimilar to the current task, and show that
RM-GP-UCB enjoys a better theoretical robustness than RM-GP-TS.
We also exploit the theoretical guarantees to optimize the weights assigned to individual previous tasks through regret minimization via online learning, which diminishes the impact of dissimilar tasks and hence further enhances the robustness.
Empirical evaluations show that (a) RM-GP-UCB performs effectively and consistently across various applications, and
(b) RM-GPTS, despite being less robust than RM-GP-UCB both in theory and in practice, performs competitively in some scenarios with less dissimilar tasks and is more computationally efﬁcient.
INTRODUCTION
Bayesian optimization (BO) has recently gained immense popularity as an efﬁcient method to optimize black-box functions [Shahriari et al., 2016] and it has found success in a variety of applications such as automated machine learning (ML)
[Snoek et al., 2012], reinforcement learning (RL)
[Wilson et al., 2014], among others.
BO uses a Gaussian process (GP)
[Rasmussen and Williams, 2006] as a surrogate to represent the belief about the objective function and, in each iteration, queries the input parameters that maximize an acquisition function .
In particular, the BO algorithms based on the GP-upper conﬁdence bound (GP-UCB)
[Srinivas et al., 2010] and GP-Thompson sampling (GP-TS)
[Chowdhury and Gopalan, 2017] acquisition functions have been shown to be asymptotically no-regret and perform competitively in practice.
When using BO to optimize a target function , we sometimes have access to a set of evaluations of some potentially related functions.
For example, when using BO for hyperparameter optimization of an ML model trained on a target dataset, we often have access to some previously completed BO tasks using other potentially related datasets [Golovin et al., 2017].
These previous tasks, if similar to the current task, may be exploited to accelerate the current BO task.
However, if some (or even all) previous tasks are in fact dissimilar to the current task, their use may turn out to incorporate harmful information and sabotage the convergence of BO [Feurer et al., 2018].
This begs the question as to whether we can leverage previous tasks to improve the efﬁciency of the current BO task, while ensuring robustness against harmful dissimilar tasks such that they do not affect the trademark no-regret convergence of BO.
Exploiting previous learning experiences to improve the efﬁciency of the current task is the goal of meta-learning [Vanschoren, 2018].
Meta-learning is a broad ﬁeld with various applications in supervised learning [Finn et al., 2017],
RL [Xu et al., 2018], active learning [Pang et al., 2018], among others.
The major challenges in meta-learning include (a) the transfer of information from previous tasks to the current task, and (b) characterization of task similarity which is crucial for identifying harmful dissimilar tasks [Vanschoren, 2018].
The application of meta-learning to BO (or meta-BO ) has been explored by previous studies which differ in how these two challenges are addressed.
Some works, such as multitask BO [Swersky et al., 2013], transfer the information from previous tasks by building a joint GP surrogate using the observations from all previous and current tasks, with the task similarity either represented by meta-features [Bardenet et al., 2013, Yogatama and Mann, 2014] or learned from observations [Swersky et al., 2013, Wang et al., 2018].
These works, however, are limited by the scalability of GP due to including all previous and current observations in a single GP [Feurer et al., 2018].
1 To this end, other recent works transfer information from previous tasks using a more scalable approach:
They build a separate GP surrogate for each individual task and use a weighted combination of either the individual surrogate functions or acquisition functions for query selection [Feurer et al., 2018, Wistuba et al., 2016, 2018].
A more detailed review of related works is presented in Sec. 7.
However, none of the previous works has provided a theoretical performance guarantee to ensure robust performances in the presence of harmful dissimilar tasks.
A robust theoretical guarantee is important for guaranteeing the consistent performances of meta-BO algorithms in various real-world applications, which is crucial for their practical deployment.
To this end, this paper introduces two scalable and provably robust meta-BO algorithms: robust meta-GP-upper conﬁdence bound (RM-GP-UCB) and robust meta-GPThompson sampling (RM-GP-TS).
Both algorithms compute the acquisition function (GP-UCB or GP-TS) for each individual task and select the next query via either a weighted combination (RM-GP-UCB) or in a probabilistic way (RM-GP-TS)
(Sec. 3).
As a result, like the works of Feurer et al.
[2018], Wistuba et al.
[2016, 2018], a separate GP surrogate is built for each previous task, making our algorithms scale well in the number of meta-tasks and observations in each meta-task.
Our major contributions include: Firstly , we prove robust theoretical convergence guarantees for both RM-GP-UCB and RM-GP-TS (Sec. 4).
In particular, both algorithms are asymptotically no-regret for any given set of previous tasks, i.e., even if some or all previous tasks are dissimilar to the target task.
Moreover, we show that RM-GP-UCB enjoys a superior robustness guarantee compared with RM-GP-TS (Sec. 4.2).
Secondly , to further enhance our robustness against dissimilar tasks, we exploit the theoretical guarantees to learn the task similarity (and hence identify dissimilar tasks) in a principled way, by minimizing the regret upper bounds via a computationally cheap online learning algorithm known as FollowThe-Regularized-Leader (Sec. 5).
Lastly , we use extensive empirical evaluations to show that: RM-GP-UCB performs effectively and consistently across a wide range of tasks
; RM-GP-TS, despite under-performing in adverse scenarios (i
e., when
a large number of previous
tasks
are dissimilar)
performs competitively in some favorable cases with less dissimilar tasks and is much more computationally efﬁcient.
Of note, our theoretical and empirical comparisons between
RM-GP-UCB and RM-GP-TS may provide useful insights for other meta-BO algorithms in general (and potentially for other related algorithms such as meta-RL) in terms of the relative strengths and weaknesses of UCB- and TS-based meta-learning algorithms.
BACKGROUND AND PROBLEM FORMULATION
Bayesian Optimization.
This work tackles the problem of sequentially maximizing an unknown function f : D → R .
In each iteration t = 1 , . . .
, T , an input x t ∈ D (a D ≥ 1dimensional vector) is queried to yield y t (cid:44) f
( x t ) + (cid:15) where (cid:15) ∼ N (0 , σ 2 ) is a Gaussian noise with variance σ 2 .
The performance of BO is typically measured by cumulative regret : R T (cid:44) (cid:80) t =1 ,...,T [ f ( x ∗ ) − f ( x t )] where x ∗ ∈ arg max x ∈D f
( x ) is a global maximizer of f .
It is desirable for a BO algorithm to achieve no regret by making its R T grow sublinearly such that its simple regret S T (cid:44) min t =1 ,...,T [ f ( x ∗ ) − f ( x t )] ≤ R T /T goes to 0 asymptotically.
During BO, we model the belief about f using a Gaussian process (GP) { f ( x ) } x ∈D .
That is, any ﬁnite subset of { f ( x ) } x ∈D follows a multivariate Gaussian distribution
[Rasmussen and Williams, 2006].
A GP is fully speciﬁed by its prior mean µ ( x ) and kernel function k ( x , x (cid:48) ) , and we assume w.l.o.g. that µ ( x ) =
and k ( x , x (cid:48) ) ≤ 1 ∀ x , x (cid:48)
∈ D .
We focus on the widely used
Squared Exponential (SE) kernel.
Given T noisy observations y T (cid:44) [ y t ] (cid:62) t =1 ,...,T at inputs x 1 , . . .
, x T , the posterior
GP belief of f at input x ∈ D is Gaussian with the following posterior mean and variance:
[Equation]
where K T
[Equation] [Equation]
[Equation]
...,T , λ is a regularization parameter.
Meta-Bayesian Optimization.
We refer to the function f being maximized as the target function and the functions
f i for i = 1 , . . .
, M of the M previous tasks as meta-functions .
We use target task/observations and metatasks/observations in a similar manner.
All functions are deﬁned on the same domain D which is assumed to be discrete for simplicity, but the theoretical results can be easily generalized to continuous domains following the analysis of previous works [Chowdhury and Gopalan, 2017, Srinivas et al., 2010].
We assume that f and all f
i ’s lie in the reproducing kernel Hilbert space (RKHS) associated with the kernel k such that their norm induced by the RKHS is bounded:
[Equation]
≤ B, ∀ i = 1 , . . .
, M .
This assumption intuitively suggests that the target and meta-functions have the same degree of smoothness.
Same as the work of Wang et al.
[2018] which has also performed theoretical analysis of a meta-learning algorithm for BO, we also assume that all meta- and target observations are corrupted by a Gaussian noise (cid:15) ∼ N (0 , σ 2 ) with variance σ 2 .
The number of observations from meta-task i is a constant denoted as N i , and
N (cid:44) max i =1 ,...,M N i .
x i,j and y i,j represent the j -th input and noisy output of meta-task i respectively.
We deﬁne the function gap
d i (cid:44) max j =1 ,...,N i (cid:12)(cid:12) f
( x i,j ) − f i ( x i,j ) (cid:12)(cid:12) < ∞ which represents the maximum difference between the function values of f and f i at any corresponding input x i,j of meta-task i .
Note that for a given set of meta-observations for meta-task i , the function gap
d i is an unknown constant characterizing the similarity between meta-task i and the target task: a smaller function gap implies a stronger similarity.
ROBUST META-BAYESIAN OPTIMIZATION
The acquisition function (2) adopted by RM-GP-UCB in iteration t is a weighted combination of M + 1 individual
GP-UCB acquisition functions [Srinivas et al., 2010] for the target task and the M meta-tasks, each of which is calculated using the observations from a particular task:
[Equation]
In (2), µ t − 1 ( x ) and σ t − 1 ( x ) represent, respectively, the GP posterior mean and standard deviation (1) at x calculated using the target observations from iterations 1 to t − 1 .
µ i ( x ) and σ i ( x ) are computed using all meta-observations from meta-task i .
β t > 0 and τ > 0 will be deﬁned in Sec. 4.
ν t ∈ [0 , 1] can be interpreted as the overall weight given to all meta-tasks in iteration t and should be chosen to be non-increasing in t , which enforces the impact of metatasks in (2) to be non-increasing.
The meta-weights ω
i ’s can be understood as the weights assigned to individual metatasks.
Note that since the dataset used to calculate µ i ( x ) and σ i ( x ) is ﬁxed with size N i , the matrix inversion in (1) (i.e., the computational bottleneck for GP) can be pre-computed.
So, after T iterations, RM-GP-UCB incurs O
( T 3 ) time for covariance matrix inversion (since only the target covariance matrix of size T × T needs to be inverted) and O ( MN 2 +
T 2 ) time during predictive inference, which are less than the respective O (( MN + T ) 3 ) and O (( MN + T ) 2 ) time when all observations are included in a single GP.
In practice, the total number of BO iterations ( T ) is usually small, therefore, the differences between these corresponding computational costs can be large, especially when M and N are large.
Hence, RM-GP-UCB is scalable in the number of metatasks ( M ) and observations in each meta-task ( N ).
The acquisition function of RM-GP-TS is deﬁned as:
[Equation]
with probability 1 −
[Equation]
probability
[Equation]
in which f t is a function sampled from the GP posterior of the target task:
[Equation]
and f ti is sampled from the GP posterior of meta-task i : f ti ∼ GP (cid:0) µ i ( · ) , τ 2 σ 2 i ( · ) (cid:1) .
Using approximation techniques such as random Fourier features (RFF) approximation [Rahimi and Recht, 2008] (which we use in all our experiments), the functions f t and f ti ’s can be sampled efﬁciently, hence making RM-GP-TS computationally efﬁcient (as we will demonstrate in Sec. 6).
Moreover, since the meta-observations of every meta-task is ﬁxed, the use of approximation techniques such as RFF allows the functions f ti ’s to be sampled beforehand before the algorithm starts.
Refer to Appendix D.5 for more details on RM-GP-TS.
In iteration t of either RM-GP-UCB or RM-GP-TS (Algorithm 1), we ﬁrst optimize the meta-weights and update ν t (Sec. 5.2), which corresponds to line 2 of Algorithm 1.
Next, the input x t is selected by maximizing the acquisition function (2) (RM-GP-UCB) or (3) (RM-GP-TS), after which we query x t and use the newly collected ( x t , y t ) to update the
GP posterior belief (1).
Algorithm 1 RM-GP-UCB/RM-GP-TS 1: for t = 1 , 2 , . .
, T do 2:
Update ω
i for i = 1 , . . .
, M via online meta-weight optimization and update ν t (Sec. 5.2) 3:
x t ← arg max x ∈D ζ UCB t ( x ) (for RM-GP-UCB) (2), or x t ← arg max x ∈D ζ TS t ( x ) (for RM-GP-TS) (3) 4:
Query x t to observe y t , and update GP posterior belief (1) using ( x t , y t ) 5: end for
THEORETICAL ANALYSIS 4.1 RM-GP-UCB
Theorem 1 presents an upper bound on the cumulative regret of RM-GP-UCB (proof in Appendix A).
Theorem 1 (RM-GP-UCB) .
Let δ ∈ (0 , 1) .
Denote by γ t the maximum information gain about f from observing any set of t observations.
If RM-GP-UCB is run with: λ = 1 + 2 /T , β t = B + σ (cid:112) 2( γ t − 1 + 1 + log(4 /δ )) , τ = B + σ (cid:112) 2( γ N + 1 + log(4 M/δ )) , ν t ∈ [0 , 1] and ν t +1 ≤ ν t , ω i ≥ 0 and (cid:80) Mi =1 ω i = 1 .
Then, with probability of ≥ 1 − 3 δ/ 4 ,
R T ≤ 2( α + τ )
T (cid:88) t
[Equation]
= (cid:101)
O
[Equation]
d
[Equation]
T (cid:88)
[Equation]
(cid:1)
[Equation]
(4) where
[Equation]
and
[Equation]
The second term γ T √ T in the regret upper bound (4) grows sub-linearly for the SE kernel for which γ T =
O ((log T ) D +1 ) .
Therefore, if ν t is designed such that ν t → 0 as t → ∞ , the ﬁrst term also grows sub-linearly and hence RM-GP-UCB is asymptotically no-regret.
Theorem 1 holds for a given set of meta-tasks with ﬁxed yet unknown d i ’s. Note that we do not impose assumptions on the values of d i ’s, i.e., the similarities between the metaand target tasks.
Therefore, Theorem 1 gives a robust regret upper bound which holds for any given set of meta-tasks.
In other words, even in adverse scenarios where some or all meta-tasks are extremely dissimilar to the target task (i.e., when some or all d i ’s are very large), RM-GP-UCB is still asymptotically no-regret, which indicates the robustness and generality of our algorithm.
This provides an assurance about the worst-case behavior in any given scenario.
In our proof, the key step (Lemma 3 in Appendix A) is to upper bound (by α in Theorem 1) the overall error induced by the use of any given set of meta-observations, instead of the target observations at the same corresponding input locations, when calculating the acquisition function (2).
These interpretations also explain the dependence of α , hence the regret bound, on d i and N i : Larger function gaps increase the error resulting from the use of the meta-observations, and a larger number of meta-observations also inﬂates the worst-case upper bound by accumulating the individual errors.
Of note, a limitation of our regret upper bound (Theorem 1) is that it does not reﬂect the beneﬁt of the use of the meta-tasks when they are indeed similar to the target task.
Next, we use our theoretical analysis to give some insights on how the meta-tasks, if similar to the target task, help improve the convergence of our algorithm.
Meta-tasks Can Improve the Convergence by Accelerating Exploration.
In addition to characterizing the worstcase behavior, we also use our theoretical analysis to illustrate how meta-tasks can help RM-GP-UCB converge faster than standard GP-UCB.
As we have proved in Appendix A.3, at the early stage of the algorithm, the meta-tasks (if similar to the target task) can help RM-GP-UCB obtain a smaller regret upper bound than GP-UCB by reducing the uncertainty at the selected input .
Equivalently, the additional information from the meta-tasks allows RM-GP-UCB to reduce the degree of exploration at the early stage .
Since initial exploration of BO usually incurs large regrets, less exploration results in smaller regrets.
At later stages when ν t becomes close to 0 , RM-GP-UCB converges to no regret at a similar rate to GP-UCB (i.e., the second term γ T √ T in the regret upper bound (4) dominates).
4.2 RM-GP-TS
Theorem 2 gives an upper bound on the cumulative regret of RM-GP-TS (proof in Appendix B).
Theorem 2 (RM-GP-TS) .
Deﬁne d (cid:48)
i (cid:44) max x ∈D | f ( x ) − f i ( x ) | .
With the same parameters as those deﬁned in Theorem 1, we have that with probability of at least 1 − 3 δ/ 4 ,
[Equation]
Note that by deﬁnition, we have that d (cid:48)
i ≥ d i , ∀ i .
Similar to RM-GP-UCB, as long as ν t is chosen such that ν t →as t → ∞ and
that ν t
= o (1 / √ γ t ) , all three terms in
Theorem 2 are sub-linear (for the SE kernel).
That is, RMGP-TS is also asymptotically no-regret for any set of metatasks, even when some or all meta-tasks are dissimilar to the target task.
Moreover, comparing the extra terms in the regret upper bounds resulting from the use of the meta-tasks for both RM-GP-UCB (i.e., the ﬁrst term of equation (4) in Theorem 1) and RM-GP-TS (i.e., the ﬁrst two terms of Theorem 2) reveals that compared with RM-GP-UCB,
RM-GP-TS suffers from a worse extra dependence on T due to the meta-tasks.
Speciﬁcally, while the ﬁrst terms of
Theorems 1 and 2 have the same dependence on T , the second term of Theorem 2 introduces an extra dependence on T which dominates the ﬁrst term.
This suggests that in adverse scenarios with a large number of dissimilar tasks,
RM-GP-TS may suffer from a worse convergence than RMGP-UCB.
In other words, RM-GP-UCB enjoys a better theoretically guaranteed robustness against dissimilar tasks.
4.3 PRACTICAL IMPLICATIONS
Besides the theoretical insights, Theorems 1 and 2 also provide two natural hints to the practical algorithmic design.
Firstly, note that both Theorems hold for all choices of metaweights ω i ’s.
Therefore, we have the ﬂexibility to choose the optimal ω
i ’s (i.e., learn the task similarity) by minimizing the regret upper bounds in Theorems 1 and 2.
Secondly, the ﬁrst term in Theorem 1 suggests that we can lower the regret by making ν t (i.e., the inﬂuence of the meta-tasks) decay faster if α in Theorem 1 (i.e., an upper bound on the error produced by using the meta-tasks) is larger.
The same reasoning applies to Theorem 2, i.e., we can decay ν t faster if (cid:80) i
=1 ,...,M ω i d (cid:48)
i in Theorem 2 is larger.
Both design choices can further strengthen the robustness of our algorithms against dissimilar meta-tasks by lessening their impact.
Unfortunately, they both require the values of the function gaps
d i ’s which are unavailable.
To this end, we devise a principled technique to estimate upper bounds on the function gaps, which is presented in the next section.
ONLINE META-WEIGHT OPTIMIZATION
In this section, we ﬁrst introduce a principled technique for estimating high-probability upper bounds on the function gaps (Sec. 5.1) that, when combined with Theorems 1 and 2, naturally yields a principled method for optimizing the metaweights through regret minimization via online learning.
5.1 ONLINE ESTIMATION OF FUNCTION GAPS
Inspired by the conﬁdence region constructed by GPUCB [Srinivas et al., 2010, Chowdhury and Gopalan, 2017] that contains the target function with high probability, after t ≥ 1 target observations have been collected, deﬁne
[Equation]
where x i,j is the j -th input of meta-task i , β t +1 is previously deﬁned in Theorem 1, and U t,i,j and L t,i,j can be interpreted, respectively, as the upper and lower conﬁdence bounds of f at x i,j after t iterations.
Lemma 2 (Appendix
A) implies that with probability of at least 1 − δ/ 4 ( δ is deﬁned in Theorem 1): L t,i,j ≤ f ( x i,j ) ≤ U t,i,j , ∀ t, i, j .
Consequently, the following result gives high-probability upper bounds on the function gaps (proof in Appendix C.1):
Lemma 1.
With probability of at least 1 − δ ,
[Equation]
≤
[Equation] [Equation]
+ max j =1 ,...,N i (cid:2) max {| y i,j − U t,i,j | , | y i,j − L t,i,j |} (cid:3) (cid:44) d i,t , for t = 1 , . .
, T and i = 1 , . .
, M .
Unlike d i , d i,t can be efﬁciently calculated as its incurred time is linear in both M and N .
5.2 ONLINE META-WEIGHT OPTIMIZATION THROUGH REGRET MINIMIZATION
In this section, we focus on RM-GP-UCB since the analysis for RM-GP-TS (deferred to Appendix C.4) is similar and leads to the same update rules for ω
i ’s and ν t .
Combining
Lemma 1 and Theorem 1 allows us to derive the following result for RM-GP-UCB (proof in Appendix C.2):
Proposition 1 (RM-GP-UCB) .
With probability of ≥ 1 − δ ,
[Equation]
where ω (cid:44) [ ω i ] i =1 ,...,M , l t (cid:44) [ l i,t ]
[Equation]
=1 ,...,M , and l i,t (cid:44)
N i (
[Equation]
N
[Equation] [Equation]
+
[Equation]
i
[Equation]
Note that l t can be efﬁciently computed after the t -th observation is collected.
The regret upper bound in Propositiondepends on ω
i ’s only through the term (cid:80)
Tt =1 ω (cid:62) l t which can be minimized to derive the optimal meta-weights.
This constitutes an online learning problem with linear loss function and its solution ω constrained to a probability simplex.
An additional entropic regularization term is usually preferred so as to encourage a solution with a large entropy to stabilize it [Bubeck, 2011].
This corresponds to encouraging the meta-weights to spread across a large number of meta-tasks, in order to discover as many similar meta-tasks as possible.
As a result, by using 1 /η ( η > 0 ) as the regularization parameter, the optimal ω in iteration t > 1 is obtained by solving the following optimization problem:
[Equation]
subject to the constraints: ω (cid:48)
i ≥ 0 , ∀ i and (cid:80)
Mi =1 ω (cid:48)
i = 1 .
When t = 1 , the optimal ω follows from optimizing only the entropic regularization term, thus naturally entailing the uniform distribution ω i = 1 /M, ∀ i .
Consequently, (6) corresponds exactly to the online learning algorithm called
Follow-The-Regularized-Leader with an entropic regularizer [Bubeck, 2011] where η represents the learning rate.
Its optimal solution in iteration t can be derived via Lagrange multiplier (Appendix C.3) as
[Equation]
for i = 1 , . . .
, M where (a) follows from assuming that all
N i ’s are close to N for simplicity.
With this simpliﬁcation, the ﬁrst (noise-correction) term in the expression of d i,t from Lemma 1 also cancels out, thus leading to a neat and elegant update rule for ω i which we use in all our experiments.
As is evident from (7), the update of ω i ’s in each iteration only involves computing d i,t ’s (incurring O ( MN ) time), adding one term to the summation on the exponent ( O ( M ) time), and a normalization step ( O ( M ) time), all of which are computationally cheap.
Intuitively, (7) assigns small weights to meta-tasks with a large cumulative estimated function gap which implies a less similar meta-task.
In addition, d i,t from Lemma 1 also allows for the estimation of an upper bound on α (Theorem 1) in each iteration (i.e., by simply replacing d i with d i,t ) and thus facilitates an adaptive selection of ν t , as mentioned in Sec. 4.
Speciﬁcally, we set ν 1 = 1 and ν t =
ν t − 1 × min( r, ( (cid:80) Mi
=1 ω i d i,t ) − (cid:15) ) for t > 1 , in which we have dropped the constants independent of d i,t .
r ∈ (0 , 1) represents the minimum decaying rate to ensure the monotonic decay of ν t such that RM-GP-UCB is no-regret (Sec. 4.1).
(cid:15) > 0 controls the aggressiveness of the adaptive decay such that a larger (cid:15) results in a faster decay.
With this scheme, when the overall estimated function gaps are larger (the meta-tasks are dissimilar), ν t decays faster and thus the impact of the meta-tasks vanishes more quickly.
Importantly, when optimizing the values of ω
i ’s and ν t as described above, we have taken into account the limitation of our regret upper bounds (i.e., they do not reﬂect the beneﬁt of the use of the meta-tasks, Sec. 4.1) and hence incorporated additional practical considerations.
Speciﬁcally, we have optimized the ω
i ’s with an additional entropic regularization term to encourage the ω i ’s to spread across a large number of meta-tasks, and optimized ν t such that it decreases faster if α (i.e., an upper bound on the error induced by the use of the meta-tasks) is larger.
EXPERIMENTS AND DISCUSSION
We use extensive real-world experiments to compare our RM-GP-UCB and RM-GP-TS with (1) standard
GP-UCB, two other GP-based scalable meta-BO algorithms: (2) ranking-weighted Gaussian process ensemble (RGPE)
[Feurer et al., 2018] and (3) transfer acquisition function (TAF)
[Wistuba et al., 2018], (4) multitask
BO (MTBO)
[Swersky et al., 2013], and (5) the method from [Wang et al., 2018] named point estimate meta-BO (PEM-BO).
Since MTBO is relatively not scalable (Sec. 1), we only apply it to those experiments with relatively small number of meta-tasks and observations for which MTBO is still computationally feasible.
We compare with PEMBO
[Wang et al., 2018] in the experiment that is most favorable for this algorithm, i.e., with the largest number of meta-observations and a discrete domain (refer to Sec. 6. for more details).
We set η = 1 /N , (cid:15) = 0 .
7 and r = 0 .in
all real-world experiments to demonstrate the robustness of our algorithm against the choice of these parameters.
In practice, the upper bound on the function gap, d i,t , from
Lemma 1 may be too conservative; so, we replace the outer max operator over j = 1 , ..., N i with the empirical mean in our experiments.
4 Some details and results are deferred to Appendix D due to lack of space.
All error bars represent standard errors.
Our code is available at https: //github.com/daizhongxiang/meta-BO .
[Figure]
6.1 SYNTHETIC EXPERIMENTS
We ﬁrstly explore the effectiveness of our online metaweight optimization (Sec. 5) and the impact of different algorithmic parameters by optimizing synthetic functions drawn from GPs.
For each objective function, we construct
M = 4 meta-tasks with N = N i =
20 meta-observations each.
The function gaps are chosen as d 1 = d 2 = 0 .and
d 3 =
d 4 = 4 . 0 such that the last 2 meta-tasks are dissimilar to the target task.
Fig.
1a plots the simple regrets averaged over 20 randomly drawn synthetic functions, with ηN = 1 . 0 , (cid:15) = 0 .
7 , and r = 0 .
The ﬁgure shows that RM-GP-UCB with online meta-weight optimization signiﬁcantly outperforms RM-GP-UCB with ﬁxed metaweights ( ω i = 1 / 4 for all i ).
Fig.
1b plots the meta-weights optimized by RM-GP-UCB for the red curve in Fig.
1a, showing that the weights given to the last two meta-tasks which are dissimilar to the target task are rapidly reduced.
These results verify the effectiveness of online meta-weight optimization in reducing the impact of dissimilar meta-tasks.
We also investigate the impact of η and (cid:15) .
Fig.
1c shows the performances of different values of η , with ﬁxed (cid:15) = 0 .
7 and r = 0 .
The ﬁgure demonstrates that an excessively small η (purple curve) negatively impacts the performance, since
RM-GP-UCB is unable to quickly reduce the weights of dissimilar meta-tasks (Fig. 4a in Appendix D.1).
Moreover, an overly large η is also slightly detrimental (green curve) since it rapidly assigns a large weight to one of the two useful meta-tasks (Fig.
4c in Appendix D.1), thus failing to utilize the other useful meta-task.
Fig.
1d illustrates the impact of (cid:15) when all function gaps are large: d i = 8 .
for all i .
The ﬁgure shows that even when all meta-tasks are dissimilar, our adaptive selection of ν t is able to diminish their negative impact and allow RM-GP-UCB to perform comparably
to
GP-UCB.
Furthermore, in this adverse scenario, a faster decline of the impact of the meta-tasks (i.e., faster decay of η t via larger (cid:15) ) leads to slightly better performance.
6.2 REAL-WORLD EXPERIMENTS
Hyperparameter Tuning for Convolutional Neural Networks (CNNs).
We apply meta-BO to hyperparameter tuning of ML models with the previous tasks using other datasets as the meta-tasks.
We tune 3 hyperparameters of
CNNs using 4 widely used image datasets: MNIST, SVHN,
CIFAR- 10 and CIFAR- 100 .
Speciﬁcally, in each experiment, one of the four datasets is selected to produce the target function f which maps a hyperparameter setting to a validation accuracy obtained using this dataset.
The meta-observations are generated from 3 independent BO tasks (each with 50 iterations) using the other 3 datasets, i.e., M = 3 and N i = 50 for i = 1 , 2 , 3 in all 4 experiments.
The results for MNIST and CIFAR- 10 are plotted in
Figs. 1e and 1f while the remaining results are shown in Appendix D.2 (Fig. 6).
The results show that RM-GP-UCB is the only method that consistently performs well in all tasks, and that RM-GP-TS performs much better than RM-GPUCB (and other methods) for MNIST, yet worse in the other tasks.
We have also adopted the Omniglot dataset [Lake et al., 2015] commonly used in meta-learning, for which
RM-GP-UCB performs the best (Fig. 7, Appendix D.2).
Non-stationary Bayesian Optimization.
Meta-BO can be naturally applied to non-stationary BO problems in which the unknown objective function evolves over time since the previous (outdated) observations can be treated as the metaobservations.
We consider here automated ML for clinical diagnosis.
As the data from new patients becomes available regularly, clinicians often need to periodically update the dataset and re-run hyperparameter optimization for the
ML model used for clinical diagnosis.
This stimulates the question as to whether the previous hyperparameter tuning tasks using the outdated patients data can help accelerate the current task.
We consider the problem of diabetes prediction [Smith et al., 1988] with logistic regression (LR) and tune 3 LR hyperparameters.
We create 5 progressively growing datasets (including the full dataset), treating (the hyperparameter tuning task using) the full dataset as the target task and the 4 smaller datasets as the meta-tasks.
Speciﬁcally, the entire dataset consists of 768 data instances, among which 77 instances are set aside to measure the validation accuracy.
The sizes of the 5 progressively growing training datasets (i.e., corresponding to the 4 meta-tasks and the target task, respectively) are 138, 276, 414, 552, and 691.
The results (Fig.
2a) show that RM-GP-TS outperforms all other methods in this task.
Moreover, we also compare the runtime of different methods in Fig.
2b: RM-GP-TS is signiﬁcantly more efﬁcient than all other methods, and the methods building separate GP surrogates for different tasks (i.e. RM-GP-UCB, RGPE and TAF) are more efﬁcient than MTBO which includes all observations in a single GP (Sec. 1).
Hyperparameter Tuning for Support Vector Machines
(SVMs).
We also tune the hyperparameters of SVMs using a tabular benchmark dataset [Wistuba et al., 2015a] which has also been adopted by RGPE [Feurer et al., 2018].
The benchmark was constructed by evaluating a ﬁxed grid of 288 SVM hyperparameter conﬁgurations using 50 diverse datasets (i.e., containing many dissimilar tasks).
We follow the setting used by RGPE [Feurer et al., 2018]: In every trial, we ﬁx one of the tasks as the target task, and the remaining
M = 49 tasks as the meta-tasks; for every meta-task i , we randomly select N i = 50 hyperparameter conﬁgurations as the meta-observations.
The results in Fig.
2c show that our
RM-GP-UCB performs comparably to RGPE, outperforming the other methods; RM-GP-TS performs unsatisfactorily in this experiment with diverse tasks.
Of note, this experiment has the most favorable setting for PEM-BO [Wang et al., 2018] because (a) PEM-BO has been shown to require a massive set of meta-observations ( ≥ 5000 ) to perform well [Wang et al., 2018], and this experiment has the largest number ( 49 × 50 = 2450 ) of meta-observations among all experiments; (b) the domain here is discrete, which is much easier for the application of PEM-BO.
Human Activity Recognition (HAR).
HAR using mobile devices has promising applications in various domains such as healthcare [Reyes-Ortiz et al., 2013].
When optimizing the conﬁgurations (hyperparameters) of the activity prediction model (ML model) for a subject, the previous optimization tasks for other subjects might be helpful.
However, cross-subject transfer in HAR is challenging due to high individual variability [Soleimani and Nazerfard, 2019], which makes HAR suitable for evaluating the robustness of a metaBO algorithm against dissimilar meta-tasks.
We use the data collected through mobile phones from 30 subjects performing 6 activities and use support vector machines (SVM) for activity prediction.
Every task corresponds to tuning 2 SVM hyperparameters for a subject.
We run a separate BO (iterations) for each of the 21 subjects to generate the metaobservations ( M = 21 , N i = 30 for i = 1 , . .
. , 21 ) and use the other 9 subjects for validation.
The results are shown in
Fig.
2d (averaged over the 9 subjects, each further averaged over 5 random initializations), in which RM-GP-UCB delivers the best performance, followed by RGPE; RM-GP-TS
[Figure]
again fails to perform effectively, suggesting that it is less robust against the individual variability in HAR.
Policy Search for Reinforcement Learning (RL).
When optimizing the RL policy of an agent in an environment, the agent’s experience in other related environments may help to make learning more efﬁcient
[Duan et al., 2016, Wang et al., 2016].
We apply meta-BO to policy search in RL to maximize the cumulative rewards in an episode, using the
Cart-Pole environment from OpenAI Gym [Brockman et al., 2016] with 8 policy parameters.
We simulate different environments by setting the agent to different initial states.
In particular, we choose M = 10 different initial states, among which the majority (i.e., 7 ) are randomly generated (i.e., dissimilar meta-tasks) and the other 3 are designed to be close to the initial state of the target task so that they are similar to the target task.
An independent BO task with 50 iterations is run for every initial state, i.e., N i = 50 for i = 1 , . . .
Figs. 2e and 2f plot the (normalized) cumulative rewards of different algorithms and their learned meta-weights for the 3 similar meta-tasks.
The results show that RM-GP-UCB achieves the best performance (Fig. 2e), and it is more effective than RGPE and TAF at identifying the 3 similar meta-tasks (Fig. 2f).
RGPE and TAF fail to correctly identify similar meta-tasks because they learn the meta-weights based on how accurately each GP surrogate predicts the pairwise ranking of the target observations (more details in Sec. 7).
However, in the Cart-Pole environment, many target observations have equal values, which confuses the pairwise ranking and makes the learned meta-weights unreliable.
RM-GP-TS again only performs comparably with standard GP-UCB (Fig. 2e).
6.3 EXPERIMENTAL DISCUSSION
In most experimental results (Figs. 1 and 2), the performance advantage of RM-GP-UCB is most evident at the initial stage.
This is likely to corroborate our theoretical insights that the meta-tasks can help improve the convergence of RM-GP-UCB at the initial stage by reducing the degree of exploration (Sec. 4.1).
A potential limitation of our online meta-weight optimization (Sec. 5) is that it does not account for the scenario where the meta-functions are shifted or scaled versions of the target function.
However, note that in some scenarios, the scale of the meta-functions is informative about task similarity and thus should not be removed.
For example, in our clinical diagnosis (i.e., non-stationary
BO) experiment, the more recently completed meta-tasks (with larger training set, smaller validation errors, and thus smaller function gaps) are expected to be more similar to the target task.
Furthermore, as demonstrated by the green curve in Fig.
1a, in some cases, even though the meta-weights are not optimized, RM-GP-UCB still performs favorably.
This implies its robustness against mis-speciﬁcation of the metaweights.
RM-GP-UCB is the only method that consistently outperforms standard GP-UCB in all experiments (Figs. 1 and 2), whereas other methods perform either comparably with or worse than GP-UCB in some experiments (e.g., RGPE in
Figs. 1e and 2a, TAF in Figs.
1e, 1f and 2d).
This might be attributed to RM-GP-UCB’s theoretically guaranteed robustness against dissimilar meta-tasks (Sec. 4) and its ability to diminish their impact in a principled way (Sec. 5).
In particular, RM-GP-UCB performs signiﬁcantly better than
RM-GP-TS in those experiments with a large number of dissimilar meta-tasks (Figs.
2c-e), which may be explained by
RM-GP-UCB’s better theoretically guaranteed robustness against dissimilar meta-tasks than RM-GP-TS (Sec. 4.2).
However, Figs.
1e-f and Fig.
2a show that RM-GP-TS performs competitively in some experiments with more favorable settings (i.e., less dissimilar meta-tasks), which might result from the repeatedly observed empirical effectiveness of TS-based algorithms [Chapelle and Li, 2011, Russo et al., 2017].
Moreover, the computational efﬁciency of RM-GPTS is markedly superior to other methods (Fig. 2b).
These theoretical and empirical comparisons between RM-GPUCB and RM-GP-TS may provide useful insights for other meta-BO algorithms and potentially for a broader range of problems (e.g., meta-learning for multi-armed bandits and
RL) in terms of the relative strengths and weaknesses of
UCB- and TS-based algorithms.
RELATED WORKS
Some previous works on meta-BO build a joint GP surrogate using all previous and current observations, and represent task similarity through meta-features [Bardenet et al., 2013,
Schilling et al., 2016, Yogatama and Mann, 2014].
However, these algorithms suffer from the requirement of handcrafted meta-features, which is avoided in other works that learn task similarity from the observations [Swersky et al., 2013,
Shilton et al., 2017].
For example, multitask BO [Swersky et al., 2013] uses a multitask GP as a surrogate and models each task as an output of the GP.
These works include all previous and current observations in a single GP surrogate and are thus limited by the scalability of GPs.
There have also been other empirical works which replace GP by Bayesian linear regression for scalability [Perrone et al., 2018], tackle sequentially arriving tasks [Golovin et al., 2017, Poloczek et al., 2016], learn a set of good initializations [Feurer et al., 2015, Wistuba et al., 2015b], learn a reduced search space for BO from previous tasks [Perrone et al., 2019], handle the issue of different function scales using Gaussian Copulas [Salinas et al., 2020], learn the task similarities through the distance between the distributions of the optima from different tasks [Ramachandran et al., 2018], or use the metaobservations to learn the entire acquisition function through
RL [Volpp et al., 2020].
Wang et al.
[2018] have learned the
GP
prior from previous tasks and given theoretical guarantees.
However, they have shown in both theory and practice that a large training set of meta-observations ( ≥ 5000 ) is required for their method to work well, while we focus on the more practical setting of meta-BO where the number of available meta-observations may be small.
We have also veriﬁed that our algorithm outperforms the method from Wang et al.
[2018] in the experiment that is most favorable for their method among all our experiments (more details in the third paragraph of Sec. 6.2).
Meta-BO is also related to the works on multi-ﬁdelity BO
[Dai et al., 2019, Kandasamy et al., 2016, Poloczek et al., 2017, Wu et al., 2020, Zhang et al., 2020, 2017], since the previous tasks can be viewed as low-ﬁdelity functions which can approximate the target function and are cheap to query.
However, multi-ﬁdelity BO allows querying the low-ﬁdelity functions during the BO process, whereas meta-BO algorithms can only query the target function, i.e., the highest-ﬁdelity function.
Moreover, meta-BO is also related to the previous works on BO which involve multiple agents (i.e., analogous to multiple tasks in meta-BO), such as federated BO [Dai et al., 2020b, 2021,
Sim et al., 2021] or BO methods based on game-theoretical approaches [Dai et al., 2020a, Sessa et al., 2019].
Some works have aimed to improve the scalability of GPbased meta-BO algorithms by building a separate GP surrogate for each task
[Feurer et al., 2018, Wistuba et al., 2016, 2018].
Wistuba et al.
[2016] use a weighted combination of the posterior mean of each individual GP surrogate as the joint posterior mean while the posterior variance is derived using only the target observations.
RGPE [Feurer et al., 2018] has extended the work of Wistuba et al.
[2016] by estimating the joint objective function as a weighted combination of individual objective functions, such that the resulting joint surrogate remains a GP (unlike Wistuba et al.
and can thus be plugged into standard BO algorithms.
Note that RGPE differs from our RM-GP-UCB algorithm in that RGPE uses a weighted combination of individual GP surrogates to derive a joint GP surrogate, whereas our RM-GP-UCB leverage a weighted combination of individual acquisition functions.
Wistuba et al.
[2018] have proposed TAF, which also uses a weighted combination of the acquisition functions (i.e., expected improvement) from the individual tasks for query selection.
In these works, the weight of a previous task is heuristically chosen to be proportional to the accuracy of the pairwise ranking of the target observations produced by either (a) the posterior mean of the GP surrogate of the previous task (TAF)
[Wistuba et al., 2018] or (b) functions sampled from the posterior GP surrogate (RGPE)
[Feurer et al., 2018].
CONCLUSION
We have introduced RM-GP-UCB and RM-GP-TS, both of which are asymptotically no-regret even if all meta-tasks are dissimilar to the target task.
We leverage the theoretical results to learn the task similarities in a principled way via online learning.
Theoretical and empirical comparisons show that RM-GP-UCB is more robust against dissimilar tasks, whereas RM-GP-TS performs effectively in more favorable cases and is more computationally efﬁcient.
Acknowledgements
This research/project is supported by A*STAR under its
RIE 2020 Advanced Manufacturing and Engineering (AME)
Industry Alignment Fund – Pre Positioning (IAF-PP) (Award A 19 E 4 a 0101 ) and by the Singapore Ministry of Education Academic Research Fund Tier 1 .