A UTOMATIC PRIOR SELECTION FOR META B AYESIAN OPTIMIZATION WITH A CASE STUDY ON TUNING DEEP NEURAL NETWORK OPTIMIZERS
Anonymous authors
Paper under double-blind review
A BSTRACT
The performance of deep neural networks can be highly sensitive to the choice of a variety of meta-parameters, such as optimizer parameters and model hyperparameters.
Tuning these well, however, often requires extensive and costly experimentation.
Bayesian optimization (BO) is a principled approach to solve such expensive hyperparameter tuning problems efﬁciently.
Key to the performance of BO is specifying and reﬁning a distribution over functions, which is used to reason about the optima of the underlying function being optimized.
In this work, we consider the scenario where we have data from similar functions that allows us to specify a tighter distribution a priori.
Speciﬁcally, we focus on the common but potentially costly task of tuning optimizer parameters for training neural networks.
Building on the meta BO method from Wang et al.
(2018b), we develop practical improvements that (a) boost its performance by leveraging tuning results on multiple tasks without requiring observations for the same meta-parameter points across all tasks, and (b) retain its regret bound for a special case of our method.
As a result, we provide a coherent BO solution for iterative optimization of continuous optimizer parameters.
To verify our approach in realistic model training setups, we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of conﬁgurations of near-state-of-the-art models on popular image and text datasets, as well as a protein sequence dataset.
Our results show that on average, our method is able to locate good hyperparameters at least 3 times more efﬁciently than the best competing methods.
I NTRODUCTION
The careful tuning of a variety of meta-parameters, such as optimizer parameters and model hyperparameters has become a basic necessity for deep learning (Bergstra et al., 2011; Feurer et al., 2015).
Such tuning requires extensive experimentation, retraining models repeatedly with different conﬁgurations, and can be challenging at realistic budgets because the tuning landscape is typically non-stationary, noisy and ill-behaved.
Tuning has become sufﬁciently costly that ﬁnding more efﬁcient and effective tuning procedures has the potential to save a substantial amount of resources or, alternatively, improve the accuracy of the ﬁnal models at a given budget.
Some hyperparameters might be common across a large number of tuning problems, such as those pertaining to an optimization algorithm.
For example, Adam is used across many deep learning applications and has four parameters that require careful tuning (Nado et al., 2021).
Thus, if we have access to the performance of different optimizer-speciﬁc hyperparameters on different model training tasks, we may be able to transfer the knowledge among those tasks.
This kind of meta-level learning is common among practitioners themselves: when faced with a new tuning problem, one might ﬁrst try reusing hyperparameter settings that worked well on another problem.
The underlying assumption is that hyperparameters should perform similarly across tasks.
In this work, we aim to formalize this assumption and automate optimizer hyperparameter tuning by leveraging knowledge from previous experiments.
Although our experiments consider optimizer parameter tuning as a practically important sub-problem of hyperparameter tuning for deep neural networks, our method applies to any hyperparameters that are common across multiple tasks.
Bayesian optimization (BO) has become a popular methodology for optimizing the hyperparameters of machine learning models (Snoek et al., 2012; Bergstra et al., 2011) and represents the state-ofthe-art (Turner et al., 2021).
BO involves specifying a probabilistic model over the function to be optimized and using this to reason about the location of the optimum.
The optimization proceeds by iteratively updating the model with new data and using the posterior distribution to reason about where to next evaluate, trading off exploration and exploitation.
The model is typically speciﬁed with only a-priori assumptions of smoothness, for example using a Gaussian process (GP) with a smooth covariance function.
Even if the model is well-speciﬁed, BO can be slow to converge due to the generality of the prior assumptions.
This would seem wasteful for problems that are repeated often or share considerable structure with previous experiments.
One natural option is to cast our problem as meta Bayesian optimization , where the goal is to learn to optimize a function by generalizing from past experience with other similar functions.
Indeed, several BO methods exist in the literature, but they are unsuitable for our scenario where we envision potentially thousands of related tasks within e.g. the context of a hyperparameter tuning service.
Existing meta BO methods either scale cubically in the number of evaluations and tasks (Swersky et al., 2013; Bardenet et al., 2013) (See §4.3 for more details), impose a restrictive set of assumptions on the available data (Wang et al., 2018b; Swersky et al., 2013) to obtain efﬁcient solutions, or make assumptions on the availability of GP parameters (Volpp et al., 2020) or descriptive task-level features (Brazdil et al., 1994; Bardenet et al., 2013; Yogatama & Mann, 2014).
To address these issues, we introduce HyperBO: a meta BO method that builds upon Wang et al.
(2018b)
with a relatively simple assumption: all the related functions being optimized are samples from the same GP prior distribution over functions.
Concretely, HyperBO assumes the functions are conditionally independent given the hyperparameters, mean and covariance function of the GP.
Compared to Wang et al.
(2018b), HyperBO does not impose any strict conditions on data or model structures, and a special case of HyperBO retains strong regret bounds.
From a computational perspective, HyperBO scales linearly in the number of tasks during training, and does not depend on the number of tasks when deployed.
By not imposing assumptions about the data collection conditions, it can be used with large ofﬂine datasets or a few related optimization trajectories.
To evaluate HyperBO, we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of conﬁgurations of near-state-of-the-art models on popular image and text datasets, as well as on a protein sequence dataset.
We compare HyperBO to several hyperparameter tuning baselines in the sequential BO setting.
Our results show that optimizers that use hyperparameters suggested by our method are able to obtain better performing models requiring at least 3 times fewer function evaluations than other baselines.
Our main contributions are two-fold: (1) a practical meta BO approach that makes minimal assumptions; and (2) a large multi-task hyperparameter tuning dataset that not only beneﬁts our method but also serves as an ideal benchmark to test future multi-task or meta-learning BO methods.
Related work There is a rich literature of innovative methodologies to improve the efﬁciency of
BO given related tasks or additional context.
Here we discuss the most closely related work and explain why these don’t solve the speciﬁc scenario which we envision.
Speciﬁcally, our goal is a methodology that is scalable enough to share information across thousands of tasks, each with potentially hundreds of observations, such as in the context of a large BO service or library.
In this work we use the term meta-BO to refer to the class of BO methods that use data from existing tasks to optimize a new task.
Multi-task BO
(Swersky et al., 2013; Poloczek et al., 2017; Yogatama & Mann, 2014) and transfer learning BO
using contextual GPs (Krause & Ong,
; Bardenet et al., 2013; Poloczek et al., 2016) are both meta BO approaches.
Some meta BO methods have also been studied for hyperparameter tuning tasks in machine learning (Feurer et al., 2015).
HyperBO assumes all tasks are independent (after conditioning on the GP), whereas both multi-task and contextual BO rely heavily on the assumption that tasks are related.
Thus the latter approaches typically scale cubically in both the number of tasks and observations in each task, meaning that they cannot gracefully scale across both without heavy approximations.
When assuming that all inputs are equal across tasks, multi-task BO can be sped up using a Kronecker decomposition of the kernel to a task kernel and an input kernel which can be inverted separately; a similar assumption is made by Wang et al.
(2018b).
In comparison, HyperBO scales linearly in the number of tasks (see §B).
End-to-end learning (Chen et al., 2017; Volpp et al., 2020) is another popular meta BO approach for hyperparameter tuning that learns a strategy to suggest new query points based on past history of
BO.
One limitation of such approaches is that the total number of BO iterations must be determined a-priori.
Furthermore, by nature of using a highly parameterized model to train the strategy, we lose the interpretability of intermediate steps that GPs and acquisition functions provide.
HyperBO builds upon Wang et al.
(2018b)
and Kim et al.
We resolve their issues with optimizing over a continuous space rather than a discrete set and limitations on using the same set of inputs across tasks.
Kim et al.
(2017; 2019) estimated a multivariate Gaussian that models values for search strategies in robot manipulation tasks, and thus only considered discrete inputs.
Wang et al.
(2018b) provided regret bounds for Kim et al.
(2017; 2019), which was identiﬁed as meta BO without the knowledge of the GP prior.
For both ﬁnite discrete search spaces and continuous ones, Wang et al.
(2018b) requires observations on the same set of inputs across tasks, which is an assumption that is not required for HyperBO; HyperBO still inherits the same regret bound as Wang et al.
(2018b)
for the special case where the same-inputs assumption is satisﬁed.
Similar ideas also appeared in
Perrone et al.
(2018), which can be viewed as a special case of HyperBO or Wang et al.
(2018b).
P ROBLEM FORMULATION
We consider the standard black-box function optimization scenario: given a real-valued function f deﬁned over a compact, hyper-rectangular space X ⊂ R d and given observations of similar functions f 1 , · · · , f N , we seek an x ∈ X optimizing f .
We inherit our problem formulation from Wang et al.
(2018b),
but we relax impractical assumptions on data availability (we do not require all observations to be made on the same inputs across tasks) and model restrictions.
Assumptions and the goal.
Concretely, we assume that there exists a Gaussian process GP ( µ, k ) with unknown mean function µ : X → R and kernel k : X × X → R .
Let N be the number of tasks and let M i be the number of observations we have for the i th task.
Conditioned on independent function samples f i ∼ GP ( µ, k ) and inputs x
( i ) j ∈ X , i ∈
[ N ] , j ∈ [ M i ] , we observe evaluations
y ( i ) j ∼ N ( f i ( x ( i ) j ) , σ 2 ) perturbed by i.i.d.
additive Gaussian noise N (0 , σ 2 ) .
Taken together, the collection of sub-datasets D f i = { ( x ( i ) j , y ( i ) j ) } M i j =1 deﬁne a dataset D N = { D f i } Ni =1 .
Finally, our goal is to maximize a new function independently sampled from the same GP, f ∼ GP ( µ, k ) ; that is, solve arg max x ∈ X f ( x ) given dataset D N but unknown functions µ, k and unknown parameter σ 2 .
An example.
In our optimizer hyperparameter tuning application, a task corresponds to ﬁnding the best optimizer hyperparameters to train a given model on a particular dataset, 2 e.g. training a
ResNet (He et al., 2016) on ImageNet (Russakovsky et al., 2015).
Notice that we do not assume that the mean function µ , kernel k and noise variance σ 2 are given.
This is consistent with the reality of solving real-world black-box optimization problems including hyperparameter tuning.
We must learn those unknown functions and parameters from data.
However, in practice, searching in functional spaces to ﬁnd the right mean µ or kernel k is a daunting task.
Hence for practical concerns, a well deﬁned search space for functions is required.
More details on this can be found at §3.1.
Metrics.
For simplicity, throughout this paper, we focus on the setting where the target function f can only be optimized by iteratively choosing where to evaluate, and defer batch evaluation setups to
Sec.
F.
As we run BO on the target function f for T iterations, we accumulate a set of observations
D f = { ( x t , y t ) }
Tt =1 , y t ∼ N ( f ( x t ) , σ 2 ) .
We evaluate the quality of the optimization using the simple regret metric: R T = max x ∈ X f ( x ) − f
(ˆ x ) , where ˆ x is the ﬁnal recommendation at the end of the optimization process.
There are various ways of setting ˆ x based on the observations D f ; we use the input that achieved the best evaluation: ˆ x = x τ ; τ = arg max t ∈ [ T ] y t .
Bayesian viewpoint.
As mentioned above, the observed functions
f 1 , · · · , f N and the evaluation target f are assumed to be independent draws from the same GP.
This assumption is consistent with a hierarchical Bayes interpretation (Fig. 1), where all observed functions are independent conditioned on the GP.
Notice that for BO, each selected input x ( i ) j depends on all previous observations.
But we only describe the generative model of a hierarchical GP for simplicity.
θ µ f i k α σ y ( i )
j x ( i )
j
M i N
More speciﬁcally, we assume that underlying functions of hyperparameter optimization tasks are deﬁned by a parameter θ ∼ p ( θ ; α ) ; mean and kernel functions µ and k are deﬁned by deterministic functions parameterized by θ .
The independent function samples { f i } i ∈
[ N ] are draws from GP ( µ, k ) .
We then learn function f from observations D N on all other conditionally i.i.d.
function samples f 1 , · · · , f N .
We forgo a fully
Bayesian approach that samples from the posterior over θ at every BO iteration.
Our method can be viewed as a type-II maximum likelihood approximation of such a Bayesian solution.
Notations.
Let [ n ] denote { 1 , · · · , n } , ∀ n ∈ Z + .
For conciseness, we write the evaluation of a function f : X → R on matrix x =
[ x i ] ni =1 ∈ R n × d as µ ( x ) :=
[ µ ( x i )]
ni =1 .
Similarly, for two matrices (i.e., the concatenation of input vectors) x , x (cid:48) , we write the corresponding kernel matrix as k ( x , x (cid:48) ) :
=
[ k ( x i , x (cid:48) j )] i ∈ [ n ] ,j ∈ [ n (cid:48) ] , and shorten k ( x ) :
= k ( x , x ) .
We denote a (multivariate) Gaussian distribution with mean u and variance Σ by N ( u, Σ) , and a Gaussian process (GP) with mean function µ and covariance function k by GP ( µ, k ) .
Let σ 2 be the noise variance in observations.
Given a set of observations D = { ( x t , y t ) }
Tt =1 , y T =
[ y t ] Tt =1 ∼ N ( f ( x T ) , σ 2 I ) , x T =
[ x t ]
Tt =1 and f ∼ GP ( µ, k ) , we denote the corresponding conditional GP distribution as GP ( µ, k | D ) .
Let ψ ( x ) =
k ( x, x T )( k ( x T ) + σ 2 I ) − 1 .
Recall that the conditional distribution, GP ( µ, k | D ) =
GP ( µ D , k D ) , is given for any x, x (cid:48)
∈ X as µ D ( x ) = µ ( x ) + ψ ( x )( y T − µ ( x T )) , k D ( x, x (cid:48) )
=
k ( x, x (cid:48) )
− ψ ( x ) k
( x T , x (cid:48) ) .
O UR METHOD
1: function H YPER BO ( f, D N ) 2:
GP (ˆ µ, ˆ k ) ← T RAIN GP ( D N ) 3:
D f ← ∅ 4: for t = 1 , · · · , T do 5: x t ← arg max x ∈ X α (cid:16) x ; GP (ˆ µ, ˆ k | D f ) (cid:17) 6: y t ← O BSERVE ( f ( x t )) 7:
D f ← D f ∪ { ( x t , y t ) } 8: end for 9: return D f 10: end function
As shown in Alg. 1, our approach trains the
GP hyperparameters on a representative set of datasets and ﬁxes them for the duration of the optimization procedure; we refer to this approach as HyperBO.
HyperBO runs in two steps.
First, we learn a GP model
GP (ˆ µ, ˆ k ) to approximate the ground-truth (unknown) GP that generated the dataset
D N .
Then, we do standard BO to optimize a new function f with the learned
GP GP (ˆ µ, ˆ k ) .
The initial learning process (line 2) is the critical difference between
HyperBO and standard BO algorithms, as well as the key contribution of this paper.
Based on the Bayesian graphical model interpretation (Fig. 1), our goal is to obtain a point estimate ˆ θ for the parameter θ .
Given this estimate, we can then estimate the mean function ˆ µ and the kernel ˆ k , which deﬁnes our learned model GP (ˆ µ, ˆ k ) .
During the BO iterations (Alg. 1, lines 4-8), we update the conditional GP, but do not re-estimate the GP mean and kernel.
By separating the data for the conditional GP update and GP parameter training, we minimize the computational cost while still maintaining good performance both theoretically and empirically.
Moreover, we avoid the BO chicken-and-egg dilemma (Wang et al., 2018b) where the search strategy is trained on data collected in the BO process and the data points are selected by the search strategy.
Next, we introduce our GP training strategy based on two types of objectives: marginal data likelihood (§ 3.1) and distance between estimates and model predictions (§ 3.2).
In the appendix, §B reveals the complexity of HyperBO that is linear in the number of tasks and § 3.3 shows that a special case of
HyperBO retains strong regret bounds (Wang et al., 2018b).
3. M ARGINAL LIKELIHOOD
A straightforward way to train a GP is by optimizing the log marginal likelihood over the GP’s hyperparameters.
This is also known as type II maximum likelihood approximation (Rasmussen
Williams, 2006).
In our case, we derive the data likelihood for the observations from multiple functions that are assumed to be given, which is a key difference to regular GP or BO setups.
The log marginal likelihood for our method is
[Equation]
where ¯ y
( i
[Equation]
=
[Equation]
i
[Equation]
, K = k (cid:0) x ( i ) (cid:1) +
[Equation]
I , x ( i ) =
[ x ( i )
j ] M i j
=1 and y ( i ) =
[ y ( i
[Equation]
j ] M i j =1 .
Our solution to the choice of mean function, kernel function and noise variance then becomes
[Equation]
For mean function µ and kernel k , this optimization is done in functional space.
While methods exist to search for functional structures (Kemp & Tenenbaum, 2008; Malkomes & Garnett, 2018), one may opt for a simple search strategy within a group of functional structures (e.g. mean µ ∈ { linear , constant } and kernel k ∈ { exponentiated quadratic , Matérn } ).
For all combinations of mean/kernel structures or functional classes, we then optimize the parameterization of them and noise variance σ 2 to eventually solve Eq. 3.
Details of how we deﬁned the search space can be found in §4.
3. D ISTANCE BETWEEN ESTIMATES AND MODEL PREDICTIONS
Although the marginal likelihood is a straightforward objective to optimize, it may not be straightforward to interpret how high of a likelihood is high enough for us to stop our search for a decent model.
Nevertheless, we may be able to directly estimate the sample mean and covariance, and the distance between those estimates and model predictions could be a good indicator of how good the model is.
We will show in §3.3 that a distance objective may lead us to better theoretical properties.
Here we consider a special case of dataset D N where part of it has matching inputs across some sampled functions.
More formally, suppose we have a matching dataset D (cid:48)
N = { ( x j , y j ) } Mj
=1 where
M is a positive integer, x j ∈ X , y j
= [ y ( i ) j ] Ni =1 ∈ R N and y ( i ) j ∼ N ( f ( x j ) , σ 2 ) .
Empirically, dataset
D (cid:48)
N can be constructed by querying a set of functions f 1 , · · · , f N at the same set of input locations x
= [ x j ] Mj =1 ∈ R M × d to obtain an observation matrix y =
[ y j ] Mj =1 ∈ R M × N .
By deﬁnition of a GP, the vector of all function queries f ( x ) is distributed according to a multivariate
Gaussian distribution N ( µ ( x ) , k ( x )) .
With our observation model, we get the distribution for observations y ∼ N ( µ ( x ) , k ( x ) + I σ 2 ) for some unknown mean function µ and kernel k .
However, given that we have access to all observations y , we can estimate the mean on inputs x as ˜ µ = 1 N y 1 N ∈ R M and estimated covariance as ˜ K = 1 N ( y − ˜ µ 1 (cid:62) N )( y − ˜ µ 1 (cid:62) N ) (cid:62) ∈ R M × M ; here 1 N is a column vector of size N ﬁlled with 1 s. We use a biased estimate of covariance to be consistent with the corresponding maximum likelihood estimator in Eq.
3 Notice that the estimated covariance includes in diagonal terms the variance of the observation noise.
For any distance function between the estimate N (˜ µ , ˜ K ) and model prediction N ( µ ( x ) , k ( x )+ I σ 2 ) , we obtain an objective to minimize, D (cid:16) N (˜ µ , ˜ K ) , N ( µ ( x ) , k
( x ) + I σ 2 ) (cid:17) .
While there are different measures of distributional discrepancy, we adopt the KL divergence.
Let µ = µ ( x ) and K
= k ( x ) + I σ 2 .
The KL divergence is deﬁned as
[Equation]
and we can estimate the mean, kernel and noise variance by minimizing D KL .
While it is difﬁcult to gauge how much a probability density is enough to obtain a good model, Eq. 4 is a “distance” that goes to 0 as the difference between two distributions reduces.
One may choose to do early stopping or model selection based on how close Eq. 4 is to 0 .
Through information theory, we also know that the KL divergence in Eq.
4 describes the number of extra bits (or nats) to encode the multivariate normal N (˜ µ , ˜ K ) .
Overall we found the KL divergence in Eq.
4 relatively more interpretable than the marginal likelihood in Eq.
The KL divergence in Eq.
4 introduces a different optimization landscape than the marginal likelihood in Eq.
The KL divergence also makes use of the matching dataset D (cid:48)
N in a way that the marginal likelihood cannot.
In fact, all matching inputs in the marginal likelihood in Eq.
3 are implicit: all inputs are passed in to mean/kernel functions, and so there is no way that
Eq.
3 can be informed that some inputs are the same across tasks.
As shown in §4, the KL divergence in Eq.
4 interestingly led to better results in our experiments.
3. T HEORETICAL ANALYSES
While it is nontrivial to prove regret bounds for general scenarios without strict assumptions, it is straightforward to show a regret bound for our method with objective D KL of Eq.
4 in the matching dataset case where BO is running on a ﬁnite set of inputs (Wang et al., 2018b).
Theorem 1.
Let N ≥ 4 log 6 δ + T + 2 .
With probability at least 1 − δ , simple regret in T iterations of Alg. 1 with special cases of either GP-UCB or PI satisﬁes
[Equation]
More details can be found at §D. Theorem 1 shows that the regret bound has a linear dependency on the observation noise σ .
This is expected because in practice, we select the best observation rather than best function value (before observing a noisy version of it) to compute the simple regret.
Another reason is that we learn the noise parameter σ jointly with the kernel, as shown by Eq. 4.
Hence when computing acquisition functions, the noise σ is always included in the predicted variance.
Intuitively, the more sub-datasets we have in the dataset, the larger N is, the better we are able to estimate the GP model, and the closer the regret bound is to the case where the GP model is assumed known.
Interestingly, the number of BO iterations T makes the regret smaller in the second term but larger in the ﬁrst term in Eq.
Usually as we get more observations, we get more information about the maximizer, and we are able to optimize the function better.
However, as we get more observations on the new function, GP conditional predictions have more freedom to deviate from the ground truth (see Lemma 1 of Wang et al.
(2018b)).
As a result, we get less and less conﬁdent about our predictions, which is eventually reﬂected in a looser regret upper bound.
It is tempting to prove similar bounds for more general settings where inputs are not the same across all sub-datasets and BO happens in continuous space.
Though the only prerequisite is to show that the difference between the learned mean/kernel and the ground truth mean/kernel is small, this prerequisite is as difﬁcult as showing we can ﬁnd a model that has bounded generalization error across the entire continuous input space of an arbitrary function.
Instead of making unrealistic assumptions just to satisfy such prerequisite, we leave the regret bound for general settings as an open question.
E XPERIMENTS
Our goal in this paper is to provide a practical approach for hyperparameter optimization when we are given data on a range of tasks over the same search space.
To analyze the effectiveness of our proposal, we take the optimizer hyperparameter tuning problem in deep learning as a case study.
Our implementation of HyperBO is based on JAX (Bradbury et al., 2018).
To reduce ambiguity, we distinguish between datasets that individual neural networks are trained on and the dataset we collected that includes optimizer hyperparameter points with their validation errors (and other metrics).
We will call the former (e.g. MNIST, CIFAR10) task datasets and call the latter the tuning dataset.
The tuning dataset is what we described as dataset D N in §2.
4. H YPERPARAMETER TUNING DATASET
In order to collect our hyperparameter tuning dataset, the PD1 Neural Net Tuning Dataset, we deﬁned a set of 24 neural network tuning tasks 5 and a single, broad search space for Nesterov momentum.
Each task is deﬁned by a task dataset (e.g. ImageNet), a speciﬁc neural network model (e.g. ResNet50), and a batch size.
Tab. 1 shows all the tasks that we consider in the tuning dataset.
We used an existing code base (Gilmer et al., 2021) for neural network model training.
The dataset used roughly 12,000 machine-days of computation for approximately 50,000 hyperparameter evaluations.
For each task, we trained the model on the task dataset repeatedly using Nesterov momentum (Nesterov, 1983; Sutskever et al., 2013), with the task’s minibatch size, with different hyperparameter settings drawn from the 4-dimensional search space detailed in Tab. 2.
We tuned the base learning rate, η , on a log scale, the momentum, β , with 1 − β on a log scale, and the polynomial learning rate decay schedule power p and decay steps fraction λ .
We used a polynomial decay schedule with the following form: η
[Equation]
p , where τ is the training step and T is the total number of training steps for the task.
[Table]
[Figure]
[Table]
[Table]
We collected two types of data: matched and unmatched data.
Matched data used the same set of uniformly-sampled hyperparameter points across all tasks and unmatched data sampled new points for each task.
All other training pipeline hyperparameters were ﬁxed to hand-selected, task-speciﬁc default values.
All of our tasks are classiﬁcation problems, so they all used the same training loss, although occasionally task-speciﬁc regularization terms were added.
For each trial (training run for a single hyperparameter point), we recorded validation error (both cross entropy error and misclassiﬁcation rate).
In many cases, poor optimizer hyperparameter choices can cause training to diverge.
We detected divergent training when the training cost became NaN and then marked the trial but did not discard it.
Please see the Appendix, supplementary material, and code (Onomous, 2021) for additional details about the tasks and training procedure.
The different tuning tasks vary in difﬁculty and numbers of data points, but generally there are roughly 500 matched datapoints and 1500 unmatched datapoints per tuning task.
For unmatched data only, we attempted to generate roughly similar numbers of non-divergent points across tasks, so tasks with a higher probability of sampling a hyperparameter point that causes training to diverge will tend to have more trials.
4. D ESCRIPTION OF ALL COMPARED METHODS
Our method HyperBO has several variants including using different acquisition functions and different objectives.
In §4, unless otherwise mentioned, we used a thresholded probability of improvement (PI) as the acquisition function.
We set PI in line 5 of Alg.
1 as ˆ µ
[Equation]
[Equation]
We empirically evaluated a variety of acquisition functions, but found PI thresholded at 0.1 to be surprisingly effective.
Because we model the observations as log error rate, this actually trades off exploration and exploitation - i.e. with larger error rates this seeks relatively more substantial improvements than with small error rates.
The list of 5 different acquisition functions we tested is as follows: PI with 0.
threshold, expected improvement and UCB with 2, 3, 4 coefﬁcients.
See their comparisons at §E.5.
We use H* NLL to denote HyperBO with negative log marginal likelihood as the objective and
H* KL to denote HyperBO with KL divergence on matching datapoints as objective.
Both objectives are optimized via L-BFGS (Nocedal, 1980) with µ ( x )
= θ (cid:62)
tanh( θ (cid:62) 1 x ) , k
( x, x (cid:48) ) =
Matérn32 (tanh( θ (cid:62) 1 x ) , tanh( θ (cid:62) 1 x (cid:48) )) , θ 1 ∈ R 4 × 8 .
These two settings of HyperBO are relatively representative of the performance of variants of HyperBO.
See comparisons over objectives at §E.4.
Our baselines include (a) Rand: Random search in the corresponding scaled space in Tab. 2.
(b) STBO:
Single-task BO where in every BO iteration, STBO optimizes the GP hyperparameters via marginal likelihood on data of the test task.
This implementation corresponds to the basic off-the-shelf BO setups.
(c) STBOH:
Single-task GP-UCB with a hand-tuned prior on hyper-parameters including
UCB coefﬁcient.
(d) MIMO: Multi-task BO with GP bases as an ensemble of feedforward neural networks with shared subnetworks (Havasi et al., 2020).
(e) RFGP: Multi-task BO with GP bases as random features.
More details can be found at §E.1.
4. R ESULTS ON OFFLINE OPTIMIZER HYPERPARAMETER TUNING TASKS
Many tasks in §4.1 can use up a lot of compute resources and time, which makes it infeasible to perform a wide variety of experiments to analyze the characteristics of BO methods.
Hence we adopt an ofﬂine approximation, which runs BO only on the ﬁnite set of points that each tuning sub-dataset contains.
More details and analyses are available in Appendix E.2.
Holding out relevant tasks.
Fig.
2 (a) shows the performance proﬁles , the fraction of all test tasks that each method from §4.2 is able to outperform a baseline criterion at each BO iteration.
We can see that MIMO is able to outperform other methods in the beginning 20 BO iterations, but its leading position soon gets surpassed by HyperBO (H* NLL and H* KL).
Fig.
2 (b,c) illustrates the BO convergence curves of all competing methods, together with the vertical slice at the 100th iterations.
RFGP and STBO are both falling much behind Rand.
STBO trains the GP on the data that the GP suggests to query.
Optimizing the marginal data likelihood on at most 100 datapoints in fact may not lead to a better model than random initialization (see Tab. 5 in §F).
Surprisingly, the contextual information learned by RFGP did not generalize to a new task.
On the other hand, MIMO is able to obtain a slightly better error rate than STBOH.
Overall, learning the GP prior through data as with
HyperBO outperforms other meta BO methods, and is a more principled and effective way to obtain the GP prior when compared with hand-tuning.
Effect of number of training tasks.
We now investigate the impact of the number of training tasks on the performance of meta BO methods.
In Fig 3 we show the BO simple regrets on tasks from Table 1 (except ImageNet ResNet50 2048) that use meta BO models trained on different number of training tasks.
To analyze the performance of all methods on less-related tasks, we ﬁrst remove training tasks that have the same task dataset as our current tuning task for testing, and then remove randomly selected training datasets from the rest.
HyperBO variants reduced the simple regret as more training tasks are given.
Interestingly, H* NLL and H* KL are already slightly better than Rand and STBOH when they started off with only 3 training tasks.
There are reasonable ﬂuctuations in the results but overall the trend of regret is going down as the number of training tasks increases.
MIMO also reduced regret when the number of tasks increased from 8 to 18.
RFGP, however, fails to learn from training tasks possibly because it did not learn good task embeddings for GP regression models.
Effect of number of data points in training tasks.
One remaining question is, how does M i in §2, the number of data points in each training tasks, affect the performance of meta
BO methods.
We analyze the impact of M i by removing a portion of all data that we have access to for each task.
In particular, we set the percentage of remaining data to be 0 .
Remaining datapoints are selected uniformly randomly, which breaks the structure of matching data.
Hence we do not include H* KL in this comparison, as H* KL only makes use of matching data.
Fig.
4 shows how the simple regret changes as the fraction of training data grows.
Below 10% training data, we observe clear trend that more data lead to lower regret for both H* NLL and
MIMO, and relatively no change for RFGP.
We also found that the performance of HyperBO (H* NLL) does not change much as the fraction of training data increases from 5% to 90% .
However,
MIMO and RFGP suffers signiﬁcantly from more data as the fraction of training data increases from 5% to 50% .
It is not entirely clear why MIMO and RFGP have such behaviors.
One conjecture is that neural network based Bayesian linear regression models may get too conﬁdent once the amount of data reaches a certain threshold.
This means much less exploration if those models are used for BO.
4. R ESULTS ON ONLINE OPTIMIZER HYPERPARAMETER TUNING TASKS
We now evaluate HyperBO methods, H* NLL and H* NLLKL (NLL plus KL divergence on matching datapoints as the objective), in the online setting, where we optimize over the full hypercube and some hyperparameters may be infeasible to evaluate.
See full results at §E.3.
Fig.
5 shows that it can be difﬁcult for STBO or MIMO to recover from a “bad” datapoint, but HyperBO methods are robust and performed the best among all methods being compared.
D ISCUSSION AND CONCLUSION
While we focused on obtaining a better prior in BO in this work, the following directions are orthogonal to what we studied: different search spaces across tasks, batch evaluation, high-dimensional or large scale data, etc.
However, it should be straightforward to combine their solutions with HyperBO.
Please ﬁnd more discussions at §F together with implications of our assumptions.
HyperBO is a novel meta BO approach that supports practical applications that involve continuous inputs queried at possibly non-aligned locations across tasks.
HyperBO uses a simple yet effective idea that is easy to implement and efﬁcient to run.
We evaluated HyperBO on real-world big model optimizer tuning tasks, and the results demonstrated its superior performance over state-of-the-art competing methods.
A H IERARCHICAL GP, A B AYESIAN PERSPECTIVE
In Bayesian viewpoint of §2, we assume that the overall setting of the hyperparameter optimization task is deﬁned by a parameter θ ∼ p ( θ ; α ) ; mean and kernel functions µ and k are drawn from p ( µ, k | θ ) .
The independent function samples { f i } i ∈
[ N ] are themselves draws from GP ( µ, k ) .
The generative story is as follows:
• Draw GP parameter θ from p ( θ ; α ) and observation noise parameter σ from p ( σ ; α ) .
• Draw mean function µ and kernel function k from p ( µ, k | θ ) .
• For each task i from 1 to N , – Draw a function f i from GP ( µ, k ) .
– For each data point j from 1 to M i , ∗ Given input x ( i )
j , we draw the observation y
( i ) j ∼ N ( f i ( x ( i ) j ) , σ 2 ) .
We simplify this hierarchical setting by deﬁning p ( µ, k | θ ) to be a sum of Dirac delta functions: both mean function µ and kernel k are deterministic functions parameterized by θ .
Thus, we can infer GP parameter θ and noise σ from their posterior p ( θ, σ | D N ∪ D f ; α ) and obtain an informed prediction for the target function
[Equation]
In other words, we learn function f from observations on all other conditionally i.i.d.
function samples f 1 , · · · , f N .
We forgo a fully Bayesian approach that samples from the posterior over θ at every BO iteration, although our method, HyperBO, can be viewed as a type-II maximum likelihood approximation of such a Bayesian solution.
B C OMPUTATIONAL COMPLEXITY
The marginal likelihood in Eq.
2 naturally decomposes into a sum of GP data likelihood terms on each sub-dataset D f i .
The time complexity to compute Eq.
2 is O
( M 3 N ) , where N is the number of sub-datasets and M = max Ni =1 M i is the maximum number of data points for these sub-datasets.
Notice that our method scales linearly in the number of tasks, N , in contrast to the cubic O ( M 3 N 3 ) scaling of multi-task or contextual BO methods (Swersky et al., 2013; Bardenet et al., 2013; Poloczek et al., 2016; Yogatama & Mann, 2014).
The only cubic cost of HyperBO is on the number of data points in sub-datasets.
To train a GP with K optimization steps on Eq. 2, the time complexity is O ( M 3 NK ) .
The distance regularizers introduced in §3.2 requires estimating mean and covariance, which takes O ( M 2 N ) for matrix multiplication.
The KL divergence in Eq.
4 has complexity O
( M 3 ) to compute and O ( M 3 K ) to optimize.
If there is any better probabilistic model than a GP to ﬁt the data with less compute time, we can easily swap it in and reduce the O ( M 3 ) complexity that the GP contributed to the O
( M 3 N ) complexity of Eq.
For example, if we approximate a GP with a linear model on V random features (Rahimi et al., 2007), the complexity of Eq.
2 becomes O
( V 3 N ) .
Another example is to train Eq.
2 with stochastic optimization methods, where the complexity of Eq.
2 on the full dataset can be reduced to O
( B 2 MN ) , where B is the mini-batch size.
Running stochastic optimization will then take
O ( B 2 MNK ) , where K is the number of optimization epochs.
C O BJECTIVE FUNCTIONS
In §3, we presented NLL and KL divergence as objectives.
Below we derive the KL divergence between a regular multivariate Gaussian and a degenerate multivariate Gaussian, which is the case for most of our matching data settings in §4.1: the number of matching data points is greater than the number of training tasks.
In the end of this section, we introduce a new objective function, combining
NLL and KL, interpreted as MAP with a data-dependent prior.
KL divergence for a degenerate multivariate Gaussian Eq.
4 of §3.2 gives the KL divergence between two Gaussians in the non-degenerate case.
In practice, when we minimize Eq. 4, we can simply remove the constants and do the following
[Equation]
Here the variables we care about, µ, k, σ 2 , only appear in mean vector µ and covariance matrix K over the matching data.
Even if the sample mean and covariance estimate N (˜ µ , ˜ K ) is degenerate, the optimization objective stays the same as reﬂected by the derivation below.
If N (˜ µ , ˜ K ) is degenerate, its base measure is at most N -dimensional rather than M -dimensional, given that there exists a full rank matrix A ∈ R M × R such that ˜ K
= AA (cid:62) ( R ≤ N ).
Note that M is the number of matching data points, N the number of training tasks and R is the rank of matrix A and ˜ K .
The KL divergence D KL (cid:16) N (˜ µ , ˜ K ) , N ( µ , K ) (cid:17) is not well-deﬁned because the base measure of
N (˜ µ , ˜ K ) is different from the base measure of N ( µ , K ) , given K is full-rank.
However, it is still possible to derive a pseudo KL divergence D ∗ KL (cid:16) N (˜ µ , ˜ K ) , N ( µ , K ) (cid:17) as below.
Let the degenerate Gaussian be
[Equation]
and the non-degenerate one be q ( x ) =
N ( µ , K ) , where | · | ∗ is the pseudo-determinant and ˜ K + the peudo-inverse of ˜ K .
We deﬁne the support of distribution p as S ( p ) =
{ ˜ µ + ˜ K 12 v | v ∈ R M } .
The pseudo KL divergence between p ( x ) and q ( x ) now becomes
[Equation]
If the covariance matrix ˜ K is in fact full rank, i.e. R = M , pseudo KL in Eq.
7 then recovers the
KL divergence D KL (cid:16) N ( ˜ µ , ˜ K ) , N ( µ , K ) (cid:17) in Eq.
4 for non-degenerate Gaussians.
If we were to minimize this pseudo KL divergence D ∗ KL (cid:16) N (˜ µ , ˜ K ) , N ( µ , K ) (cid:17) , we still would get the minimization task in Eq.
However, pseudo KL divergence D ∗ KL does not satisfy properties including nonnegativity.
In practice, we may also choose to add small epsilon values to the diagonal terms of both
K and ˜ K to make ˜ K “less degenerate” and enable the existence of D KL (cid:16) N ( ˜ µ , ˜ K ) , N ( µ , K ) (cid:17) .
Combining marginal likelihood and distance metrics It is also possible to additively combine the KL divergence with the negative log marginal likelihood objective, and treat this distance as a regularizer.
In the case of KL divergence, it is equivalent to adding a data-dependent prior on the GP itself: µ , K ∼ exp( − λ D KL ( N (˜ µ , ˜ K ) , N ( µ ,K ) ) ) /
Z for some normalization constant Z , and the posterior is p
[Equation]
[Equation]
We can then obtain an MAP estimation for the unknown functions and variables µ, k, σ 2 .
D D ETAILS OF REGRET BOUNDS
Theorem 1 is a direct result of Theorem 16 in Wang et al.
(2018b).
The only subtle difference is that we adopted a biased estimate of the covariance matrix, a factor of NN − 1 different from the unbiased estimate.
But this can be corrected in the acquisition functions.
We provide more details below.
Proposition 2.
For any M, d, N ∈ Z + , x ∈ R M × d , µ ∈ R M , V ∈ R N × M and K = V (cid:62) V , there exists a Gaussian process GP (ˆ µ, ˆ k ) such that D KL (cid:16) N ( µ , K ) , N (ˆ µ ( x ) , ˆ k ( x )) (cid:17) ≡ 0 .
Proposition 2 is easy to show.
We can train a simple memory based model for mean function ˆ µ and kernel ˆ k .
The model stores each element of vector µ and matrix K at the corresponding locations in input x .
When making a prediction on a new input x (cid:48) ∈ R d , the model simply retrieves the values of the closest element in x .
Given Proposition 2, a regret bound follows (Wang et al., 2018b).
By Proposition 2, we are able to obtain a GP ( µ, k ) where µ ( x ) and k ( x ) are equal to the sample mean and covariance on a matching dataset ( x , y ) , following the notations in §3.2.
Hence, our problem setup is consistent with the case with discrete input space in Wang et al.
(2018b).
The following theorem is a rewrite of Theorem 16 in Wang et al.
(2018b), taking into account our biased estimators.
Theorem 3.
Assume there exist constant c ≥ max x ∈ X k ( x ) and a training dataset is available whose size is N ≥ 4 log 6 δ + T + 2 .
Deﬁne
[Equation]
δ , for any t ∈ [ T ] , and ρ T = max A ∈ X , | A | = T 12 log | I
+ σ − 2 k
( A ) | .
Then, with probability at least 1 − δ , the best-sample simple regret in T iterations of meta BO with GP-UCB that uses
[Equation] [Equation]
as its hyperparameter in α GP-UCB t −
[Equation]
where η GP-UCB =
( ι T −
[Equation]
With probability at least 1 − δ , the best-sample simple regret in T iterations of meta BO with PI that uses ˆ f ∗ ≥ max x ∈ X f ( x ) as its target value satisﬁes
[Equation]
where η PI =
[Equation]
= arg min t ∈ [ T ] k t − 1 ( x t ) .
The proof can be found in Wang et al.
(2018b).
Theorem 1 is a condensed version of Theorem 3.
While Theorem 3 provides us with some understanding of HyperBO in a speciﬁc setting, in practice, we need to query in a continuous input space that goes beyond the ﬁnite set of points present in the training dataset.
It may or may not be possible to obtain data on a wide range of tasks to ensure
N ≥ 4 log 6 δ + T + 2 .
In fact, in all of our experiment, this criterion on number of tasks is not satisﬁed.
However, we still obtained good performance.
E E XPERIMENT DETAILS AND MORE RESULTS
In this section, we provide more detailed setups and empirical results on the impact of objective functions and acquisition functions in HyperBO.
All experiment setups are the same as §E.2.1: ofﬂine and holding out related tasks.
E. B ASELINES
Our ﬁrst set of baselines include those that do not use information from training tasks:
• Rand: Random search in the corresponding scaled space in Tab. 2.
• STBO:
Single task BO with a constant mean function, Matern32 kernel and PI acquisition function (same as above).
Every BO iteration, STBO optimizes the GP hyperparameters via marginal likelihood on data of the test task.
This implementation corresponds to the basic off-the-shelf BO setups.
• STBOH:
Single task GP-UCB (coefﬁcient=1.8) with constant mean, Matern52 kernel and hand-tuned prior on hyper-parameters including UCB coefﬁcient.
Speciﬁcally, log amplitude follows Normal(-1, 1), log length scale (one per input parameter) follows Normal(0,1), and log observation noise variance follows Normal(-6, 3).
The hyperparameters are postprocessed by tensorﬂow-probability
’s SoftClip bijector to constrain the values between 1-st and 99-th quantiles.
These prior distributions were manually tuned to obtain reasonable convergence rates on 24 analytic functions in COCO (Hansen et al., 2021).
The GP parameters are then optimized via maximum marginal
likelihood every BO iteration.
For multi-task BO baselines, we included scalable methods that replace the GP with a regression model that can be trained using SGD and thus scales linearly in the number of observations.
Following the multi-task setup of Springenberg et al.
(2016), we jointly trained a 5-dimensional embedding of each task, which was then added to the input of the following two models.
• MIMO: We trained an ensemble of feedforward neural networks with shared subnetworks (Havasi et al., 2020).
We used 1 shared dense layer of size 10 and 2 unshared layers of size 10.
We used tanh activation based on (Snoek et al., 2015, Figure 2).
The network has one output unit with linear activation and another with softmax(10 − 4 , 1) activation, corresponding respectively to the mean and standard deviation parameters of a normal distribution.
We trained for 1000 epochs using the Adam optimizer with learning rate 10 −and batch size 64.
• RFGP:
We used the open-source implementation of approximate GP by Liu et al.
We trained for 1000 epochs using the Adam optimizer with learning rate 10 − 3 and batch size 64.
All methods share the same input and output warping.
The input warping is done according to the scaling function in Tab. 2: η ← log η, 1 − β ← log(1 − β ) .
The output warping is done for the best validation error rate r ← − log( r + 10 − 10 ) .
E. O FFLINE BO EXPERIMENTS
In all ofﬂine experiments, we ran ofﬂine BO on the data from the test task starting from zero initial data from this task.
Each method was repeated 5 times with different random seeds to initialize its model.
We ran all methods without de-duplication to best simulate online BO.
We evaluate on regret on error rate which denotes the simple regret on the ﬁnite set of data points in each tuning sub-dataset.
E.2. H OLDING OUT RELEVANT TASKS
We ﬁrst conducted experiments in a setting where a new task dataset is presented, and a BO method is trying to tune the optimizer hyperparameters for a selected model on that task dataset.
A training dataset for meta BO is composed of at most 18 tuning sub-datasets on training tasks that do not involve the same task dataset as the test task.
All methods then proceed to solve the test task on the new task dataset.
Fig.
6 shows performance proﬁles of the BO methods described in §4.2.
The performance proﬁles show the fraction of all test tasks that each method is able to outperform a baseline criterion at each BO iteration.
6 We chose the criterion to be the median of best error rates achieved by all methods at 3 different BO iterations: 25th, 50th or 100th.
The larger the fraction of tasks at each BO iteration, the better the method is.
From all 3 criteria, we can see that MIMO is able to outperform other methods in the beginning 10 to 20 BO iterations, but its leading position soon gets surpassed by HyperBO (H* NLL and H* KL) and STBOH.
HyperBO methods are gaining a similar if not larger fraction than the best alternative, STBOH, throughout BO iterations.
Fig.
6 (c) has the most stringent performance criterion and it shows that HyperBO with the KL objective outperforms
HyperBO with the NLL objective in this set of experiments with a small margin.
And both methods in HyperBO are doing considerably better than others.
Fig.
7 illustrates the BO convergence curves of all competing methods, together with the vertical slices at the 50th and 100th iterations.
RFGP and STBO are both falling much behind Random search.
STBO trains the GP on the data that the GP suggests to query, which creates a loop that could be harmful for data acquisition.
Optimizing the marginal data likelihood on at most 100 datapoints in fact may not lead to a better model than random initialization (see Tab. 5 in §F).
Surprisingly, RFGP, though equipped with the tuning dataset and initially reached some good values, performed similarly to STBO in the end.
Clearly, the contextual information learned by RFGP did not generalize to a new task.
On the other hand, MIMO is able to obtain a slightly better error rate than STBOH.
Fig. 6 and Fig.
7 both show that learning the GP prior through data like what HyperBO does is performing much better than other meta BO methods, and it is a more principled and effective approach to obtain the GP prior than hand-tuning.
As a reference, we include Tab. 3 which shows the task-wise best validation error rates obtained by the top 5 methods in 100 BO iterations
[Table] [Table]
To more precisely quantify HyperBO’s advantage, we also computed how much faster HyperBO can get a better error rate than best alternatives, which can be different from task to task.
We found that on average, on over 50% tasks, H* NLL is at least 2.86 times faster than best non-HyperBO alternatives; while on over 57% tasks, H* KL is at least 3.26 times faster than best non-HyperBO alternatives.
Moreover, on over 73% tasks, H* NLL is at least 7.74 times faster than random search; and on over 75% tasks, H* KL is at least 6.07 times faster than random search.
E.2. E FFECT OF NUMBER OF TRAINING TASKS
We now investigate the impact of number of training tasks on the performance of meta BO methods.
In Fig 8 we show the BO simple regrets on tasks from Table 1 (except ImageNet ResNet50 2048) that use meta BO models trained on different number of training tasks.
To analyze the performance of all methods on less-related tasks, we ﬁrst remove training tasks that have the same task dataset as our current tuning task for testing, and then remove randomly selected training datasets from the rest.
HyperBO variants were able to reduce the simple regret as more training tasks are given.
Interestingly, both H* NLL and H* KL are already slightly better than Rand and STBOH when they started off with only 3 training tasks.
There are reasonable ﬂuctuations in the results but overall the trend of regret is going down as the number of training tasks increases.
MIMO also reduced regret when the number of tasks increased from 8 to 18.
RFGP, however, fails to learn from training tasks possibly because it did not learn good task embeddings for GP regression models.
E.2. E FFECT OF NUMBER OF DATA POINTS IN TRAINING TASKS
One remaining question is, how does M i in §2, the number of data points in each training tasks, affect the performance of meta BO methods.
We analyze the impact of M i by removing a portion of all data that we have access to for each task.
In particular, we set the percentage of remaining data to be 0 .
Remaining datapoints are selected uniformly randomly, which breaks the structure of matching data.
Hence we do not include H* KL in this comparison, as H* KL only makes use of matching data.
Fig.
9 shows how the simple regret changes as the fraction of training data grows.
Below 10% training data, we observe clear trend that more data lead to lower regret for both H* NLL and MIMO, and relatively no change for RFGP.
We also found that the performance of HyperBO (H* NLL) does not change much as the fraction of training data increases from 5% to 90% .
However, MIMO and RFGP suffers signiﬁcantly from more data as the fraction of training data increases from 5% to 50% .
It is not entirely clear why MIMO and RFGP have such behaviors.
One conjecture is that neural network based Bayesian linear regression models may get too conﬁdent once the amount of data reaches a certain threshold.
This means much less exploration if those models are used for BO.
E.2. T RAINING ON ALL BUT ONE TASK
We also studied the case where meta BO approaches have access to both training tasks that do not use the same task dataset and training tasks that use the same task dataset but different model conﬁgurations.
This is especially common when we do architecture search: we aim to ﬁnd the best model and we are tuning the optimizer hyperparameters for a new machine learning model given tuning data of the same task dataset on some other models.
For this section only, we added a new baseline, MAF: We refer to the meta BO method from Volpp et al.
(2020) as MAF (Meta Acquisition Function) to avoid confusion.
MAF used reinforcement learning to learn an acquisition function modeled by a neural network over a set of transfer learning tasks.
All MAF results were generated using the code from Volpp et al.
See App.
E.6 for experimental details.
As MAF takes signiﬁcantly longer to run than HyperBO and other methods, we only include its results for this section.
We carried out a series of leave-one-out experiments, where we picked one task as the BO test function and let meta BO methods train on the remaining tasks.
In Fig.
10, we aggregated results from all 23 tasks to show the trend of how each method performs.
The conclusions are similar to those from §E.2.1.
As expected, STBO without any tricks to avoid pitfalls of vanilla BO did not show very good results.
We inspected its learned GP which mimicked a
Dirac function that is ﬂat almost everywhere except some locations, and hence it got very conﬁdent that it landed at a good spot and lost its ability to explore.
STBOH, on the other hand, achieved very competitive results.
This is because it used hand-tuned priors on all of its GP parameters, although they were tuned on somewhat different problems than the ones we consider.
As part of the goals of meta learning, we would like to show that it is possible for meta BO methods to exceed or at least match STBOH.
Both HyperBO variants obtained better results than the hand-tuned STBOH.
Especially in the beginning few BO iterations, it is able to locate much better hyperparameters than all other methods.
Tab. 4 presents mean and standard error of the best validation error rates achieved in 100 BO iterations on the 23 tasks.
HyperBO and its variants were able to achieve the best performance on 20 out of 23 tasks.
In Fig.
11, we show the optimization curves of 4 individual tasks that are considered most because
E. R ESULTS ON ONLINE OPTIMIZER HYPERPARAMETER TUNING TASKS
Finally, we look into the online BO setting where we optimize over the full hypercube.
In the online setting, some combinations of hyperparameters may be infeasible to evaluate.
For example, an overly big learning rate may lead to divergence in gradients, in which case we do not obtain a valid model.
To address this, we pre-process the function values to [ − 2 , 2) such that infeasible evaluations map to − 2 , while bad evaluations approach asymptotically to − 2 .
More precisely, for each subdataset D f i , we applied for each successful y ∈ { y j ( i ) } M i j =1 the following mapping:
[Equation]
where y is the median of { y j ( i ) } M i j =1 .
[Table]
In this section
[Table]
we set HyperBO variants and STBO to share exactly the same GP-UCB acquisition function as STBOH, MIMO and RFGP.
The UCB coefﬁcient for all methods is 1 . 8 .
The variants of
HyperBO are as follows:
• H* NLL: HyperBO with UCB as the acquisition function and negative log marginal likelihood (NLL) as the objective function.
• H*
NLLKL: HyperBO with UCB as the acquisition function and NLL plus 10 times KL divergence on matching datapoints as the objective function.
See §C for more details.
In Fig.
12, we include the online tuning results for selected tasks due to limited compute resources.
We noticed that for some methods, e.g. STBO and MIMO, it is very difﬁcult for them to recover from a “bad” datapoint.
This is partly because predictions from these models are signiﬁcantly tied to the initial observations.
For example, STBO may overﬁt to the initial bad value and believe there are bad values in the entire search space.
Nevertheless, in 7 out of 9 tasks, HyperBO methods performed the best among all methods being compared.
E. I MPACT OF OBJECTIVE FUNCTIONS
Here we investigate how different objective functions in HyperBO can impact its performance.
Besides NLL and KL, which are already described in details in §3, we also include NLL+KL, which corresponds to Eq.
8 with λ = 10 .
λ
= 10 is an arbitrary choice, and one may ﬁnd other better options to set λ in Eq. 8.
Figure 13 shows the performance proﬁles and BO simple regret curves of NLL, KL and NLL+KL when HyperBO uses different acquisition functions.
As comparisons, we also included the better performing baselines: Rand, STBOH and MIMO.
While the ranks of performance by different objectives do vary depending on which acquisition function we used, it is clear that all HyperBO variants outperform the baselines.
For EI and PI, the KL objective gives better performance, but NLL or NLL+KL might be preferred for UCB with coefﬁcient 2, 3 or 4.
E. I MPACT OF ACQUISITION FUNCTIONS
As explained brieﬂy in §
, we used 5 acquisition functions in our experiments: the vannila EI method,
PI with coefﬁcient 0.1, α PI (cid:16) x ; GP (ˆ µ, ˆ k | D f ) (cid:17) =
ˆ µ Df ( x ) − max t ( y t +0 . 1) ˆ σ Df ( x ) , and UCB with coefﬁcient ζ = 2 , 3 and 4 in α UCB (cid:16) x ; GP (ˆ µ, ˆ k | D f ) (cid:17) =
ˆ µ D f ( x ) + ζ ˆ σ D f ( x ) .
Our goal is to verify that HyperBO maintains good performance across different choices of acquisition functions.
To do so, we investigated in the performance of HyperBO variants under different objectives.
We avoid over cluttering the ﬁgures by only including STBOH as baseline, since it is roughly the best baseline according to the main results in Fig.
As shown in Figure 14, HyperBO with EI and PI as acquisition functions perform relatively better than HyperBO with UCB variants.
However, HyperBO with UCB3 can still be very competitive when it is coupled with NLL objective.
Overall, HyperBO with all of the 5 acquisition function options outperforms the best performing baselines.
E. MAF IMPLEMENTATION DETAILS
We compared to (Volpp et al., 2020) using the code and default hyperparameters provided by the authors.
This code assumes that each task is additionally accompanied by the optimal set of hyperparameters for the GP used to model the task (including the task used for evaluation).
Following the MAF approach, we learned these hyperparameters using the GP Y library, and provided them to the MAF algorithm.
Given that MAF takes signiﬁcantly longer to run than HyperBO, each subdataset was evaluated using only one random seed.
Each neural acquisition function was trained for a total of 1000 iterations.
As was done in (Volpp et al., 2020), we selected the optimal training iteration for the neural acquisition function by cross-validation on the transfer learning tasks; in this case, we randomly sampled 3 transfer learning task, and chose the training iteration with the lowest average simple regret.
Finally, to reuse the MAF code, we also had to ensure that (a) each subtask had the same number of evaluation points, and (b) that there were no duplicated hyperparameters.
For this reason, we ﬁrst removed all duplicate hyperparameters within each subdataset, then capped each subdataset to the ﬁrst 1559 points (the size of the smallest sub-dataset).
F D ISCUSSION
In this work, we focused on the question of how to make use of multi-task data to enable better
Bayesian optimization.
For the convenience of our investigation, we made simpliﬁcations such as sequential evaluations and a shared search space across tasks, although these are mostly unnecessary (see below).
Our method also relies on an important assumption: functions of all tasks are i.i.d.
samples from the same GP.
In this section, we explore how reasonable the i.i.d.
assumption is and discuss extensions to our work that would enable even more ﬂexible uses.
Assumption on i.i.d.
GP samples.
To get a better idea on how much our assumptions helped on training the GP, we compare NLLs associated with 23 tasks in §4.1 with models obtained viascenarios:
(a) No training: a randomly initialized model with no training; (b) Single task: models trained on 100 randomly selected data points of the test task; (c) H*: models trained on 18 irrelevant tasks selected in §E.2.
Here case (c) corresponds to the method HyperBO used for training a GP and case (b) corresponds to the model STBO can obtain with 100 initial observations.
In Tab. 5, we show NLLs of thesemethods on all tasks 8 and NLLs on the test task.
Note that the held-out tasks for some test tasks are the same because of the held-out rules in §E.2.1.
Comparing NLLs of the test tasks using models without training and trained via marginal likelihood like STBO, it is perhaps surprising to see that training on a subset of data points of the sub-dataset of the test task not only did not contribute to lowering NLL on the entire sub-dataset, but it even made it worse in 20 out of 23 test tasks.
The training process by optimizing the NLL on a part of a sub-dataset leads to severe over-ﬁtting.
We can observe the same results of NLLs on all tasks.
Without any training, our NLL is 148211 .
Yet single-task training leads to higher NLLs for all models trained on different sub-datasets.
Our method H*, on the other end, consistently achieves lower NLLs on both the test task and all tasks.
Although it is not entirely clear what the relation is between a better NLL of the GP and better BO results, achieving lower NLLs typically means that the model has a better ﬁt to the dataset.
Hence, by the assumption of typical BO methods, the test function should look like a sample from our model, and so lower NLLs of model will contribute to matching the assumption of typical BO methods.
By enhancing the assumption with ours on i.i.d.
GP samples, Tab. 5 shows we then will be able to obtain models with a much better ﬁt to the data.
model (No training) on all tasks is 148211 .
The KL value of randomly initialized model (No training) is 2177 .
Training on a subset of a sub-dataset in the test task (Single task) often leads to much worse marginal likelihood on the entire sub-dataset.
Training on irrelevant tasks (H*) achieves much lower (pseudo) KLs on matching datasets and lower NLLs for both the test task only and all tasks
[Table]
4 MNIST CNNPoolTanh12147 .
.556254 .−
606 .1255 .−
9 MNIST CNNPoolReLU26870 .
.1542 .113 .−
4 MNIST CNNPoolReLU15601 .6734 .
.−
782 .361 .−
5 MNIST CNNReLU13939 .
.3153 .743233 .−
231 .877 .− 61 .
7 MNIST CNNReLU10111 .
.1365 .977295 .−
779 .1373 .−
2 Fashion CNNPoolTanh2072 .11433 .−
381 .1139702 .−
1051 .1910 .−
8 Fashion CNNPoolTanh2800 .4115 .−
251 .1278018 .− 1051 .4208 .−
8 Fashion CNNPoolReLU4677 .725 .−
405 .69173 .−
1051 .205 .−
8 Fashion CNNPoolReLU3925 .4254 .−
755 .296739 .−
1051 .1027 .−
37 . 8 Fashion CNNReLU4667 .6778 .251 .193488 .−
1051 .597 .− 37 .
8 Fashion CNNReLU3295 .29348 .−
235 .1526829 .− 1051 .3341 .−
8 CIFAR100 WRN1271 .15813 .−
[Table]
We also computed the (pseudo) KL divergence across all matching datasets in the last columns of
Tab. 5.
See Appendix C for a comprehensive analysis on pseudo KL divergence for degenerate multivariate Gaussians.
Note that pseudo KL divergence can be negative.
Here we use pseudo KL divergence if required by the matching dataset.
Again, single-task training leads to unstable (pseudo)
KL values, sometimes even higher than without training ( 2177 . 2 ).
On the contrary, training with H* leads to much more stable and lower values for KL.
This indicates that the model learned to predict similarly to the sample mean/covariance estimate, which is known to help better selection of BO query points by Theorem 1.
Batch evaluation.
For simplicity of this paper, we did not consider batch evaluation but rather only focused on the prior selection dimension of the challenges in BO.
However, it is straightforward to adopt any batch BO methods in conjunction with HyperBO to support obtaining observations in parallel.
For example, we can directly use batch methods in Snoek et al.
(2012), Kathuria et al.
(2016), or Wang et al.
(2017) to replace line 5 of Alg.
High-dimensional and large scale data.
Similar to batch BO, our method can also be naturally combined with most high-dimensional and large scale BO methods to offer more capabilities.
For these cases, typically a probabilistic model different from vanilla GPs may be adopted.
In line 2 of
Alg. 1, it is straightforward to adapt our method to optimize the cumulative marginal likelihood in
Eq.
3 instead for the new model.
Our meta-learning idea in this paper in fact also brings beneﬁt to high-dimensional and large scale BO methods so that they can better identify their critical special structures, e.g. low-dimensional embedding Wang et al.
(2016), cylindrical kernels Oh et al.
(2018) or additive Mondrian kernels Wang et al.
(2018a).
Different search spaces.
Roughly speaking, there could be two circumstances for difference search spaces.
Case I is that tasks share the same search variables, but the search ranges for some variables are different.
For example, we may have each function f
i : X i → R , i ∈ [ N ] and
X i = (cid:81) dj
=1 [ l ij , h ij ] ⊂ R d .
In this case, our solution still applies by simply setting a union search space as X =
(cid:83) Ni =1 X i for learning and use the designated search space of new tasks for optimization.
Case II is more complicated: the search space for each function f
i is X i ⊂ R
d i and each dimension of X i may have a different meaning than another search space
X j
( i (cid:54) = j ).
This paper does not have a solution for this scenario.
Further research will be needed to reduce Case II to Case I which can be then immediately combined with HyperBO.