T RANSFER L EARNING FOR B AYESIAN HPO WITH E ND - TO -E ND M ETA -F EATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
Hyperparameter optimization (HPO) is a crucial component of deploying machine learning models, however, it remains an open problem due to the resourceconstrained number of possible hyperparameter evaluations.
As a result, prior work focuses on exploring the direction of transfer learning for tackling the sample inefﬁciency of HPO.
In contrast to existing approaches, we propose a novel
D eep K ernel Gaussian Process surrogate with L andmark M eta-features (DKLM) that can be jointly meta-trained on a set of source tasks and then transferred efﬁciently on a new (unseen) target task.
We design DKLM to capture the similarity between hyperparameter conﬁgurations with an end-to-end meta-feature network that embeds the set of evaluated conﬁgurations and their respective performance.
As a result, our novel DKLM can learn contextualized dataset-speciﬁc similarity representations for hyperparameter conﬁgurations.
We experimentally validate the performance of DKLM in a wide range of HPO meta-datasets from OpenML and demonstrate the empirical superiority of our method against a series of stateof-the-art baselines.
I NTRODUCTION
Hyperparameter optimization (HPO) is an essential open problem in machine learning (ML) due to the black-box nature of methods’ empirical performances as a function of their hyperparameters.
The major challenge lies in the computational infeasibility of training and evaluating a large sample of hyperparameters in order to identify the best generalization performances.
As a result, transfer learning lends itself as a promising direction for improving the sample efﬁciency of HPO methods (Wistuba et al., 2016; Perrone et al., 2018; Wistuba & Grabocka, 2021).
Prior approaches for transfer learning in HPO rely on exploring existing evaluations on a pool of datasets where the model under investigation is evaluated.
The similarity between datasets is often captured via features describing their characteristics (a.k.a.
meta-features), such as descriptive statistics of dataset features (Michie et al., 1994; Wistuba et al., 2016), or landmark measures in the form of the accuracies gathered from a set of basic classiﬁers (nearest neighbors, decision trees,
SVMs, etc.)
on the datasets (Pfahringer et al., 2000; Feurer et al., 2014).
A recent trend highlights the potential of learning parametric meta-feature extractors for tabular datasets (Jomaa et al., 2021a), which are further meta-trained (Finn et al., 2017) to improve the HPO transferability to new datasets (Jomaa et al., 2021b).
Unfortunately, typical transfer learning from a set of unrelated source tasks suffers from the negative transfer phenomenon (Wang et al., 2019), which implies a poor generalization performance on target tasks that are dissimilar to the source tasks, according to a predeﬁned dissimilarity measure, e.g. similarity of response curves.
This can happen, for example, when a model is learned jointly across tasks without task-speciﬁc attributes.
To resolve the negative transfer of HPO performance predictors (a.k.a.
surrogates)
we introduce a novel direction that conditions Gaussian
Process (GP) surrogates in Bayesian Optimization on the meta-features of datasets.
In that manner, we can transfer knowledge only from similar datasets, and hence conditioned on the similarity of meta-features.
However, in contrast to ad-hoc dataset meta-features that are hand-crafted by domain experts, we propose a novel architecture for deep GP kernels (Wilson et al., 2016a) that are enriched with novel end-to-end neural network components that generate meta-features only from the tuples of past hyperparameter conﬁgurations and their evaluated performances.
Our meta-feature network is a set-based neural network that is invariant to the permutation/sequence of past hyperparameter evaluations.
We jointly train D eep K ernel Gaussian Process surrogate with L andmark M eta-features (DKLM) through HPO meta-learning (Wistuba & Grabocka, 2021).
To validate the empirical performance of our method we present extensive results on a large-scale benchmark that involves 16 different search spaces and 101 datasets from OpenML for a total of 3.4 million hyperparameter evaluations (PinedaArango et al., 2021).
Detailed experiments against a series of traditional HPO methods, as well as recent transfer HPO baselines, demonstrate the superiority of meta-learning the initialization of
DKLM.
Overall, we make the following contributions:
• Introduce the ﬁrst paper that tackles the negative transfer phenomenon in Bayesian HPO, by conditioning GP surrogates on meta-features, i
e. on dataset characteristics;
• Propose an end-to-end deep GP which implicitly learns networks that generate metafeatures, with no ad-hoc inductive bias from experts on manually designing meta-features; • Demonstrate the empirical superiority of our method on a very-large-scale experimental protocol (3.4 million hyperparameter evaluations), against a large number of baselines.
R ELATED W ORK
Hyperparameter optimization (HPO) has been extensively studied over the past decade for improving the performance of machine learning models beyond simple search techniques (Larochelle et al., 2007; Bergstra & Bengio, 2012).
Non-transfer learning solutions often deﬁne a probabilistic surrogate that estimates the true hyperparameter response surface using Gaussian Processes (Rasmussen, 2003), Bayesian Neural Networks (Snoek et al., 2015; Springenberg et al., 2016), or tree-based models (Hutter et al., 2011; Bergstra et al., 2011).
Hyperparameters are then selected via an acquisition function (Wilson et al., 2017) and the process is reiterated with the new set of observations until a speciﬁed budget is exhausted, e.g. runtime or number of trials.
HPO is further expedited when deﬁned within the context of transfer learning, i.e. by leveraging related tasks (or datasets) to improve the generalization over unseen tasks.
Transfer learning for HPO has been observed by modeling tasks jointly (Swersky et al., 2013; Yogatama & Mann, 2014; Perrone et al., 2018; Salinas et al., 2020), or through some weighted-combination of the surrogates (Schilling et al., 2016;
Wistuba et al., 2016; Feurer et al., 2018).
Other directions include pruning the hyperparameter search space (Wistuba et al., 2015a; Perrone & Shen, 2019), or learning to initialize the surrogate by identifying good initial hyperparameters (Wistuba et al., 2015b).
Apart from learning a transferable surrogate, recently, transferable acquisition functions (Wistuba et al., 2018; Volpp et al., 2020) have also been proposed to replace engineered acquisition functions.
The success of meta-learning for domain adaptation has also been investigated for HPO.
Wistuba & Grabocka (2021) explore few-shot Bayesian Optimization by learning a deep kernel Gaussian
Process surrogate across a set of tasks to quickly adapt to new a target task.
Similarly, Jomaa et al.
(2021b) learn a shared neural network surrogate jointly coupled with a meta-feature extractor deﬁned over the dataset itself.
Meta-features (Vanschoren, 2018), or dataset characteristics, have also been widely adopted in HPO algorithms for warm-start initialization (Feurer et al., 2015; Wistuba et al., 2016) or as additional attributes to better marginalize the surrogate on individual tasks (Bardenet et al., 2013).
Nevertheless, extracting meta-features requires direct access to the datasets, which might be difﬁcult in real settings where only the meta-dataset is available.
In this paper, we propose to extract landmark meta-features from existing evaluations (Leite et al., 2012; Sun & Pfahringer, 2013) in an end-to-end fashion using a deep Gaussian kernel approach.
P RELIMINARIES 3. H YPERPARAMETER O PTIMIZATION
We denote by D = { ( x i , y i ) } ni =1 a task of interest, such that x i ∈ X ⊆ R n represents a hyperparameter conﬁguration in the domain of a (bounded) hyperparameter search space for some model under investigation.
Furthermore, let y i
= f
( x i ) + (cid:15) be the response of an unknown black-box function f :=
X → R + , with (cid:15) as an additive i.i.d Gaussian noise with some homoscedastic variance σ 2 .
Typically, y represents a metric of interest that should be optimized to obtain better model generalization, e.g. validation loss.
The objective of hyperparameter optimization is then to ﬁnd the optimal hyperparameter such that x ∗ =
arg min x ∈X f ( x ) given a ﬁxed budget T of trials.
HPO is commonly treated as sequential decision-making process, where a surrogate model
ˆ y :
X → R is iteratively ﬁt to the history H t := { ( x i , y i ) } ti =1 of evaluated hyperparameters and a policy (or acquisition function)
A : ( X × R ) ∗ → X is used to select the next candidate which minimizes the expected hyperparameter response.
Among the existing acquisition functions, expected improvement is widely adopted (Mockus, 1974).
3. D EEP K ERNEL G AUSSIAN P ROCESSES
Given a training task D
= { ( x i , y i ) } ni =1 , the response can be modeled using a Gaussian process (GP), i.e. as a multivariate Gaussian distribution, such that y ∼ N ( m ( X ) , k
( X, X )) .
A GP is a non-parametric approach that deﬁnes a prior over functions directly, and is deﬁned by its mean function, m , and kernel function k .
Given some observed data points, it is possible to compute the posterior over these functions to approximate unobserved data points as
[Equation]
with K n = k
( X, X | θ ) + σ 2 n I , K ∗
=
k
( X, X ∗ | θ ) , and K ∗∗ = k
( X ∗ , X ∗ | θ ) .
The mean and covariance of the posterior predictive distribution is then estimated as
[Equation]
The standard approach of ﬁtting GPs is to optimize the weights of the kernel function, e.g. squared exponential kernel, θ .
Nevertheless, these engineered kernels are often employed under false assumptions (Cowen-Rivers et al., 2020), which leads to sub-optimal performances.
Recently deep kernel learning (Wilson et al., 2016b) has emerged as a powerful extension that leverages the representative capacity of non-linear function approximation, e.g. neural networks, and facilitates learning the kernel directly.
Speciﬁcally, we denote by φ : X → R N a mapping from the domain to a latent space which serves as an input to the kernel, such that:
[Equation]
where w represents the parameters of φ .
The weights θ and w are then jointly optimized for maximizing the marginal likelihood (Wistuba & Grabocka, 2021).
D EEP K ERNEL G AUSSIAN P ROCESS WITH L ANDMARK M ETA - FEATURES
Inspired by landmark meta-features (Pfahringer et al., 2000), which are typically estimated by measuring the response of given datasets to machine learning algorithms, we propose a novel deep kernel GP that is conditioned on task-speciﬁc landmark meta-features.
However, instead of computing meta-features through ad-hoc approaches, we introduce a novel parametric meta-feature extractor network that is integrated into the kernel function of a GP and subsequently meta-learned over a set of source tasks together with the parameters of the GP kernel.
In that manner, we learn meta-features that describe a set of tasks in terms of minimizing the estimation of the tasks’ response functions.
By adding the task-speciﬁc information of the meta-features, the GP surrogate can infer a more accurate response surface on a new task based on similar source tasks that share similar meta-features.
Therefore, our method is the ﬁrst to tackle the negative transfer phenomenon for Bayesian HPO.
4. L ANDMARK M ETA -F EATURE N ETWORKS
We propose a simple idea to learn landmark meta-features by learning a deep representation of the evaluated set of hyperparameter-response pairs as part of a deep kernel Gaussian process.
With the success of set-based algorithms (Lee et al., 2020; Zaheer et al., 2017; Lee et al., 2019) for function approximation, we propose to use a Deepset (Zaheer et al., 2017) formulation that provides a ﬁxedsize vector representation from the dynamic set of observations.
Although other methods have been developed for set-based function estimation, we focus here on deepsets because they have already been shown to perform well for learning meta-feature in task-agnostic settings (Jomaa et al., 2021a) as well as for hyperparameter optimization (Jomaa et al., 2021b).
Suppose that we are given a collection of data points { ( x i , y i ) } ni =1 where x ∈ X is an observed hyperparameter and y ∈ R its corresponding response.
We denote by H t − 1 :
= { ( x i , y i ) } t − 1 i =1 an associated set of data points that have been observed prior to x t .
Furthermore, we formulate the proposed meta-feature network as:
[Equation]
where [ ] symbolizes standard concatenation, φ 2 : ( X × R ) ∗ → R N and φ 1 : X × R N → R M are parametric neural networks with respective weights w , and where ( X × R ) ∗ represents the set of evaluated hyperparameter and their responses.
With this formulation, we ensure that the relationship of the covariates and the responses in H t − 1 is properly encoded, and thus φ is conditioned on these latent representations.
It is also important to note that φ 2 is permutation invariant, i.e. φ 2 ( H t − 1 ) =
φ 2 ( π ( H t − 1 )) , with π :=
( X × R ) ∗ → ( X × R ) ∗ as a random permutation function.
This is critical, as the ordering of the data points in H t − 1 should not affect the landmark meta-features.
Additionally, given φ 2 ( H t − 1 ) , this information about the marginal distribution of the meta-features can be encoded with the speciﬁc attribute x which in turn allows the deep kernel GP to be marginalized over individual tasks given the context, and thus transfer (joint) learning becomes easier with minimal overhead.
4. M ETA - LEARNING OUR D EEP GP S
The parameters θ and w are optimized jointly by maximizing the following log marginal likelihood:
[Equation]
By using established practices (Wistuba & Grabocka, 2021; Patacchiola et al., 2020), we can optimize Equation 7 in terms of w, θ via stochastic gradient descent (SGD), that is proven to maintain convergence guarantees (Chen et al., 2020).
We direct the interested reader to the prior work for more details on optimizing the parameters of deep GPs (Wilson et al., 2016b).
Given the diverse number of tasks, which vary in the number of available data points, we propose to jointly learn the shared surrogate via ﬁrst-order meta-learning (Nichol et al., 2018).
Meta-learning has found resounding success in the research community as an initialization scheme (Finn et al., 2017; Wistuba & Grabocka, 2021; Jomaa et al., 2021b), which allows for fast adaption to new domains.
Consequently, the meta-trained model resides on a joint minimum across all the source tasks, such that given limited information about the new (unseen) target task, it can converge faster to the new task’s local optima.
In this direction, we show the pseudocode of our meta-learning optimization in Algorithm 1.
M OTIVATION
Meta-features help to model the posterior uncertainty.
To motivate our approach, we present an ablation of the effect of our deep GP kernel with meta-features, compared to the same deep
GP kernel without meta-features (i.e. Ours vs. FSBO (Wistuba & Grabocka, 2021)).
We created a synthetic meta-dataset of K = 50 tasks in the form of randomly sampled sinusoidal functions
1: Require: training dataset E ; kernel parameters θ , network parameters w ; learning rate η ; inner update steps v ; meta-batch size n , batch size b .
2: while not converged do 3: t ∼ U ([ T min , T max ]) 4:
D 1 , . . .
, D n ∼ U
, D ]) 5: for i = 1 to n do 6:
Sample t − 1 data points to form H t − 1 ∼ D i 7:
Sample batch B := { ( x i , y i ) } bi =1 ∼ D
i 8: θ i ← θ ; w i ← w 9: for j = 1 to v do 10:
Deﬁne as L the objective function of Equation11: θ i ← θ i + η ∇ θ L 12: w i ← w i + η ∇ w L 13:
Update θ ← θ + η 1 n (cid:80) ni =
( θ i − θ ) 14:
Update w ← w + η 1 n (cid:80) ni
=1 ( w i − w ) f k ( x )
= a ( k ) sin( x + b ( K ) ) , k ∈ { 1 , . .
, K } by drawing each a ( k ) ∼ U (0 . 1 , 5) and b ( k ) ∼
U (0 , 2 π ) .
Furthermore, we meta-learn our deep GP on these source functions and then transfer the surrogate as an initialization for a new sinusoidal curve (with new parameters a, b ) as shown in
Figure 1 (top).
We show the comparison of our surrogate with meta-features after 3 trials (for a total of 8 data points, including 5 initial conﬁgurations) to an FSBO deep GP that has been meta-trained identically.
We notice that our surrogate (bottom row, leftmost plot) computes a better posterior variance compared to FSBO (bottom row, middle plot).
The effect of the superior modeling of the uncertainty leads to better exploration in a Bayesian Optimization setup, and consequently to better empirical accuracies of the discovered hyperparameters (as will be shown in Section 6).
Meta-features capture task characteristics.
We postulated that our proposed meta-features can capture task characteristics.
To illustrate the argument, we create another simpler collection of
K = 50 sinusoidal functions
f k ( x ) = a ( k ) sin( x ) , k ∈ { 1 , . . .
, K } by drawing each a ( k ) ∼
U (0 . 1 , 5) .
As these sine waves change only in terms of the amplitude parameter a , then if we metatrain our meta-feature network with a 1-dimensional (1D) output layer from these source functions, it must strongly learn to correlate the 1D meta-feature with the sinusoidal amplitude a .
As can be seen in the rightmost plot of the bottom row in Figure 1, this is exactly the case.
In this plot, the y-axis shows 1D meta-feature values computed from only 5 random pairs of conﬁgurations and responses ( x, f ( x )) from one random task, and the x-axis shows the amplitude of that respective task.
Although our meta-feature networks have no design bias in terms of modeling sinusoidal functions, they are perfectly able to extract a latent representation of the amplitude, based on the end-to-end meta-learning of the deep GP for approximating random observations on the sine waves.
E XPERIMENTS
Our experimental protocol is designed to primarily address one simple research question: Do deep
GPs with our meta-feature networks outperform state-of-the-art HPO algorithms in the transfer and non-transfer learning settings?
6. M ETA - DATASET AND B ASELINES
We evaluate our approach on HPO-B-v3, a new hyperparameter optimization benchmark designed for comparing black-box HPO methods (Pineda-Arango et al., 2021).
The benchmark contains a collection of 935 black-box tasks for 16 hyperparameter search spaces (algorithms) evaluated ondatasets and divided into predeﬁned training, validation, test splits.
Following the same experimental protocol speciﬁed at the HPO-B benchmark, we compare our approach to the following large set of 10 HPO baselines:
Random Search (Bergstra & Bengio, 2012); 2.
GP (Rasmussen, 2003) is a hyperparameter tuning strategy that relies on a Gaussian Process as a surrogate model with squared exponential kernels (Matern 5/2 kernel) with automatic relevance determination; 3.
DNGO (Snoek et al., 2015) utilizes a neural network to extract adaptive basis function of hyperparameters, which in turn are fed to a Bayesian linear regression model to generate a posterior distribution; 4.
BOHAMIANN (Springenberg et al., 2016) is based on Bayesian neural networks that are trained via a stochastic gradient Hamiltonian Monte Carlo; 5.
DGP (Patacchiola et al., 2020) ﬁts a deep kernel Gaussian process as a surrogate; 6.
TST-R (Wistuba et al., 2016) is an ensemble approach where the Gaussian process surrogate of the target task is weighted with surrogates of the training datasets based on the ranking similarity of the evaluated hyperparameters; 7.
RGPE (Feurer et al., 2018) is another ensemble approach, similar to TST-R, which estimates the weights by optimizing a ranking loss between the surrogates of the training datasets and that of the target task; 8.
ABLR (Perrone et al., 2018) is a multi-task Bayesian linear regression approach that optimizes a shared feature extractor across the training datasets as an initialization strategy for the target task; 9.
GCP+Prior
(Salinas et al., 2020) utilizes a Gaussian Copula process (Wilson & Ghahramani, 2010) trained jointly on the training tasks, where a quantile-transformation is applied on their respective responses.
The pretrained process is used as parametric prior for the target dataset; 10.
FSBO (Wistuba & Grabocka, 2021) uses deep Kernel Gaussian processes (Patacchiola et al., 2020) to estimate the response of the target dataset.
The parameters are initialized via meta-learning the joint response surface over the training datasets.
In a nutshell, our experimental protocol based on HPO-B is a large scale one by the standards of the prior papers, as it involves 10 baselines, 16 search spaces (algorithms whose hyperparameters we tune), 101 datasets, and totally 935 black-box tasks containing 6.3 million evaluations.
6. I MPLEMENTATION D ETAILS
We implement the Deep Kernel Gaussian Process using GPyTorch 1.5 (Gardner et al., 2018) with a
Matern 5/2 kernel.
As described in Equation 4, DKLM is composed of two modules, φ :=
φ 1 ◦ φ 2 .
The parameters of the network have been selected based on the performance on a held-out validation set.
Function f is 4 dense layers with 32 hidden units and ReLU activation functions, whereas g is 1 dense layer with 32 hidden units.
Finally, φ 1 is 4 dense layers with 32 hidden units and ReLU activation.
All parameters of the Deep Kernel are estimated by maximizing the marginal likelihood.
We achieve this by using gradient ascent and Adam (Kingma & Ba, 2015) with a learning rate of 0 .
001 and batch size of 64 with t ∈ [2 , 100] .
6. HPO R ESULTS AND D ISCUSSION
We start by comparing non-transfer HPO methods to our deep GPs with meta-features DKLM (RI) that are randomly initialized (no meta-learning).
As depicted in Figure 2, DKLM (RI) outperforms the baselines after 100 hyperparameter trials in terms of both the mean normalized regret and the mean rank metrics.
We also notice how the performance improves gradually with the increasing number of trials, indicating the impact of the posterior variance modeling of our method (Section 5) as more observations are present on the black-box responses.
Afterwards, we demonstrate the comparison of state-of-the-art transfer against our method in Figure 3.
DKLM outperforms the rest of the baselines with lower mean normalized regret and lower mean rank.
The superiority of landmark meta-features also becomes evident after a larger number of trials (more than 50) .
To further inspect the results we show the performance of DKML and all other baselines in the selected individual search spaces of Figure 4.
We notice primarily that meta-learning the initialization in DKLM improves the general performance in most cases.
Nevertheless, we notice in 4796 that effect of transfer learning is not evident, as DKLM (RI) and Deep Kernel GP are better than the meta-initialized variants, DKLM, and FSBO respectively.
Still, landmark meta-features prove yet again that they are effective in a single task setting.
6. E ND - TO -E ND L ANDMARK M ETA - FEATURES
To motivate the importance of landmark meta-features, we illustrate in Figure 5 the 2D latent dimensions of the landmark meta-features for every test task in the 16 spaces of HPO-B-v3.
Each point on the graph represents a set of meta-features extracted from 100 randomly sampled data points, i.e. t = 100 , from the individual tasks after meta-initialization of the weights of DKLM.
We observe that the same color-coded meta-features, i.e. belonging to the same task, lie generally within the vicinity of each other, and distant from other tasks.
As pointed out by Jomaa et al.
(2021a), any meta-feature extractor should be able to preserve inter-and intra-dataset similarity, a property that is evident here.
6. R ESILIENCE TO N EGATIVE T RANSFER
To test the phenomenon of negative transfer we design a speciﬁc ablation experiment, where the datasets in the meta-test split are uncorrelated to the datasets in the meta-train split.
First of all, we deﬁne the correlation of two datasets using Kendall’s τ correlation coefﬁcient (Kendall, 1945).
We sample 100 random hyper-parameter conﬁgurations and query the validation accuracies of these conﬁgurations on both datasets from an HPO-B search space, to compute the Kendall’s τ coefﬁcient between the accuracies’ vectors.
Afterward, for every HPO-B search space, we i) compute the correlation between each meta-test dataset and all the meta-training datasets, ii) keep the 10% of the meta-training datasets that have the smallest mean Kendall’s τ coefﬁcient to the meta-test datasets.
We report the ablation results in Figure 6, where DKLM (NT) and FSBO (NT) denote methods trained using only the 10% most uncorrelated meta-training datasets.
In contrast, DKML and FSBO are trained using all the 100% meta-training dataset.
We notice that performance gap of the ranks between DKLM (NT) and FSBO (NT) is larger than that between DKLM and FSBO, validating
DKML’s resilience to the negative transfer phenomenon.
Additionally, we report the the critical difference diagrams @100 trials.
We notice that the performance of DKLM (NT) is statistically signiﬁcant compared to FSBO (NT), further illustrating the resilience of DKLM to negative transfer.
L IMITATIONS
Despite the fact that our method signiﬁcantly reduces the time for ﬁtting Machine Learning, we caution practitioners against overtuning their model for a large number of conﬁguration trials, only to get a very small improvement in accuracy, unless it is absolutely necessary from a business need.
C ONCLUSION
In this paper, however, we propose DKLM as a simple yet effective method to better condition deep kernel Gaussian Processes on tasks.
Inspired by landmark meta-features, we design a set-based meta-feature extractor that captures the interaction between the available hyperparameters and their respective responses, and consequently generates distinct task-speciﬁc attributes.
DKLM is metalearned on a set of source tasks in an end-to-end fashion to jointly approximate the response surface over the shared hyperparameter and landmark meta-feature space.
We show in a battery of experiments the signiﬁcance of landmark meta-features, outperforming state-of-the-art HPO baselines in non-transfer and transfer learning settings.
E THICS S TATEMENT
In our work, we use only publicly available data without privacy concerns.
Furthermore, our algorithm reduces the overall time for ﬁtting machine learning algorithms, therefore, saving computational resources and yielding a positive impact on energy consumption.
R EPRODUCIBILITY S TATEMENT
We promote reproducibility as detailed below:
• We use only publicly available datasets.
• All our baselines are publicly available and provided by the HPO-B benchmark (PinedaArango et al., 2021).
• We clearly describe our method in Section 4 and provide implementation details in Section 6.2.
Finally, we plan to make the source code of our method publicly available.
A A LGORITHMS
We describe in Algorithm 2 the general procedure for BO optimization.
Our surrogate model,
DKLM, is ﬁt to the history of observations H t − 1 at each trial via Algorithm 3.
Input
Output do
Fit φ and kernel
H t −
Algorithm 3 ;
Obtain evaluation
A
Observe
; 6 end 7 Return
Input: Learning rates η 1 and η 2 , N evaluations H
= { ( x i , y i ) }
Ni =0 on target task, kernel parameters θ , network parameters w
Output: Gaussian Process posterior p ( f ∗ | x ∗ , H ) 1 while not converged do
Compute marginal likelihood L on H (Equation 6); θ ← θ + η 1 ∇ θ L ; w ← w + η 2 ∇ w L ; 5 end 6 Compute Gaussian Process posterior p ( f ∗ | x ∗ , H ) = N ( µ, σ 2 ) (Equation 2)
B E XPERIMENTAL P ROTOCOL
We followed the protocol presented in HPO-B, (Pineda-Arango et al., 2021), a benchmark collection of tasks collected from OpenML.
The tasks are grouped in 16 search spaces, and every search space contains three splits: meta-training, meta-validation and meta-testing.
According to the authors, the splitting for the percentage of tasks is 80%, 10% and 10%, respectively.
We refer to the repositoryto know the exact task IDs for every split.
In our experiments, we meta-train a model per search space, whereas we used the meta-validation data for early-stopping.
We use η = 0 .
001 , v = 1 , n = 1000 , T min = 2 , T max = 100 , b = 64 for meta-training (Algorithm 1).
We optimize the black-box function following Sequential Model Based
Optimization (SMBO, Algorithm 2), we use Expected Improvement as the acquisition function:
[Equation] [Equation]
which uses the mean and variance computations presented in section 3.2 for estimating the expectation
We ﬁt DKLM using ADAM (Kingma & Ba, 2015) with learning rates η 1 =
η 2 = 0 .
001 , (see Algorithm 3).
We use the meta-validation dataset for choosing the number of hidden layers in
Equation 5 and the learning rates η .
C I MPACT OF S AMPLE S IZE
We mentioned in Section 6.5 that the performance of DKLM improves with the increasing number of observed data points.
To further examine this observation, we present in Table 1 the negative log likelihood given a history of size t .
To get these results, we sample t points for the history H t and compute the negative log-likelihood on 100 separate data points upon meta-initialization of DKLM.
The process is repeated 100 times for each task, for a total of 10000 data points.
We highlight the trend of the negative log-likelihood by ﬁtting a linear model to the 10000 points and reporting the slope.
In most search spaces, the performance improves with the increasing number of observed data points, where we notice some over-ﬁtting in others, e.g. space 5860 that can be alleviated by further regularization.
[Table]
D D ETAILED E XPERIMENTAL RESULTS
We present the detailed results for the 16 search spaces in Figure 7.
Additionally, we plot the 2D landmark meta-features in Figure
[Table]