H YPERPARAMETER T RANSFER A CROSS D EVELOPER A DJUSTMENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used to speedup a new HPO?
This question poses a challenging problem, as developer adjustments can change which hyperparameter settings perform well, or even the hyperparameter search space itself.
While many approaches exist that leverage knowledge obtained on previous tasks , so far, knowledge from previous development steps remains entirely untapped.
In this work, we remedy this situation and propose a new research framework: hyperparameter transfer across adjustments (HT-AA).
To lay a solid foundation for this research framework, we provide four simple HT-AA baseline algorithms and eight benchmarks changing various aspects of ML algorithms, their hyperparameter search spaces, and the neural architectures used.
The best baseline, on average and depending on the budgets for the old and new HPO, reaches a given performance 1.2–3.6x faster than a prominent HPO algorithm without transfer.
As HPO is a crucial step in ML development but requires extensive computational resources, this speedup would lead to faster development cycles, lower costs, and reduced environmental impacts.
To make these beneﬁts available to ML developers off-the-shelf and to facilitate future research on HT-AA, we provide python packages for our baselines and benchmarks.
I NTRODUCTION : A N EW H YPERPARAMETER T RANSFER F RAMEWORK
The machine learning (ML) community arrived at the current generation of ML algorithms by performing many iterative adjustments.
Likely, the way to artiﬁcial general intelligence requires many more adjustments.
Each algorithm adjustment could change which settings of the algorithm’s hyperparameters perform well, or even the hyperparameter search space itself (Chen et al., 2018; Li et al., 2020).
For example, when deep learning developers change the optimizer, the learning rate’s optimal value likely changes, and the new optimizer may also introduce new hyperparameters.
Since ML algorithms are known to be very sensitive to their hyperparameters (Chen et al., 2018; Feurer & Hutter, 2019), developers are faced with the question of how to adjust their hyperparameters after changing their code.
Assuming that the developers have results of one or several hyperparameter optimizations (HPOs) that were performed before the adjustments, they have two options:
Somehow manually transfer knowledge from old HPOs.
This is the option chosen by many researchers and developers, explicitly disclosed, e.g., in the seminal work on AlphaGo (Chen et al., 2018).
However, this is not a satisfying option since manual decision making is time-consuming, often individually designed, and has already lead to reproducibility problems (Musgrave et al., 2020).
Start the new HPO from scratch.
Leaving previous knowledge unutilized can lead to higher computational demands and worse performance (demonstrated empirically in Section 5).
This is especially bad as the energy consumption of ML algorithms is already recognized as an environmental problem.
For example, deep learning pipelines can have CO 2 emissions on the order of magnitude of the emissions of multiple cars for a lifetime (Strubell et al., 2019), and their energy demands are growing furiously:
Schwartz et al.
(2019) cite a “300,000x increase from 2012 to 2018”.
Therefore, reducing the number of evaluated hyperparameter settings should be a general goal of the community.
The main contribution of this work is the introduction of a new research framework: Hyperparameter transfer across adjustments (HT-AA) , which empowers developers with a third
option
Automatically transfer knowledge from previous HPOs.
This option leads to advantages in two aspects: The automation of decision making and the utilization of previous knowledge.
On the one hand, the automation allows to benchmark strategies, replaces expensive manual decision making, and enables reproducible and comparable experiments; on the other hand, utilizing previous knowledge leads to faster development cycles, lower costs, and reduced environmental impacts.
To lay a solid foundation for the new HT-AA framework, our individual contributions are as follows:
• We formally introduce a basic version of the HT-AA problem (Section 2).
• We provide four simple baseline algorithms for our basic HT-AA problem (Section 3).
• We provide a comprehensive set of eight novel benchmarks for our basic HT-AA problem (Section 4).
• We show the advantage of transferring across developer adjustments: some of our simple baseline algorithms outperform HPO from scratch up to 1.2–3.6x on average depending on the budgets (Section 5).
• We empirically demonstrate the need for well-vetted algorithms for HT-AA: two baselines modelled after actually-practiced manual strategies perform horribly on our benchmarks (Section 5).
• We relate the HT-AA framework to existing research efforts and discuss the research opportunities it opens up (Section 6).
• To facilitate future research on HT-AA, we provide open-source code for our experiments and benchmarks and provide a python package with an out-of-the-box usable implementation of our HT-AA algorithms.
H YPERPARAMETER T RANSFER A CROSS A DJUSTMENTS
After presenting a broad introduction to the topic, we now provide a detailed description of hyperparameter transfer across developer adjustments (HT-AA).
We ﬁrst introduce hyperparameter optimization, then discuss the types of developer adjustments, and ﬁnally describe the transfer across these adjustments.
Hyperparameter optimization (HPO)
The HPO formulation we utilize in this work is as follows:
[Equation]
where f A ( x ) is the objective function for ML
algorithm A with hyperparameter setting x , b is the number of available evaluations, and X is the search space.
We allow the search space X to contain categorical and numerical dimensions alike and consider only sequential evaluations.
We refer to a speciﬁc HPO problem with the 3-tuple ( X , f A , b ) .
For a discussion on potential extensions of our framework to different HPO formulations, we refer the reader to Section 6.
Developer adjustments We now put developer adjustments on concrete terms and introduce a taxonomy of developer adjustments.
We consider two main categories of developer adjustments: ones that do not change the search space X (homogeneous adjustments) and ones that do (heterogenous adjustments).
Homogeneous adjustments could either change the algorithm’s implementation or the hardware that the algorithm is run on.
Heterogeneous adjustments can be further categorized into adjustments that add or remove a hyperparameter (hyperparameter adjustments) and adjustments that change the search space for a speciﬁc hyperparameter (range adjustments).
Figure 1 shows an illustration of the adjustment types.
Knowledge transfer across adjustments In general, a continuous stream of developer adjustments could be accompanied by multiple HPOs.
We simplify the problem in this fundamental work and only consider the transfer between two HPO problems; we discuss a potential extension in Section 6.
The two HPO problems arise from adjustments Ψ to a ML algorithm
A old and its search space X old , which lead to A new , X new := Ψ( A old , X old ) .
Speciﬁcally, the hyperparameter transfer across adjustments problem is to solve the HPO problem (
X new , f A new , b new ) , given the results for ( X old ,
f A old , b old ) .
Compared to HPO from scratch, developers can choose a lower budget b new , given evidence for a transfer algorithm achieving the same performance faster.
B ASELINE A LGORITHMS FOR HT-AA
In this section we present four baselines for the speciﬁc instantiation of the hyperparameter transfer across adjustments (HT-AA) framework discussed in Section 2.
We resist the temptation to introduce complex approaches alongside a new research framework and instead focus on a solid foundation.
Speciﬁcally, we focus on approaches that do not use any knowledge from the new HPO for the transfer and include two baselines that are modelled after actually-practiced manual strategies.
We ﬁrst introduce the two basic HPO algorithm that the transfer approaches build upon, then introduce notation for two decompositions of HPO search spaces across adjustments, and ﬁnally, we present the four baselines themselves.
3. P RELIMINARIES
Background TPE For basic hyperparameter optimization and parts of the transfer algorithms, we employ the Tree-Structured Parzen Estimator (TPE) algorithm (Bergstra et al., 2011) (or Bayesian optimization as below), which is the default algorithm in the popular HyperOpt package (Bergstra et al., 2013). TPE uses kernel density estimators to model the densities l ( x ) and g ( x ) , for the probability of a given hyperparameter conﬁguration x being worse ( l ), or better ( g ), than the best already evaluated conﬁguration. To decide which conﬁguration to evaluate, TPE then solves x ∗ ∈ arg max x ∈X g ( x ) / b ( x ) approximately.
In our experiments, we use the TPE implementation and hyperparameter settings from Falkner et al.
Background GP We also employ Bayesian optimization (BO) (Shahriari et al., 2016) with Gaussian processes (Rasmussen & Williams, 2006).
Given observations D
= { ( x k , f A ( x k ) }
nk =1 , BO ﬁts a probabilistic model p ( f A | D ) for the objective function f A .
Then, to decide which hyperparameter setting to evaluate next, BO maximizes an acquisition function α ( x | D ) , which uses this probabilistic model.
Here, we use Gaussian process regression to ﬁt a model, and the widely used expected improvement (Jones et al., 1998) as the acquisition function.
In our experiments, we use the BO implementation from the SMAC3 package (Lindauer et al., 2017) with default kernels, hyperparameters, and optimization procedures.
In the following we will refer to BO with Gaussian processes with GP.
Search space decomposition: Hyperparameter adjustments For hyperparameter adjustments the new search space X new and the old search space X old only differ in hyperparameters, not in hyperparameter ranges, so we can decompose the search spaces as X new =
X only-new × X both and
X old =
X both × X only-old , where X both is the part of the search space that remains unchanged across adjustments (see Figure 2 for reference).
All baselines use this decomposition and project the hyperparameter settings that were evaluated in the old HPO from X old to X both .
Search space decomposition:
Range adjustments A range adjustment can remove values from the hyperparameter range or add values.
For an adjustment of hyperparameter range X H i old to X H
i new this can be expressed as X H
i new =
X H i both ∪ X
[Table]
3. O NLY O PTIMIZE N EW H YPERPARAMETERS
A common strategy by developers to manually transfer across adjustments is to set previous hyperparameters in X both to the best setting of the previous HPO and only optimize hyperparameters in
X only-new (Agostinelli et al., 2014; Huang et al., 2017; Wu & He, 2018).
If the previous best setting is not a valid conﬁguration anymore, i.e., it has values in X H i both,range-only-old for a hyperparameter H i still in X both , this strategy uses the best setting that still is a valid conﬁguration.
In the following, we refer to this strategy as only-optimize-new .
3. D ROP U NIMPORTANT H YPERPARAMETERS
A strategy inspired by manual HT-AA efforts is to only optimize important hyperparameters.
The utilization of importance statistics was, for example, explicitly disclosed in the seminal work on
AlphaGo (Chen et al., 2018).
Here, we determine the importance of each individual hyperparameter with functional analysis of variance (fANOVA) (Hutter et al., 2014) and do not tune hyperparameters with below mean importance.
Therefore, this strategy only optimizes hyperparameters in X only-new and hyperparameters in X both with above mean importance.
In the following, we refer to this strategy as drop-unimportant .
3. F IRST E VALUATE B EST
The best-ﬁrst strategy uses only-optimize-new for the ﬁrst evaluation, and uses standard GP for the remaining evaluations.
This strategy has a large potential speedup and low risk as it falls back to standard GP.
3. T RANSFER GP (TGP) / T RANSFER TPE (T2PE)
This strategy ﬁts a model on the results of the previous HPO projected to X both with X both as the search space (a Gaussian process for TGP and a TPE model for TPE).
To choose the values for hyperparameters in X only-new , TGP/TPE uses a random sample (Figure 2).
Further, as in the basic GP strategy, TGP/T2PE uses a random sample over X new in 1 / 3 of cases, and after dim( X new ) evaluations,
B ENCHMARKS FOR HT-AA
We introduce eight novel benchmarks for the basic hyperparameter transfer across adjustments (HT-AA) problem discussed in Section 2.
As is common in hyperparameter optimization research, we employ tabular and surrogate benchmarks to allow cheap and reproducible benchmarking (Perrone et al., 2018;
Falkner et al., 2018).
Tabular benchmarks achieve this with a lookup table for all possible hyperparameter settings.
In contrast, surrogate benchmarks ﬁt a surrogate model for the objective function (Eggensperger et al., 2014).
We base our surrogate models and lookup tables on code and data from existing hyperparameter optimization (HPO) benchmarks (detailed in Appendix A) that, together, cover four different machine learning algorithms: a fully connected neural network (FCN), neural architecture search for a convolutional neural network (NAS), a support vector machine (SVM), and XGBoost (XGB).
While the NAS and FCN benchmarks are based on lookup tables, the SVM and XGB based benchmarks use surrogate models.
For each of these base benchmarks, we simulate two different types of developer adjustments (Table 1), to arrive at a total of eight benchmarks.
Additionally, for each algorithm and adjustment, we consider multiple tasks in our benchmarks.
Further, we provide a python package with all our benchmarks and refer the reader to
Appendix A for additional details on the benchmarks.
[Table]
Heterogeneous
FCN-B
Add per-layer choice of activation function
Heterogeneous
Change learning rate schedule
Homogeneous NAS-A Add 3x3 average pooling as choice of operation to each edge Heterogeneous
NAS-B
Add node to cell template (adds 3 hyperparameters) Heterogeneous
XGB-A
Expose four booster hyperparameters
Heterogeneous
XGB-B Change four unexposed booster hyperparameter values
Homogeneous
SVM-A Change kernel
Homogeneous
Remove hyperparameter for old kernel
Heterogeneous
Add hyperparameter for new kernel
Heterogeneous
SVM-B Increase range for cost hyperparameter
Heterogeneous
[Table]
E XPERIMENTS AND R ESULTS
In this section [Table] we empirically evaluate the four baseline algorithms presented in Section 3 as solutions for the hyperparameter transfer across adjustments problem.
We ﬁrst describe the evaluation protocol used through all studies and then present the results.
[-O v e r G P 10 Previous Evaluations GP Evaluations for Reference Objective [#] 20 Previous Evaluations   
-]Evaluation protocol We use the benchmarks introduced in Section 4 and focus on the speedup of transfer strategies over either Bayesian optimization with Gaussian processes, which we refer to as
GP, or over Tree Parzen Estimator, which we abbreviate as TPE.
Speciﬁcally, we measured how much faster a transfer algorithm reaches a given objective value compared to GP or TPE in terms of the number of evaluations.
We repeated all measurements across 25 different random seeds and report results for validation objectives, as not all benchmarks provide test objectives, and to reduce noise in our evaluation.
We terminate runs after 400 evaluations and report ratio of means.
To aggregate these ratios across tasks and benchmarks, we use the geometric mean.
We use the geometric mean, as, intuitively, two speedups of e.g., 0.1x and 10x average to 1x, and not 5.05x.
We want to note that the standard mean is an upper bound for the geometric mean, so using the geometric mean in fact makes the speedups slightly less impressive than had we used the standard arithmetic mean.
To determine the target objective values, we measured GP’s average performance for 10, 20, and 40 evaluations.
We chose this range of evaluations as a survey among NeurIPS2019 and ICLR2020 authors indicates that most hyperparameter optimizations (HPOs) do not consider more than 50 evaluations (Bouthillier & Varoquaux, 2020).
Further, for transfer approaches, we perform this experiment for different evaluation budgets for the HPO before the adjustments (also for 10, 20, and 40 evaluations).
The transfer GP (TGP) / transfer TPE (T2PE) and best-ﬁrst strategy lead to large speedups, while drop-unimportant and only-optimize-new perform poorly.
Here, in this paragraph we focus on the GP based evaluation.
On average and depending on the budgets for the old and new HPO, TGP reaches the given objective values 1.1–2.1x faster than GP, and best-ﬁrst 1.1–3.3x faster.
The combination of TGP/T2PE and best-ﬁrst leads to further speedups over best-ﬁrst if the budget for the old HPO was 20 or 40, and when the old budget and new budget both equal 10.
These additional average speedups when combining TGP with best-ﬁrst are between 0.1 and 0.3.
We visualize these results with violin plots (Figure 3), as they take into account the multi modality of the results, and also provide a table with exact averages (Table 2).
There are two main trends visible: (1) The more optimal the target objective, the smaller the speedup, and (2) the higher the budget for the previous HPO, the higher the speedup.
For a more ﬁne-grained visualization that shows violin plots over task averages for each benchmark, we refer to Appendix B.
The approaches inspired by actually-practiced manual strategies, drop-unimportant and only-optimize-new, do not reach the performance of GP in a large percentage of cases, even while given at least 10x the budget compared to GP (Figure 4).
These high failure rates makes an evaluation for the speedup unfeasible and indicates that actually-practiced manual strategies perform worse than starting from scratch.
For the failure rates for GP, TGP, and best-ﬁrst (0-0.8%) we refer the reader to Appendix C.
S p ee dup
O v e r
G P 10 Previous Evaluations 20 Previous Evaluations 40 Previous Evaluations S p ee dup O v e r T P E
GP/TPE Evaluations for
Reference Objective [#][-20 Previous Evaluations   
 Previous Evaluations-]
Best First
[Figure]
GP
[Figure]
Additionally, we provide a study on the improvement in objective value for a ﬁxed number of evaluations in Appendix D; in Appendix E we show the results of a control study that compares GP with different ranges of random seeds; and in Appendix F we compare random search and a popular
HPO algorithm with GP to validate its reliability.
R ELATED W ORK AND R ESEARCH O PPORTUNITIES
In this section, we discuss work related to hyperparameter transfer across adjustments (HT-AA) and present several research opportunities in combining existing ideas with HT-AA.
Transfer learning Transfer learning studies how to use observations from one or multiple source tasks to improve learning on one or multiple target tasks (Zhuang et al., 2019).
If we view the HPO problems before and after speciﬁc developer adjustments as tasks, we can consider HT-AA as a speciﬁc transfer learning problem.
As developer adjustments may change the search space, HT-AA would then be categorized as a heterogeneous transfer learning problem (Day & Khoshgoftaar, 2017).
Transfer learning across adjustments Recently, Berner et al.
(2019) transferred knowledge between deep reinforcement learning agents across developer adjustments.
They crafted techniques
[Figure]
[Figure] F a il e d R un
[Figure]
[ % ] 10 Previous Evaluations
GP Evaluations for
Reference Objective [#] 20 Previous Evaluations
Previous Evaluations
Drop Unimportant
Only Optimize New to preserve, or approximately preserve, the neural network policy each type of adjustment they encountered.
Their transfer strategies are inspired by Net2Net knowledge transfer (Chen et al., 2015), and they use the term surgery to refer to this practice.
Their work indicates that transfer learning across adjustments is not limited to knowledge about hyperparameters, but extends to a more general setting, leaving room for many research opportunities.
Continuous knowledge transfer In this paper, we focus on transferring knowledge from the last
HPO performed, but future work could investigate a continuous transfer of knowledge across many cycles of adjustments and HPOs.
Transferring knowledge from HPO runs on multiple previous versions could lead to further performance gains, as information from each version could be useful for the current HPO.
Such continuous HT-AA would then be related to the ﬁeld of continual learning (Thrun & Mitchell, 1995; Lange et al., 2020).
Hyperparameter transfer across tasks (HT-AT)
There exists an extensive research ﬁeld that studies the transfer across tasks for HPOs (Vanschoren, 2018).
The main difference to hyperparameter transfer across adjustments is that the former assumes an unchanging search space, whereas dealing with such changes is one of the main challenges in HT-AA.
In HT-AT, the search space and the ML algorithm remain unchanged, but the task that the algorithm is applied to changes.
Another difference is that most work on HT-AT considers large amounts of meta-data; up to more than a thousand tasks and function evaluations (Wang et al., 2018; Metz et al., 2020).
Homogeneous hyperparameter transfer across adjustments (homogeneous HT-AA) problems, where none of the adjustments changes the search space, are syntactically equivalent to HT-AT problems.
For this homogeneous HT-AA, existing approaches for HT-AT could, in principle, be applied without modiﬁcation; this includes, for example the transfer acquisition function (Wistuba et al., 2018),
multi-task bayesian optimization (Swersky et al., 2013), multi-task adaptive bayesian linear regression (Perrone et al., 2018), ranking-weighted gaussian process ensemble (Feurer et al., 2018), and difference-modelling bayesian optimisation (Shilton et al., 2017).
Further, an adaptation of across-task strategies to the across-adjustments setting could lead to more powerful HT-AA approaches in the future.
Finally, the combination of across-task and acrossadjustments hyperparameter transfer is an exciting research opportunity that could provide even larger speedups than either transfer strategy on its own.
Advanced hyperparameter optimization HT-AA can be combined with one of the many extensions to the basic hyperparameter optimization (HPO) formulation.
One such extension is multi- ﬁdelity HPO, which allows the use of cheap-to-evaluate approximations to the actual objective (Li et al., 2017; Falkner et al., 2018).
Similarly, cost-aware HPO adds a cost to each hyperparameter setting, so a cost model can prioritize the evaluation of cheap hyperparameter settings over expensive ones (Snoek et al., 2012).
Yet another extension is to take different kinds of evaluation noise into account (Kersting et al., 2007) or to consider not one, but multiple objectives to optimize for (Khan et al., 2002).
All these HPO formulations can be studied in conjunction with HT-AA, to either provide further speedups or deal with more general optimization problems.
Guided machine learning The ﬁeld of guided machine learning (gML) studies the design of interfaces that enables humans to guide ML processes (Westphal et al., 2019).
An HT-AA algorithm could be viewed as a ML algorithm that receives incremental guidance in the form of arbitrary developer adjustments; the interface would then be the programming language(s) the ML algorithm is implemented in.
On a different note, gML could provide HT-AA algorithms with additional information about the adjustments to the ML algorithm.
For example, when adding a hyperparameter, there are two distinctions we can make: Either an existing hyperparameter is exposed (e.g., the dropout rate was previously hardcoded as 0.5, and is now tuned) , or a new component is added to the algorithm that introduces a new hyperparameter (e.g., a new learning rate schedule that introduces a decay hyperparameter) .
From the HPO problem itself, we cannot know which case it is, and neither which ﬁxed value an exposed hyperparameter had.
Guided HT-AA algorithms could ask for user input to ﬁll this knowledge gap.
Alternatively, HT-AA algorithms with code analysis could automatically extract this knowledge from the source code.
Programming by optimization Relatedly, the programming by optimization (PbO) framework (Hoos, 2012) proposes the automatic construction of a search space of algorithms, based on code annotations, and the subsequent automated search in this search space.
While this framework considers evolving search spaces over incremental developer actions, each task and development step restarts the search from scratch.
This is in contrast to our hyperparameter transfer framework that alleviates the need to restart from scratch after each developer adjustment.
C ONCLUSION
In this work we introduced hyperparameter transfer across developer adjustments to improve efﬁciency during the development of ML algorithms.
In light of rising energy demands of ML algorithms and rising global temperatures, more efﬁcient ML development practices are an important issue now and will become more important in the future.
As already two of the simple baseline algorithm considered in this work lead to large empirical speedups, our new framework represents a promising step towards efﬁcient ML development
A B ENCHMARK S UITE D ETAILS A. O VERVIEW
[Table]
A. FCN-A & FCN-B
Base benchmark We use code and data from (Klein & Hutter, 2019).
We use the tasks HPOBench-Protein, HPO-Bench-Slice, HPO-Bench-Naval
[Table]
and HPO-Bench-Parkinson.
Budget For FCN-A the budget is set to 100.
For FCN-B, additional to the changes in the search space (Table 6)
[Table]
the budget is increased from 50 to 100 epochs.
[Table]
[Table] [Table]
[Table] [Table]
{ ReLu
[Table]
tanh }
[Table] [Table]
[Table]
Base benchmark We use code and data from (Dong & Yang, 2019).
We use the tasks CIFAR10,
CIFAR
[Table]
and ImageNet.
[Table] [Table]
Base benchmark We use an open source implementation by the HPOlib authors following Perrone et al.
This implementation uses data from Kühn et al.
and employs a random forest as a surrogate model (Eggensperger et al., 2014).
For our benchmarks, we randomly selected the ten tasks monks-problems-2, tic-tac-toe, kc2, monks-problems-1, qsar-biodeg, nomao, pc3, bank-marketing, ada-agnostic, and hill-valley.
[Table]
[Table]
Base benchmark We use an open source implementation by the HPOlib authors following Perrone et al.
This implementation uses data from Kühn et al.
and employs a random forest as a surrogate model (Eggensperger et al., 2014).
For our benchmarks, we randomly selected the ten tasks monks-problems-2, tic-tac-toe, kc2, monks
[Table]
problems
[Table]
1, qsar
[Table]
biodeg, nomao, pc3, bank-marketing, ada-agnostic, hill-valley
[Table] [Table]
B
D ETAILED S PEEDUPS B. GP B ASED E VALUATION
FCN-A
FCN-B
S p ee dup
O v e r
G P
NAS-A
NAS-B
SVM-A
SVM-B
GP Evaluations for
Reference Objective [#]
[Figure]
[Figure]
FCN-A
FCN-B
S p ee dup
O v e r
G P
NAS-A
NAS-B
SVM-A
SVM-B
GP Evaluations for
Reference Objective [#]
XGB
[Figure] [Figure]
GP FCN-A FCN-B S p ee dup
O v e r
G P
NAS-A
NAS-B
SVM-A
SVM-B
GP Evaluations for
Reference Objective [#]
[Figure]
[Figure]
[Figure]
VALUATION
FCN-A
FCN-B
S p ee dup
O v e r
T P E
NAS-A
NAS-B
SVM
[Figure]
A
[Figure]
B TPE Evaluations for Reference Objective [#] [Figure]
[Figure]
FCN-A
FCN-B
S p ee dup
O v e r
T P E
NAS-A
NAS-B
SVM-A
SVM-B
TPE Evaluations for
{+Reference Objective [#] 
+}
XGB
[Figure]
A XGB [Figure] Transfer TPE
FCN-A
FCN-B
S p ee dup O v e r
T P E
NAS-A NAS-B SVM-A
SVM-B
TPE Evaluations for
Reference Objective [#]
XGB-A
XGB
[Figure]
B
[Figure]
Transfer TPE
[Figure]
GP Evaluations for
Reference Objective [#] 20 Previous
[Figure]
40 Previous Evaluations
[Figure] [Figure]
10 Previous Evaluations TPE Evaluations for
Reference {+Objective [#] 20 Previous Evaluations   
+}
Previous Evaluations
[Figure]
TPE
[Figure]
TPE
F a il e d R un s [ % ] 10 Previous Evaluations
TPE Evaluations for
{+Reference Objective [#] 20 Previous Evaluations   
+}
Previous Evaluations
Drop Unimportant
Only Optimize New
D O BJECTIVE I MPROVEMENTS
Here, we show the standardized improvement over the control algorithm. We compute the standardized mean improvement by averaging the ﬁnal performance across repetitions, then subtract the mean performance of the control algorithm, and divide by the standard deviation of the control algorithm.
This metric is known as glass’ delta and intuitively gives the improvement of an algorithm over a control algorithm in the standard units of this control algorithm.
As some benchmarks had a standard deviation of 0, we added a small constant in those cases.
We chose this constant according to the 0.2-quantile of the observed values.
For the plots we clip the improvement to [ − 10 , ∞ ) , as for some plots there are extreme outliers.
We provide results for two aggregation levels: the distribution over task standardized mean improvements, where we compute the standardized mean improvement over repetitions and show results on a per-benchmark level; and the distribution over benchmark means, i.e., the means across task standardized mean improvements for all tasks in a given benchmark.
D.
T GP, B EST F IRST , AND T WITH B EST F VS .
GP
[Figure] I m p r ov e m e n t O v e r
G P [ S D
G P + ε ]
[Figure]
[Figure]
I m p r
ov e m e n t
O v e r
G P [
[Figure]
D
G P + ε
[Figure]
[Figure]
FCN-A
FCN-B
[Figure]
I m p r
ov e m e n t
O v e r
G P [ S D
G P + ε ]
NAS-A
NAS-B
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
D.
T
[Figure]
I m p r
ov e m e n t
O v e r
T P E [ S D
G P + ε ]
[Figure]
[Figure]
I m p r
ov e m e n t
O v e r
T P E [ S D
G P + ε ]
[Figure]
[Figure]
[Figure]
I m p r
ov e m e n t
O v e r
T P E [ S D
G P + ε ]
[Figure]
[Figure]
FCN-B −−
I m p r
ov e m e n t
O v e r
T P E [ S D
G P + ε ]
NAS-A
NAS-B
[Figure]
[Figure]
SVM-
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
B
[Figure]
I m p r
ov e m e n t
O v e r
G P [ S D
G P + ε ]
[Figure]
[Figure]
I m p r
ov e m e n t
O v e r
G P [[-S D G P + ε ] 10 Previous Evaluations Evaluations [#] 20 Previous Evaluations   
 Previous-]
[Figure] [Figure]
I m p r ov e m e n t O v e r
G P [
[Figure] [Figure]
D. TPE B
[Figure]
I m p r ov e m e n t
O v e r
T P E [ S D
G P + ε ]
[Figure]
[Figure]
I m p r ov e m e n t O v e r T P E [ S D
G P
[Figure]
[Figure] I m p r
ov e m e n t O v e r T P E [ S D G P
[Figure]
[Figure]
I m p r
ov e m e n t O v e r T P E [ S D G
[Figure]
E C ONTROL S TUDY : GP AND TPE FOR D IFFERENT R ANDOM S EED R ANGES
As a sanity check, and to gauge the inﬂuence of random seeds [Figure] we compare GP (and TPE) to itself with different random seed ranges (GP 1 and GP 2 ).
We observe little differences in GP 1 and GP(likewise for TPE) compared to the differences between GP and the transfer approaches (Figureand Figure 31) [Figure]
[Figure]
F C ONTROL S TUDY : R ELIABILITY OF GP AND TPE
As a sanity check for the reliability of GP and TPE, we compare them to random search.
[Figure]
S p ee dup
O v e r
R a ndo m
[Figure]