π BO: A UGMENTING A CQUISITION F UNCTIONS WITH U SER B ELIEFS FOR B AYESIAN O PTIMIZATION
A BSTRACT
Bayesian optimization (BO) has become an established framework and popular tool for hyperparameter optimization (HPO) of machine learning (ML) algorithms.
While known for its sample-efﬁciency, vanilla BO can not utilize readily available prior beliefs the practitioner has on the potential location of the optimum.
Thus, BO disregards a valuable source of information, reducing its appeal to ML practitioners.
To address this issue, we propose π BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
In contrast to previous approaches, π BO is conceptually simple and can easily be integrated with existing libraries and many acquisition functions.
We provide regret bounds when π BO is applied to the common Expected Improvement acquisition function and prove convergence at regular rates independently of the prior.
Further, our experiments show that π BO outperforms competing approaches across a wide suite of benchmarks and prior characteristics.
We also demonstrate that π BO improves on the state-of-theart performance for a popular deep learning task, with a 12 .
5 × time-to-accuracy speedup over prominent BO approaches.
I NTRODUCTION
The optimization of expensive black-box functions is a prominent task, arising across a wide range of applications.
Bayesian optimization (BO) is a sample-efﬁcient approach to cope with this task, and has been successfully applied to various problem settings, including hyperparameter et al NAS) Ru et al
NAS
HPO (Zimmer et al al
Despite the demonstrated effectiveness of BO for HPO (Bergstra et al., 2011; Turner et al., 2021), its adoption among practitioners remains limited.
In a survey covering NeurIPS 2019 and ICLR 2020 (Bouthillier & Varoquaux, 2020), manual search was shown to be the most prevalent tuning method, with BO accounting for less than 7% of all tuning efforts.
As the understanding of hyperparameter settings in deep learning (DL) models increase (Smith, 2018), so too does the tuning proﬁciency of practitioners (Anand et al., 2020).
As previously displayed (Smith, 2018;
Anand et al., 2020; Souza et al., 2021; Wang et al., 2019), this knowledge manifests in choosing single conﬁgurations or regions of hyperparameters that presumably yield good results, demonstrating a belief over the location of the optimum.
BO’s deﬁcit to properly incorporate said beliefs is a reason why practitioners prefer manual search to BO (Wang et al., 2019), despite its documented shortcomings (Bergstra & Bengio, 2012).
To improve the usefulness of automated HPO approaches for ML practictioners, the ability to incorporate such knowledge is pivotal.
Well-established BO frameworks (Snoek et al., 2012; Hutter et al., 2011; The GPyOpt authors, 2016;
Kandasamy et al., 2020; Balandat et al., 2020) support user input to a limited extent, such as by biasing the initial design, or by narrowing the search space; however, this type of hard prior can lead to poor performance by missing important regions.
BO also supports a prior over functions p ( f ) via the Gaussian Process kernel.
However, this option for injecting knowledge is not aligned with the knowledge that experts possess: they often know which ranges of hyperparameter values tend to work best (Perrone et al., 2019; Smith, 2018; Wang et al., 2019), and are able to specify a probability distribution to quantify these priors.
For example, many users of the Adam optimizer (Kingma & Ba, 2015) know that its best learning rate is often in the vicinity of 1 × 10 − 3 .
In practice, DL experiments are typically conducted in a low-budget setting of less than 50 full model trainings (Bouthillier &
Varoquaux, 2020).
As such, practitioners want to exploit their knowledge efﬁciently without wasting early model trainings on conﬁgurations they expect to likely perform poorly.
Unfortunately, this suits standard BO poorly, as BO requires a moderate number of function evaluations to learn about the response surface and make informed decisions that outperform random search.
While there is a demand to increase knowledge injection possibilities to further the adoption of BO, the concept of encoding prior beliefs over the location of an optimum is still rather novel: while there are some initial works (Ramachandran et al., 2020;
Li et al., 2020; Souza et al., 2021), no approach exists so far that allows the integration of arbitrary priors and offers ﬂexibility in the choice of acquisition function; theory is also lacking.
We close this gap by introducing a novel, remarkably simple, approach for injecting arbitrary prior beliefs into BO that is easy to implement, agnostic to the surrogate model used and converges at standard BO rates for any choice of prior.
Our contributions After discussing our problem setting, related work, and background (Section 2), we make the following contributions:
We introduce π BO, a novel generalization of myopic acquisition functions that accounts for user-speciﬁed prior distributions over possible optima, is demonstrably simple-to-implement, and can be easily combined with arbitrary surrogate models (Section 3.1 & 3.2); 2.
We formally prove that π BO inherits the theoretical properties of the well-established Expected Improvement acquisition function (Section 3.3); 3.
We demonstrate on a broad range of established benchmarks and in DL case studies that π BO can yield 12 .
5 × time-to-accuracy speedup over vanilla BO (Section 4).
B ACKGROUND AND R ELATED W ORK 2. B LACK - BOX O PTIMIZATION
We consider the problem of optimizing a black-box function f across a set of feasible inputs X ⊂ R d :
[Equation]
We assume that f ( x ) is expensive to evaluate, and can potentially only be observed through a noisy estimate, y .
In this setting, we wish to minimize f in an efﬁcient manner, typically adhering to a budget which sets a cap on the number of points that can be evaluated.
Black-Box Optimization with Probabilistic User Beliefs
In our work, we consider an augmented version of the optimization problem in Eq.
(1), where we have access to user beliefs in the form of a probability distribution on the location of the optimum.
Formally, we deﬁne the problem of black-box optimization with probabilistic user beliefs as solving Eq.
(1), given a user-speciﬁed prior probability on the location of the optimum deﬁned as
[Equation]
where regions that the user expects to likely to contain an optimum will have a high value.
We note that, without loss of generality, we require π to be strictly positive on all of X , i.e., any point in the search space might be an optimum.
Since the user belief π ( x ) can be inaccurate or even misleading, optimizing Eq.
(1) given (2) is a challenging problem.
2. B AYESIAN O PTIMIZATION
We outline Bayesian optimization (Mockus et al., 1978; Brochu et al., 2010; Shahriari et al., 2016b).
Model BO aims to globally minimize f by an initial experimental design D
= { ( x i , y i ) }
Mi =1 and thereafter sequentially deciding on new points x n to form the data D n = D n − 1 ∪ { ( x n , y n ) } for the n -th iteration with n ∈ { 1 . . . N } .
After each new observation, BO constructs a probabilistic surrogate model of f and uses that surrogate to evaluate an acquisition function α ( x , D n ) .
The combination of surrogate model and acquisition function encodes the policy for selecting the next point x n +1 .
When constructing the surrogate, the most common choice is Gaussian processes (Rasmussen &
Williams, 2006), which model f as p ( f |D n ) = GP ( m, k ) , with prior mean m (which is typically 0 ) and positive semi-deﬁnite covariance kernel k .
The posterior mean m n and the variance s 2 n are
[Equation]
where ( K n ) ij = k
( x i , x j ) , k n ( x ) =
[ k ( x , x 1 ) , . .
k ( x , x n )] (cid:62) and σ 2 n is the estimation of the observation noise variance σ 2 .
Alternative surrogate models include Random forests (Hutter et al., 2011) and Bayesian neural networks (Springenberg et al., 2016).
Acquisition Functions To obtain new candidates to evaluate, BO employs a criterion, called an acquisition function, that encapsulates an explore-exploit trade-off.
By maximizing this criterion at each iteration, one or more candidate point are obtained and added to observed data.
Several acquisition functions are used in BO; the most common of these is Expected Improvement (EI) (Jones et al., 1998).
For a noiseless function, EI selects the next point x n +1 , where f ∗ n is the minimal objective function value observed by iteration n , as
[Equation]
where Z = ( f ∗ n − m n ( x ))
/s n ( x ) .
Thus, EI provides a myopic strategy for determining promising points; it also comes with convergence guarantees (Bull, 2011).
Similar myopic acquisition functions are Upper Conﬁdence Bound (UCB) (Srinivas et al., 2012), Probability of Improvement (PI) (Jones, 2001; Kushner, 1964) and Thompson Sampling (TS) (Thompson, 1933).
A different class of acquisition functions is based on non-myopic criteria, such as Entropy Search (Hennig &
Schuler, 2012), Predictive Entropy Search (Hernández-Lobato et al., 2014) and Max-value Entropy
Search (Wang & Jegelka, 2017), which select points to minimize the uncertainty about the optimum, and the Knowledge Gradient (Frazier et al., 2008), which aims to minimize the posterior mean of the surrogate at the subsequent iteration.
Our work applies to all acquisition functions in the ﬁrst class, and we leave its extension to those in the second class for future work.
2. R ELATED W ORK
There are two main categories of approaches that exploit prior knowledge in BO: approaches that use records of previous experiments, and approaches that incorporate assumptions on the black-box function provided either directly or indirectly by the user.
As π BO exploits prior knowledge from users, we brieﬂy discuss approaches which utilize previous experiments, and then comprehensively discuss the literature on exploiting expert knowledge.
Learning from Previous Experiments Transfer learning for BO aims to automatically extract and use knowledge from prior executions of BO.
These executions can come, for example, from learning and optimizing the hyperparameters of a machine learning algorithm on different datasets (van Rijn & Hutter, 2018;
Swersky et al., 2013; Wistuba et al., 2015; Perrone et al., 2019; Feurer et al., 2015; 2018), or from optimizing the hyperparameters at different development stages (Stoll et al., 2020).
For a comprehensive overview of meta learning for hyperparameter optimization, please see the survey from Vanschoren (2018).
In contrast to these transfer learning approaches, π BO and the related work discussed below does not hinge on the existence of previous experiments, and can therefore be applied more generally.
Incorporating Expert Priors over Function Structure BO can leverage structural priors on how the objective function is expected to behave.
Traditionally, this is done via the surrogate model’s prior over functions, e.g., the kernel of the GP.
However, there are lines of work that explore additional structural priors for BO to leverage.
For instance, both SMAC (Hutter et al., 2011) and iRace (LópezIbáñez et al., 2016) support structural priors in the form of log-transformations, Li et al.
(2018) propose to use knowledge about the monotonicity of the objective function as a prior for BO, and
Snoek et al.
(2014) model non-stationary covariance between inputs by warping said inputs.
Oh et al.
and Siivola et al.
both propose structural priors tailored to high-dimensional problems, addressing the issue of over-exploring the boundary described by Swersky (2017).
Oh et al.
(2018) propose a cylindrical kernel that expands the center of the search space and shrinks the edges, while Siivola et al.
(2018) propose adding derivative signs to the edges of the search space to steer BO towards the center.
Lastly, Shahriari et al.
(2016a) propose a BO algorithm for unbounded search spaces which uses a regularizer to penalize points based on their distance to the center of the user-deﬁned search space.
All of these approaches incorporate prior information on speciﬁc properties of the function or search space, and are thus not always applicable.
Moreover, they do not generally direct the search to desired regions of the search space, offering the user little control over the selection of points to evaluate.
Incorporating Expert Priors over Function Optimum Few previous works have proposed to inject explicit prior distributions over the location of an optimum into BO.
In these cases, users explicitly deﬁne a prior that encodes their beliefs on where the optimum is more likely to be located.
Bergstra et al.
(2011) suggest an approach that supports prior beliefs from a ﬁxed set of distributions.
However, this approach cannot be combined with standard acquisition functions.
BOPrO (Souza et al., 2021) employs a similar structure that combines the user-provided prior distribution with a data-driven model into a pseudo-posterior.
From the pseudo-posterior, conﬁgurations are selected using the EI acquisition function, using the formulation in Bergstra et al.
While BOPrO is able to recover from misleading priors, its design restricts it to only use EI.
Moreover, it does not provide the convergence guarantees of π BO.
Li et al.
(2020) propose to infer a posterior conditioned on both the observed data and the user prior through repeated Thompson sampling and maximization under the prior.
This method displays robustness against misleading priors but lacks in empirical performance.
Additionally, it is restricted to only one speciﬁc acquisition function.
Ramachandran et al.
(2020) use the probability integral transform to warp the search space, stretching high-probability regions and shrinking others.
While the approach is model- and acquisition function agnostic, it requires invertible priors, and does not empirically display the ability to recover from misleading priors.
In Section 4, we demonstrate that π BO compares favorably for priors over the function optimum, and shows improved empirical performance.
Additionally, we do a complete comparison of all approaches in Appendix C.
In summary, π BO sets itself apart from the methods above by being simpler (and thus easier to implement in different frameworks), ﬂexible with regard to different acquisition functions and different surrogate models, the availability of theoretical guarantees, and, as we demonstrate in
Section 4, better empirical results.
M ETHODOLOGY
We now present π BO, which allows users to specify their belief about the location of the optimum through any probability distribution.
A conceptually simple approach, π BO can be easily implemented in existing BO frameworks and can be combined directly with the myopic acquisition functions listed above.
π BO augments an acquisition function to emphasize promising regions under the prior, ensuring such regions are to be explored frequently.
As optimization progresses, the π BO strategy increasingly resembles that of vanilla BO, retaining its standard convergence rates (see Section 3.3).
π BO is publicly available as part of the SMAC ( https://github.com/automl/SMAC3 ) and
HyperMapper ( https://github.com/luinardi/hypermapper ) HPO frameworks.
3. P RIOR - WEIGHTED A CQUISITION F UNCTION
In π BO, we consider π ( x ) in Eq.
(2) to be a weighting scheme on points in X .
The heuristic provided by an acquisition function α ( x , D n ) , such as EI in Eq.
(2.2), can then be combined with said weighting scheme to form a prior-weighted version of the acquisition function.
The resulting strategy then becomes:
[Equation]
This emphasizes good points under π ( x ) throughout the optimization.
While this property is suitable for well-located priors π , it risks incurring a substantial slowdown for poorly-chosen priors; we will now show how to counter this by decaying the prior over time.
3. D ECAYING P RIOR - WEIGHTED A CQUISITION F UNCTION
As the optimization progresses, we should increasingly trust the surrogate model over the prior; the model improves with data while the user prior remains ﬁxed.
This cannot be achieved with the formulation in Eq.
(5), as poorly-chosen priors would permanently slow down the optimization.
Rather, to accomplish this desired behaviour, the inﬂuence of the prior needs to decay over time.
Building on the approaches of Lee et al.
(2020) and Souza et al.
(2021), we accomplish this by raising the prior to a power γ n ∈ R + , which decays towards zero with growing n .
Thus, the resulting prior π n ( x ) =
π ( x ) γ n reﬂects a belief on the location of an optimum that gets weaker with time, converging towards a uniform distribution.
We set γ n = β/n , where β ∈ R
+ is a hyperparameter set by the user, reﬂecting their conﬁdence in π ( x ) .
We provide a sensitivity study on β in Appendix A.
For a given acquisition function α ( x , D n ) and user-speciﬁed prior π ( x ) , we deﬁne the decaying prior-weighted acquisition function at iteration n as
[Equation]
(6) and its accompanying strategy as the maximizer of α π,n .
With the acquisition function in Eq.
(6), the prior will assume large importance initially, promoting the selection of points close to the prior mode.
With time, the exponent on the prior will tend to zero, making the prior tend to uniform.
Thus, with increasing n , the point selection of α π,n becomes increasingly similar to that of α .
Algorithmdisplays the simplicity of the new strategy, highlighting the required one-line change (Line 6) in the main BO loop.
In Line 3, the mode of the prior is used as a ﬁrst initial sample if available.
Otherwise, only sampling is used for initialization.
To illustrate the behaviour of π BO, we consider a toy problem with Gaussian priors on three different locations of the 1D space (center, left and right) as displayed in Figure 1.
We deﬁne a 1D-Log-Branin toy problem by setting the second dimension of the 2D Branin function to the global optimum x 2 = 2 . 275 and optimizing for the ﬁrst dimension.
Initially (iteration 4 in the top row), π BO ampliﬁes the acquisition function α in high-probability regions, putting a lot of trust in the prior.
As the prior decays (iteration 6 and 8 in the middle and bottom rows, respectively), the inﬂuence of the prior on the point selection decreases.
By later iterations, π BO has searched substantially around the prior mode, and moves gradually towards other parts of the search space.
This is of particular importance for the scenarios in the right column, where π BO recovers from a misleading prior.
In
Appendix B, we show that π BO is applicable to different surrogate models and acquisition functions.
3. T HEORETICAL A NALYSIS
We now study the π BO method from a theoretical standpoint when paired with the EI acquisition function.
For the full proof, we refer the reader to Appendix E. To provide convergence rates, we rely on the set of assumptions introduced by Bull (2011).
These assumptions are satisﬁed for popular kernels like the Matérn (1960) class and the Gaussian kernel, which is obtained in the limit ν → ∞ , where the rate ν controls the smoothness of functions from the GP prior.
Our theoretical results apply when both length scales (cid:96) and the global scale of variation σ are ﬁxed; these results can then be extended to the case where the kernel hyperparameters are learned using Maximum Likelihood
Estimation (MLE) following the same procedure as in Bull (2011) (Theorem 5).
We deﬁne the loss over the ball B R for a function f of norm ||
f || H (cid:96) ( X ) ≤ R in the reproducing kernel Hilbert space (RKHS)
H (cid:96) ( X ) given a symmetric positive-deﬁnite kernel K (cid:96) as
[Equation]
where n is the optimization iteration and u a strategy.
We focus on the strategy that maximizes EI π , the prior-weighted EI, and show that the loss in Equation (7) can, at any iteration n , be bounded by the vanilla EI loss function.
We refer to EI π,n and EI n when we want to emphasize the iteration n for the acquisition functions EI π and EI, respectively.
Theorem 1.
Given D n , K (cid:96) , π , β , σ , (cid:96) , R and the compact set
X ⊂ R d as deﬁned above, the loss
L n incurred at iteration n by EI π,n can be bounded from above as
[Equation]
Using Theorem 1, we obtain the convergence rate of EI π .
This trivially follows when considering the fraction of the losses in the limit and inserting the original convergence rate on EI as in Bull (2011):
Corollary 1.
The loss of a decaying prior-weighted Expected Improvement strategy, EI π , is asymptotically equal to the loss of an Expected Improvement strategy, EI:
[Equation]
so we obtain a convergence rate for EI π of L n ( EI π,n , D n , H (cid:96) ( X ) , R )
=
O ( n − ( ν ∧ 1) /d
(log n ) γ ) .
Thus, we determine that the weighting introduced by EI π does not negatively impact the worst-case convergence rate.
The short-term performance is controlled by the user in their choice of π ( x ) and β .
This result is coherent with intuition, as a weaker prior or quicker decay will yield a short-term performance closer to that of EI.
In contrast, a stronger prior or slower decay does not guarantee the same short-term performance, but can produce better empirical results, as shown in Section 4.
R ESULTS
We empirically demonstrate the efﬁciency of π BO in three different settings.
As π BO is a general method to augment acquisition functions, it can be implemented in different parent BO packages, and the implementation in any given package inherits the pros and cons of that package.
To minimize confounding factors concerning this choice of parent package, we keep comparisons within the methods in one package where possible and provide results in the other packages in Appendix C.
In
Sec. 4.2, using Spearmint as a parent package, we evaluate π BO against three intuitive baselines to assess its performance and robustness on priors with different qualities, ranging from very accurate to purposefully detrimental.
To this end, we use toy functions and cheap surrogates, where priors of known quality can be obtained.
Next, in Sec. 4.3, we compare π BO against two competitive approaches (BOPrO and BOWS) that integrate priors over the optimum similarly to π BO, using
HyperMapper (Nardi et al., 2019) as a parent framework, in which the most competitive baseline
BOPrO is implemented.
For these experiments we adopt a Multilayer Perceptron (MLP) benchmark on various datasets, using the interface provided by HPOBench (Eggensperger et al., 2021), with priors constructed around the defaults provided by the library.
Lastly, in Sec. 4.4, we apply π BO and other approaches to two deep learning tasks, also using priors derived from publicly available defaults.
Further, we demonstrate the ﬂexibility of π BO in Appendix B, where we evaluate π BO in
SMAC (Hutter et al., 2011; Lindauer et al., 2021) with random forests, as another framework with another surrogate model, and adapt it to use the UCB, TS and PI acquisition functions instead of EI.
4. E XPERIMENTAL S ETUP
Priors For our surrogate and toy function tasks, we follow the prior construction methodology in BOPrO (Souza et al., 2021) and create three main types of prior qualities, all Gaussian: strong, weak and wrong.
The strong and weak priors are located to have a high and moderate density on the optimum, respectively.
The wrong prior is a narrow distribution located in the worst region of the search space.
For the OpenML MLP tuning benchmark, we utilize the defaults and search spaces provided in HPOBench (Eggensperger et al., 2021), and construct Gaussian priors for each hyperparameter with their mean on the default value, and a standard deviation of 25% of the hyperparameter’s domain.
For the DL case studies, we utilize defaults from each task’s repository and, for numerical hyperparameters, once again set the standard deviation to 25% of the hyperparameter’s domain.
For categorical hyperparameters, we place a higher probability on the default.
As such, the quality of the prior is ultimately unknown, but serves as a proxy for what a practitioner may choose and has shown to be a reasonable choice (Anastacio & Hoos, 2020).
For all experiments, we run π BO with β = N/ 10 , where N is the total number of iterations, in order to make the prior inﬂuence approximately equal in all experiments, regardless of the number of allowed BO iterations.
We investigate the sensitivity to β in Appendix A, and the sensitivity to prior quality in Appendix G.
Baselines We empirically evaluate π BO against the most competitive approaches for priors over the optimum described in Section 2.3: BOPrO (Souza et al., 2021) and BO in Warped Space (BOWS)
(Ramachandran et al., 2020).
To contextualize the performance of π BO, we provide additional, simpler baselines: random sampling, sampling from the prior and BO with prior-based initial design.
The latter is initialized with the mode of the prior in addition to its regular initial design.
In our main results, we choose Spearmint (with EI) (Snoek et al., 2012) for this mode-initialized baseline, simply referring to it as Spearmint.
See Appendix F for complete details on the experiments.
4. R OBUSTNESS OF π BO
First, we study the robustness of π BO.
To this end, we show that π BO beneﬁts from informative priors and can recover from wrong priors, being consistent with our theoretical results in Section 3.3.
To this end, we consider a well-known black-box optimization function, Branin (2D), as well as two surrogate HPO tasks from the Profet suite (Klein et al., 2019): FC-Net (6D) and XGBoost (8D).
For these tasks, we exemplarily show results for π BO implemented in the Spearmint framework.
As
Figure 2 shows, π BO is able to quickly improve over sampling from the prior.
Moreover, it improves substantially over Spearmint (with mode initialization) for all informative priors, staying up to an order of magnitude ahead throughout the optimization for both strong and weak priors.
For wrong priors, π BO displays desired robustness by recovering to approximately equal regret as Spearmint.
In contrast, Spearmint frequently fails to substantially improve from its initial design on the strong and weak prior, which demonstrates the importance of considering the prior throughout the optimization procedure.
This effect is even more pronounced on the higher-dimensional tasks FCNet and XGBoost, where BO typically spends many iterations at the boundary (Swersky, 2017).
Here, π BO rapidly improves multiple orders of magnitude over the initial design, displaying its ability to efﬁciently exploit the information provided by the prior.
4. C OMPARISON OF π BO AGAINST OTHER P RIOR -G UIDED A PPROACHES
Next, we study the performance of π BO against other state-of-the-art prior-guided approaches.
To this end, we consider optimizing 5 hyperparameters of an MLP for classiﬁcation (Eggensperger et al., 2021) on 6 different OpenML datasets (Vanschoren et al., 2014) and compare against BOPrO (Souza et al., 2021) and BOWS (Ramachandran et al., 2020).
For minimizing confounding factors, we implement π BO and BOWS in HyperMapper (Nardi et al., 2019), the same framework that BOPrO runs on.
Moreover, we let all approaches share π BO’s initialization procedure.
We consider a budget of 50 iterations as it is common with ML practitioners (Bouthillier & Varoquaux, 2020).
In Figure 3, we see that π BO offers the best performance on four out of six tasks, and displays the most consistent performance across tasks.
In contrast to them BOWS and BOPrO, π BO also comes with theoretical guarantees and is ﬂexible in the choice of framework and acquisition function.
4. C ASE S TUDIES ON D EEP L EARNING P IPELINES
Last, we study the impact of π BO on deep learning applications, which are often fairly expensive, making efﬁciency even more important than in HPO for traditional machine learning.
To this end, we consider two deep learning case studies: segmentation of neuronal processes in electron microscopy images with a U-Net(6D)
(Ronneberger et al., 2015), with code provided from the NVIDIA deep learning examples repository (Przemek et al.), and image classiﬁcation on ImageNette-128 (6D) (Howard, 2019), a light-weight adaptation of ImageNet (Deng et al., 2009), with code from the repository of the popular FastAI library (Howard et al., 2018).
We mimic the setup from Section 4.3 by using the HyperMapper framework and identical initialization procedures across approaches.
Gaussian priors are set on publicly available default values, which are results of previous tuning efforts of the original authors.
We again optimize for a practical budget of 50 iterations (Bouthillier & Varoquaux, 2020).
As test splits for both tasks were not available to us, we report validation scores.
As shown in Figure 4, π BO achieves a 2 .
5 × time-to-accuracy speedup over Vanilla BO.
For ImageNette, the performance of π BO at iteration 4 already surpasses the performance of Vanilla BO at
Iteration 50, demonstrating a 12 .
5 × time-to-accuracy speedup.
Ultimately, π BO’s ﬁnal performance establishes a new state-of-the-art validation performance on ImageNette with the provided pipeline, with a ﬁnal accuracy of 94.14% (vs. the previous state of the art with 93.55% 1 ).
C ONCLUSION AND F UTURE W ORK
We presented π BO, a conceptually very simple Bayesian optimization approach for leveraging user beliefs about the location of an optimum, which relies on a generalization of myopic acquisition functions.
π BO modiﬁes the selection of design points through a decaying weighting scheme, promoting high-probability regions under the prior.
Contrary to previous approaches, π BO imposes only minor restrictions on the type of priors, surrogates or frameworks that can be used.
π BO provably converges at regular rates, displays state-of-the art performance across tasks, and effectively recovers from poorly speciﬁed priors.
Moreover, we have demonstrated that π BO can yield substantial performance gains for practical low-budget settings, improving on the state-of-the-art for a real-world
CNN tuning tasks even with trivial choices for the prior.
For practitioners who have historically relied on manual or grid search for HPO, we hope that π BO will serve as an intuitive and effective tool for bridging the gap between traditional tuning methods and BO.
π BO sets the stage for several follow-up studies.
Amongst others, we will examine the extension of π BO to non-myopic acquisition functions, such as entropy-based methods.
Non-myopic acquisition functions do not ﬁt well in the current π BO framework, as they do not necessarily beneﬁt from evaluating inputs expected to perform well.
We will also combine π BO with multi-ﬁdelity optimization methods to yield even higher speedups, and with multi-objective optimization to jointly optimize performance and secondary objective functions, such as interpretability or fairness of models.
E THICS S TATEMENT
Our work proposes an acquisition function generalization which incorporates prior beliefs about the location of the optimum into optimization.
The approach is foundational and thus will not bring direct societal or ethical consequences.
However, π BO will likely be used in the development of applications for a wide range of areas and thus indirectly contribute to their impacts on society.
In particular, we envision that π BO will impact a multitude of ﬁelds by allowing ML experts to inject their knowledge about the location of the optimum into Bayesian Optimization.
We also note that we intend for π BO to be a tool that allows users to assist Bayesian Optimization by providing reasonable prior knowledge and beliefs.
This process induces user bias into the optimization, as π BO will inevitably start by optimizing around this prior.
As some users may only be interested in optimizing in the direct neighborhood of their prior, π BO could allow them to do so if provided with a high β value in relation to the number of iterations.
Thus, if improperly speciﬁed, π BO could serve to reinforce user’s beliefs by providing improved solutions only for the user’s region of interest.
However, if used properly, π BO will reduce the computational resources required to ﬁnd strong hyperparameter settings, contributing to the sustainability of machine learning.
R EPRODUCIBILITY
In order to make the experiments run in π BO as reproducible as possible, we have included links to repositories of our implementations in both Spearmint and HyperMapper, with instructions on how to run our experiments.
Moreover, we have included in said repositories all of the exact priors that we have used for our runs, which run out of the box.
The priors we used were, in our opinion, well motivated as to avoid subjectivity, which we hope serves as a good frame of reference for similar works in the future.
Speciﬁcally, Appendix 4.4 describes how we ran our DL experiments, Appendix F.1 goes into the implementation in further detail, and Appendix D displays the exact priors for all our experiments and prior strengths.
Our Spearmint implementation of both π BO and BOWS is available at https://github.com/piboauthors/PiBO-Spearmint , and our HyperMapper implementation is available at https://github.com/piboauthors/
PiBO-Hypermapper .
For our results on the convergence of π BO, we have provided a complete proof in Appendix E.  A CKNOWLEDGEMENTS
Luigi Nardi was supported in part by afﬁliate members and other supporters of the Stanford DAWN project — Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware.
Carl Hvarfner and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Artur Souza was supported by CAPES, CNPq, and FAPEMIG. Frank Hutter acknowledges support by the European Research Council (ERC) under the European Union Horizon 2020 research and innovation programme through grant no. 716721, through TAILOR, a project funded by the EU Horizonresearch and innovation programme under GA No 952215, by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828 and by the state of BadenWürttemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG.
Marius Lindauer acknowledges support by the European Research Council (ERC) under the Europe Horizon programme.
The computations were also enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC partially funded by the
Swedish Research Council through grant agreement no. 2018-05973.
A B ETA A BLATION S TUDY
We consider the effect of the β hyperparameter of π BO introduced in Section 3.2, controlling the speed of the prior decay.
To show the effect of this hyperparameter, we display the performance of π BO for the toy and surrogate-based benchmarks across all prior qualities.
We emphasize the trade-off between high-end performance on good priors and robustness to bad priors.
In general, a higher value of β yields better performance for good priors, but makes π BO slower to recover from bad priors.
This behaviour follows intuition and the results provided in Section 3.3.
In Figure 5, we display how π BO performs for different choices of β , and once again provide sampling from the prior and Spearmint as baselines.
Following the prior decay parameter baseline by (Souza et al., 2021), we show that the choice of β = 10 onsistently gives one of the best performances for strong priors, while retaining good overall robustness.
Nearly all choices of β give a ﬁnal performance better than that of Spearmint for good priors.
Additionally, there is a clear relationship between ﬁnal performance and β on all good priors.
This is best visualized in the weak XGBoost experiment, where the ﬁnal performances are distinctly sorted by increasing β .
Similar patterns are not as apparent in the ﬁnal performance on wrong priors.
This behaviour highlights the beneﬁts of slowly decaying the prior.
Overall, π BO is competitive for a wide range of β , but suffers slightly worse ﬁnal performance on good priors for low values of β .
[h]
B π BO V ERSATILITY
We show the versatility of π BO by implementing it in numerous variants of SMAC Hutter et al.
(2011), a well-established HPO framework which supports both GP and RF surrogates, and a majority of the myopic acquisition functions mentioned in Section 2.
We showcase the performance of π BO-EI, π BO-PI, π BO-UCB and π BO-TS on the general formulation of π BO with a GP surrogate, as well as π BO-EI with an RF surrogate, which requires a minor adaptation.
B. G ENERAL FORMULATION OF π BO
To allow for the universality of π BO across several acquisition function, we must consider the various magnitudes of acquisition functions.
As UCB and TS typically output values in the same order of magnitude and sign as the objective function, we do not want the behaviour of π BO to be affected by such variations.
The solution to the problem referenced above is to add a simple afﬁne transformation to the observations, { y i } ni =1 , by subtracting by the incumbent, y ∗ n .
As such, we consider at each time step not the original dataset, D n = { ( x i , y i ) } ni =1 , but the augmented dataset ˆ D n = { ( x i , y i − y ∗ n ) } ni =1 .
With this formulation, we get the desired scale- and sign-invariance in the UCB and TS acquisition functions, without changing the original strategy of any of the acquisition function.
Notably, this change leaves prior-weighted EI and PI unaffected.
ANDOM
We now demonstrate π BO with a RF surrogate model.
In the SMAC implementation of the RF surrogate, the model forms piece-wise constant mean and covariance functions.
Naturally, this leads to the EI, PI or UCB acquisition function surface being piece-wise constant as well.
Consequently, an acquisition function with a RF surrogate will typically have a region of global optima.
The choice of the next design point is then selected uniformly at random among the candidate optima.
We wish to retain this randomness when applying π BO.
As such, we require the prior to be piece-wise constant, too.
To do so, we employ a binning approach, that linearly rounds prior values after applying the decay term.
The granularity of the binning decreases at the same rate as the prior, allowing the piece-wise constant regions of the prior grow in size as optimization progresses.
In Figure 9, we demonstrate the importance of the piece-wise constant acquisition function by showing the point selection when applying a π BO with a continuous prior to an RF surrogate (left) and when applying the binning approach (right).
Notably, the smooth prior on the left repeatedly proposes design points very close to previous points, as the prior forces the selection of points near the boundary of a promising region.
Thus, the surrogate model rarely improves, and the optimization gets stuck at said boundary for multiple iterations at a time.
This is best visualized at iteration 5 and 10, where similar points have been selected for all iterations in the time span.
With the binned prior on the right, the selection of design points occurs randomly within a region, avoiding the static point selection and updating of non-modiﬁed approach.
In Figure 8, we report the performance of π BO with a RF surrogate and the binning approach.
This approach is competitive, as it provides substantial improvement over SMAC, improves over sampling from the prior, and quickly recovers from misleading priors.
Notably, the binning is not required for discrete parameters, as the piece-wise constant property holds by default.
Thus, this adaptation is only necessary for continuous parameters.
C O THER P RIOR - BASED A PPROACHES
We now demonstrate the performance of π BO for ﬁve different functions and HPO Surrogates: Branin,
Hartmann-6, as well as three tasks from the Profet suite - SVM, FCNet and XGBoost.
We compare all frameworks for priors over the optimum - namely BOPrO Souza et al.
(2021), BOWS Ramachandran et al.
(2020), TPE Bergstra et al.
(2011), PS-G Li et al.
The performance of π BO is shown on two different frameworks - Spearmint and Hypermapper - to allow for fair comparison and display cross-framework consistency.
As BOWS is implemented in Spearmint and BOPrO in
Hypermapper, they appear in the plots retaining to their framework.
We display each approach with vanilla Spearmint/Hypermapper, with normal initialization, as an additional baseline.
Moreover, we display the performance of π BO implemented in Spearmint, as well as Mode + Spearmint, on the
MLP tuning tasks.
D P RIOR C ONSTRUCTION
We now present the method by which we construct our priors.
For the synthetic benchmarks, we mimic (Souza et al., 2021) by offsetting a Gaussian distribution from the optima.
For our case studies, we choose a Gaussian prior with zero correlation between dimensions.
This was required in order to have a simple, streamlined approach that was compatible with all frameworks.
We constructed the priors once before conducting the experiments, and kept them ﬁxed throughout.
Synthetic and Surrogate-based HPO Benchmarks For these benchmarks, the approximate optima of all included functions could be obtained in advance, either analytically or empirically through extensive sampling.
Thus, the correctness of the prior is ultimately known in advance.
For a function of dimensionality d with optimum at x ∗ , the strong and weak prior qualities were constructed by using a quality-speciﬁc noise term (cid:15) = { (cid:15) i } di =1 and quality-speciﬁc standard deviation as a fraction of the search space.
For the strong prior π s ( x ) , we use a small standard deviation σ s = 1% and construct the prior as
[Equation]
We construct the weak priors analogously by using a larger standard deviation σ w = 10% .
For our 20 runs of the strong and weak prior, this procedure yielded us 20 unique priors per quality type, with varying offsets from the true optimum.
Additionally, the density on the optimum is substantially larger for the strong prior than the weak prior.
No priors with a mean outside the search space were allowed, such priors were simply replaced.
For Branin, we only considered one of the three Branin optima for this procedure, since not all included frameworks support multi-modal distributions.
For the wrong prior, we construct it similarly to the strong prior, but around the empirical maximum, x ¯ ∗ , of the objective function in the search space.
Since this point was far away from the optimum for all benchmarks, we did not add additional noise.
So, the wrong prior π m is constructed as
[Equation]
which means that the wrong prior is identical across runs for a given benchmark.
E P ROOFS
Here, we provide the complete proofs for the Theorem and Corollary introduced in 3.3.
In addition, we provide insight into the interplay between β , the prior π and the value of the derived bound C π,n .
Theorem 1.
Given D n , K (cid:96) , π , σ , (cid:96) , R and the compact set
X ⊂ R d as deﬁned above, the loss L n incurred at iteration n by EI π,n can be bounded from above as
[Equation]
Proof.
To bound the performance of EI π to that of EI, we primarily need to consider Lemma 7 and
Lemma 8 by Bull Bull (2011).
In Lemma 7, it is stated that for any sequence of points { x i } ni =1 , dimensionality d , kernel length scales (cid:96) , and p ∈ N , the posterior variance s 2 n on x n +1 will, for a large value C , satisfy the following inequality at most p times,
[Equation]
Thus, we can bound the posterior variance by assuming a point in time n p where Eq. 13 has held p times.
We now consider Lemma 8 where, through a number of inequalities, EI is bounded by the actual improvement
I n
[Equation]
where I n = ( f ( x ∗ n ) − f ( x ))
+ , τ ( z ) =
z Φ( z )+ φ ( z ) and s = s n ( x n ; (cid:96) ) .
Since π BO re-weights EI n by π n , these bounds need adjustment to hold for EI π,n .
For the upper bound provided in Lemma 8, we make use of max x ∈X π n ( x ) to bound EI π,n ( x ) for any point x ∈ X :
[Equation]
For the lower bounds, we instead rely on min x ∈X π n ( x ) in a similar manner:
[Equation]
Consequently, EI π can be bounded by the actual improvement as
[Equation]
With these bounds in place [Equation] we consider the setting as in the proof for Theorem 2 in Bull Bull (2011), which proves an upper bound for the EI strategy in the ﬁxed kernel parameters setting.
At an iteration n p , p ≤ n p ≤ 3 p , the posterior variance will be bounded by Cp − ( ν ∧ 1) /d
(log p ) γ .
Furthermore, since
I n ≥ 0 and ||
f || H (cid:96) ( X )) ≤ R , we can bound the total improvement as
[Equation]
leaving us a maximum of p times that I n ≥
Rp − 1 .
Consequently, both the posterior variance s 2 n p and the improvement I n p are bounded at n p .
For a future iteration n, 3 p ≤ n ≤ 3( p + 1) , we use the bounds on EI π , s n p
and
I n p to obtain the bounds on the EI π loss:
[Equation]
where the last inequality is a factor C π,n = (cid:16) max x ∈X π
[Equation]
x ) min x ∈X π ( x ) (cid:17) β/n larger than the bound on
L n ( EI , D n , H (cid:96) ( X ) , R ) .
Corollary 1.
The loss of a decaying prior-weighted Expected Improvement strategy, EI π , is asymptotically equal to the loss of an Expected Improvement strategy, EI:
[Equation]
so we obtain a convergence rate for EI π of L n ( EI π,n , D n , H (cid:96) ( X ) , R )
=
O ( n − ( ν ∧ 1) /d
(log n ) γ ) .
Proof.
We simply compute the fraction of the losses in the limit,
[Equation]
E. S ENSITIVITY ANALYSIS ON C π,n
We now provide additional insight into how C π,n depends on the choices of prior and β made by the user.
To do so, we consider a typical low-budget setting and display values of C π,n at iteration 50.
We consider a one-dimensional search space where with a Gaussian prior located in the center of the search space.
In the plot below, we display how the choice of σ , given as a percentage of the search space, and β , the prior conﬁdence parameter, yield different values of C π,n .
We see that, for approximately half of the space, the upper bound on the loss is at least 80% (bright green or yellow) of the upper bound of EI, and only a small region of very narrow priors (dark blue) give a low guaranteed convergence rate.
F E XPERIMENT D ETAILS F. F RAMEWORKS
Our implementations of π BO require little change in the supporting frameworks, Spearmint and
HyperMapper, and we stay as close to the default settings as possible for each framework.
For both Spearmint and HyperMapper, we consider a Matérn 5/2 Kernel.
For particularly strong priors, rounding errors can cause the prior to be zero in parts of the search space, potentially affecting π BO’s convergence properties.
To avoid these rounding errors and ensure a strictly positive prior, we add a small constant, (cid:15) =
10 − 12 , to the prior throughout the search space for all prior qualities.
For the initial sampling from the prior, we truncate the distribution by disallowing sampled points from outside the search space, instead re-sampling such points.
During optimization, we do not to explicitly truncate the prior, as points outside the search space are never considered during acquisition function maximization.
Thus, the prior is effectively truncated to ﬁt the search space without requiring additional consideration.
To the best of our knowledge, there is no publicly available implementation of BOWS, so we reimplemented it in Spearmint.
For the Spearmint implementation of BOWS, we provide warped versions of each benchmark, obtaining 20 unique warpings per prior quality and benchmark.
We truncate the prior by restricting the warped search space to only include the region which maps back to the original search space through the inverted warping function.
For all other approaches, we use the original, publicly available implementations.
Notably, the available implementation of Hyperopt TPE does not support bounded search spaces under our priors; as a compromise, when asked to evaluate outside the search space we return an empirically obtained maximum on the objective function inside the search space.
We use the search spaces, prior locations and descriptions used by (Souza et al., 2021) for the toy and surrogate HPO problems.
We now provide additional details about the benchmarks and case study tasks used, their associated search spaces and priors, and the resources used to run these studies.
F. B ENCHMARKS AND C ASE S TUDIES
Branin The Branin function is a well-known synthetic benchmark for optimization problems.
The
Branin function has two input dimensions and three global minima.
Hartmann-
The Hartmann-6 function is a well-known synthetic benchmark for optimization problems, which has one global optimum and six dimensions.
SVM A hyperparameter-optimization benchmark in 2D based on Profet (Klein et al., 2019).
This benchmark is generated by a generative meta-model built using a set of SVM classiﬁcation models trained on 16 OpenML tasks.
The benchmark has two input parameters, corresponding to SVM hyperparameters.
FCNet A hyperparameter and architecture optimization benchmark in 6D based on Profet.
The
FC-Net benchmark is generated by a generative meta-model built using a set of feed-forward neural networks trained on the same 16 OpenML tasks as the SVM benchmark.
The benchmark has six input parameters corresponding to network hyperparameters.
XGBoost A hyperparameter-optimization benchmark in 8D based on Profet.
The XGBoost benchmark is generated by a generative meta-model built using a set of XGBoost regression models in
UCI datasets.
The benchmark has eight input parameters, corresponding to XGBoost hyperparameters.
OpenML MLP The OpenML MLP tuning tasks are provided through HPOBenchEggensperger et al.
(2021), and train binary classiﬁers on real-world datasets.
The 5D parameter space consists of four continous parameters and one integer parameter.
U-Net Medical
The U-Net (Ronneberger et al., 2015) is a popular convolutional neural network architecture for image segmentation.
We use the implementation and evaluation setting from the popular NVIDIA deep learning examples repository (Przemek et al.)
to build a case study for optimizing hyperparameters for U-Net.
The NVIDIA repository is aimed towards the segmentation of neuronal processes in electron microscopy images for the 2D EM segmentation challenge dataset
[Table]
the probabilities for pooling, [ Avg, Max ] , were set to [0 . 2 , 0 . 8] , respectively.
[Table] [Table]
-Carreras
[Table]
al
[Table] [Table]
Cardona
[Table]
al., 2010).
We optimize 6 hyperparameters of the U-Net pipeline.
ImageNette ImageNette (Howard, 2019) is a subset of 10 classes of ImageNet (Deng et al., 2009) and is primarily used for algorithm development for the popular FastAI library (Howard et al., 2018).
The FastAI library contains a convolutional neural network pipeline for ImageNette, that is used by all competitors on the ImageNette leaderboard.
We base our case study on the 80 epoch,resolution setting of this leaderboard and optimize 6 of the hyperparameters of the FastAI ImageNette pipeline.
F. S EARCH S PACES AND P RIORS
The search spaces for each benchmark are summarized in Table 1 (Branin and Profet), Table(OpenML MLP), and Table 3 (ImageNette and U-Net).
For the Profet benchmarks, we report the original ranges and whether or not a log scale was used.
However, in practice, Profet’s generative model transforms the range of all hyperparameters to a linear [0 , 1] range.
We use Emukit’s public implementation for these benchmarks (Paleyes et al., 2019).
[Table]
Training details deep learning case studies Both case studies are based on existing deep learning code, whose hyperparameters we vary according to the HPO.
In both case studies, we enabled mixed precision training, and for ImageNette-128 to work in conjunction with Spearmint, we had to enable the MKL_SERVICE_FORCE_INTEL environment ﬂag.
For all further details, we refer to the supplementary material containing our code.
Resources used for deep learning case studies For U-Net Medical we used one GeForce RTX 2080 Ti GPU, whereas for ImageNette-
we used two GeForce RTX 2080 Ti GPU’s.
Also, we used 4 cores and 8 cores respectively, of an AMD EPYC 7502 32-Core Processor.
In Table 4 we list the GPU hours needed for running the deep learning case studies as well as the emitted COequivalents.
[Table]
Assets deep learning case studies
In addition to the assets we list in the main paper, the U-Net
Medical code base we used employs the 2D EM segmentation challenge dataset (Arganda-Carreras et al., 2015; Cardona et al., 2010), which is available for for the purpose of generating or testing non-commercial image segmentation software.
We include licenses of all existing code assets we used in the supplementary material containing our code.
G S ENSITIVITY TO P RIOR S TRENGTH
We investigate the performance of π BO when providing priors over the optimum of various qualities.
To show the effect of decreasing the prior strength, a grid of prior qualities, with varying widths and offsets from the optimum, are provided.
Thus, priors range from the strong prior in the results, to weak, correct priors and sharp, misplaced priors.
From Figures 14- 18, it is shown that π BO provides substantial performance across most prior qualities for all benchmarks but Branin, and recoups its early losses on the worst priors in the bottom left corner.
π BO demonstrates sensitivity to the width of the prior, as the optimization does not progress as quickly for well-located priors with a larger width.
Additionally, π BO’s improvement over the Spearmint + Mode baseline is further emphasized, as this baseline often fails to meaningfully improve over the mode in early iterations.