Joint Entropy Search for Maximally-Informed Bayesian Optimization
Abstract
Information-theoretic Bayesian optimization techniques have become popular for optimizing expensive-to-evaluate black-box functions due to their non-myopic qualities.
Entropy Search and Predictive Entropy Search both consider the entropy over the optimum in the input space, while the recent Max-value Entropy Search considers the entropy over the optimal value in the output space.
We propose Joint Entropy
Search (JES), a novel information-theoretic acquisition function that considers an entirely new quantity, namely the entropy over the joint optimal probability density over both input and output space.
To incorporate this information, we consider the reduction in entropy from conditioning on fantasized optimal input/output pairs.
The resulting approach primarily relies on standard GP machinery and removes complex approximations typically associated with information-theoretic methods.
With minimal computational overhead, JES shows superior decision-making, and yields state-of-the-art performance for information-theoretic approaches across a wide suite of tasks.
As a light-weight approach with superior results, JES provides a new go-to acquisition function for Bayesian optimization.
Introduction
The optimization of expensive black-box functions is a prominent task, arising across a wide range of applications.
Bayesian optimization (BO) [24, 34] is a sample-efficient approach, and has been successfully applied to various problems, including machine learning hyperparameter optimization [2, 19, 32, 36], robotics [3, 6, 22, 23], hardware design [10, 26] and tuning reinforcement learning agents like AlphaGo [7].
In BO, a probabilistic surrogate model is used for modeling the (unknown) objective.
The selection policy employed by the BO algorithm is dictated by an acquisition function, which draws on the uncertainty of the surrogate to guide the selection of the next query.
The choice of acquisition function is significant for the success of the BO algorithm.
A popular line of acquisition functions takes an information-theoretic angle, and considers the expected information gain regarding the location of the optimum that is obtained from an upcoming query.
Entropy Search (ES) [14], Predictive Entropy Search ( PES ) [15] and the earlier work of IAGO [45] select queries by maximizing this quantity.
While ES and PES are efficient in the number of queries to optimize the objective, they both require significant computational effort and complex approximations of the expected information gain, which impacts their performance and practical use [15, 46].
A related information-theoretic family of approaches considers the information gain on the optimal objective value [17, 30, 46].
Max-value Entropy Search ( MES ) [46] was the first information-theoretic approach to have a proven convergence rate, albeit only in a noiseless problem setting.
Moreover, its consideration of a one-dimensional density over the output space as opposed to a D -dimensional input space and a reduction in intricate approximations yielded a computationally efficient alternative to the ES / PES line of approaches.
Despite its empirical success, some crucial shortcomings of MES have been highlighted in recent works.
Its convergence rate has been disputed [41], and crucially, it does not differentiate between the (unobserved) maximal objective value f ∗ and the observed noisy maximum y max [25, 27, 40, 41].
As such, its assumption on the posterior distribution of the output p ( y |D , x ) does not hold in any setting where noise is present, and several follow-ups have been proposed to address the noisy problem setting [25, 27, 40, 41].
We propose an approach which merges the ES / PES and MES lines of work, and provides an allencompassing perspective on information gain regarding optimality.
We introduce Joint Entropy
Search (JES), a novel acquisition function which has the following advantages over existing infomation-theoretic approaches:
It utilizes two sources of information, by considering the entropy over both the optimum and the noiseless optimal value; 2.
It utilizes the full optimal observation, allowing it to rely primarily on exact computation through standard GP machinery instead of complex approximations; and 3.
It is computationally light-weight, requiring minimal pre-computation relative to other information-theoretic approaches which consider the input space.
Simultaneously to our work, a similar approach aimed at the multi-objective setting, was proposed by Tu et al.
The authors independently came up with the same JES acquisition function, with a subtly different approximation scheme to the one we present.
We see our work as being complementary to theirs because we both demonstrate the effectiveness of this new acquisition function in different settings - theirs being multi-objective and batch evaluations, ours being singleobjective and large levels of output noise.
Our code for reproducing the experiments is available at https://github.com/hvarfner/JointEntropySearch .
Background and related work
Bayesian optimization.
We consider the problem of optimizing a black-box function f across a set of feasible inputs X ⊂ R d :
[Equation]
We assume that f ( x ) is expensive to evaluate, and can potentially only be observed through a noise-corrupted estimate, y , where
y = f ( x ) + ϵ, ϵ ∼ N (0 , σ 2 ϵ ) for some noise level σ 2 ϵ .
In this setting, we wish to maximize f in an efficient manner, typically while adhering to a budget which sets a cap on the number of points that can be evaluated.
BO aims to globally maximize f by an initial design and thereafter sequentially choosing new points x n for some iteration n , creating the data D n = D n − 1 ∪ { ( x n , y n ) } .
After each new observation, BO constructs a probabilistic surrogate model p ( f |D n ) and uses that surrogate to build an acquisition function α ( x , D n ) .
The combination of surrogate model and acquisition function encodes the strategy for selecting the next point x n +1 .
After the full budget of N iterations is exhausted, a best configuration x ∗ N is returned as either the arg max of the observed values, or the optimum as predicted by the surrogate model.
Gaussian processes.
When constructing the surrogate, the most common choice is Gaussian processes (GPs) [29].
Formally, a GP is an infinite collection of random variables, such that every finite subset of those variables follows a multivariate Gaussian distribution.
The GP utilizes a covariance function k , which encodes a prior belief for the smoothness of f , and determines how previous observations influence prediction.
Given observations D n at iteration n , the posterior p ( f |D n ) over the objective is characterized by the posterior mean m n and variance s n of the GP:
[Equation]
where ( K n ) ij = k
( x i , x j ) , k n ( x ) =
[ k ( x , x 1 ) , . .
k ( x , x n )]
⊤ and σ 2 ϵ is the noise variance.
Alternative surrogate models include random forests [18] and Bayesian neural networks [37, 38].
Acquisition functions.
The acquisition function acts on the surrogate model to quantify the attractiveness of a point in the search space.
Acquisition functions employ a trade-off between exploration and exploitation, typically using a greedy heuristic to do so.
Simple, computationally cheap heuristics are Expected Improvement ( EI ) [5, 20].
For a noiseless function, EI selects the next point x n +1 as
[Equation]
where Z = ( y ∗ n − m n ( x ))
/s n ( x ) .
Other acquisition functions which use similar heuristics are the
Upper Confidence Bound ( UCB ) [39], and Probability of Improvement (PI) [21].
A more sophisticated approach related to EI is Knowledge Gradient (KG) [11].
Information-theoretic acquisition functions.
Information-theoretic acquisition functions [14, 15, 31, 46] and their various adaptations [1, 16, 33] seek to maximize the expected information gain I from observing a subsequent query ( x , y ) regarding the optimum, x ∗ .
This equates to reducing the uncertainty of the density over the optimum, p ( x ∗ |D )
=
P ( x = arg max x ′ ∈X f ( x ′ ) |D ) , using the information obtained through ( x , y ) .
By quantifying uncertainty through the differential entropy H, design points are selected based on the expected reduction in this quantity over p ( x ∗ |D ) .
Formally, this is expressed as the difference between the current entropy over p ( x ∗ |D ) , and the expected entropy of that density after observing the next query:
[Equation]
By utilizing the symmetric property of the mutual information, one can arrive at an equivalent expression, where the entropy is computed with regard to the density over the output y ,
[Equation]
Eq. 4 is the original formulation used in ES [14] and Eq.
5 is the formulation introduced with PES [15].
Both formulations require a series of approximations and expensive computational steps to compute the entropy in the second term.
For PES specifically, with n data points of dimension d , the second term is estimated through Monte Carlo (MC) methods by computing Cholesky decompositions of size O ( n + d 2 / 2) 3 , and approximating the Hessian at the optimum for each MC sample.
MES [46] avoids this computational hurdle by considering the information gain I (( x , y ); y ∗ |D ) regarding the optimal value y ∗ .
As such, it computes the entropy reduction for a one-dimensional density:
[Equation]
Here, it is assumed that the posterior predictive distribution p ( y |D , x , y ∗ ) is a truncated Gaussian distribution, for which the entropy can be computed in closed form.
However, p ( y |D , x , y ∗ ) takes this form only in a strictly noiseless setting [27, 40], where it holds true that f ∗ =
y max , i.e. when the maximal observation and the optimal value of the objective function coincide.
For noisy applications, this assumption leads to an overestimation of the entropy reduction [27].
Joint Entropy Search
We now present Joint Entropy Search ( JES ), a novel information-theoretic approach for Bayesian optimization.
As for other information-theoretic acquisition functions, JES considers a mutual information quantity.
However, unlike its predecessors, JES adds an additional piece of information: compared to ES / PES , it adds the density over the noiseless optimal value f ∗ , and compared to MES , it adds the density over x ∗ .
It utilizes a novel two-step reduction in the predictive entropy from conditioning on sampled optima and their associated values.
Throughout the section, we will refer to a sampled optimum and its associated value, ( x ∗ , f ∗ ) , as an optimal pair .
3.1 Joint density over the optimum and optimal value: pictorial
JES considers the joint probability density p ( x ∗ , f ∗ ) over both the optimum x ∗ and the true, noiseless optimal value f ∗ .
Fig.
1 visualizes the densities p ( x ∗ ) and p ( f ∗ ) , considered by ES / PES and MES , respectively, and the joint density p ( x ∗ , f ∗ ) , considered by JES .
As highlighted by the vertical dashed lines for the point selection of each strategy (bottom), PES chooses strictly to reduce the
[Figure] [Figure]
uncertainty over x ∗ , and as such, considers a region where the uncertainty over the optimal value is low.
However, it can effectively determine that the right side of the local optimum is more promising to query next.
MES seeks to reduce the tail of the probability density over f ∗ (right), which in this case leads to an exploratory query.
JES ’ joint probability density over optimum and optimal value captures uncertainties over both “where” and “how large” the optimum will be.
As such, it selects a point which is uncertain under both measures.
As such, JES will learn about likely locations for the optimum, while simultaneously learning probable lower and upper bounds for the optimal value, which by itself yields an effective query strategy [46] and provides valuable knowledge for future queries.
For the selected query in Fig.
1, JES will learn substantially about both x ∗ and f ∗ by querying it, whereas PES and MES learn only about one of them.
3.2 The Joint Entropy Search acquisition function
We consider the mutual information between the random variables ( x ∗ , f ∗ ) and a future query ( x , y ) :
[Equation]
[Equation]
Eq. 8 is similar to Eq.
but with the addition of x ∗ and the replacement of y ∗ with f ∗ in the conditioning of the second term.
The expectation is computed with respect to a D + 1 -dimensional joint probability density over x ∗ and f ∗ .
In Eq.
9, we make it explicit that the conditional density inside the expectation is obtained after 1.
conditioning the GP on the previous data D , plus one additional noiseless optimal pair ( x ∗ , f ∗ ) , and 2. knowing that the noiseless optimal value is in fact f ∗ .
By utilizing the complete observation ( x ∗ , f ∗ ) , we can treat it like any (noiseless) observation.
As such, we quantify much of the entropy reduction by utilizing standard GP conditioning functionality.
For 2., we cannot globally condition on f ( x ′ ) ≤ f ∗ , ∀ x ′ .
As such, we follow previous work [25, 27, 41, 46] and enforce the condition locally at the current query x .
The resulting effect is to truncate the GP’s posterior over f locally at x , upper bounding it to f ∗ .
Notably, utilizing the fantasized observation ( x ∗ , f ∗ ) guarantees that the conditioned optimal value f ∗ in JES is actually obtained, rather than serving as a possibly unattained upper bound, which is typical in the MES family of acquisition functions.
The expectation in Eq.
9 is approximated through MC by sampling L optimal pairs { ( x ∗ ℓ , f ∗ ℓ ) } Lℓ =1 from p ( x ∗ , f ∗ ) using an approximate version of Thompson Sampling ( TS ) [42], as explained in Sec. 3.3.
In Fig.
2, the resulting posterior distribution of the two-step conditioning is shown in greater detail.
As pointed out in [27, 40], after conditioning on f ∗ , the posterior predictive density over y is a sum of a truncated Gaussian distribution over f and the Gaussian noise ϵ .
The entropy reduction from the two-step conditioning yields two separate variance reduction steps over p ( y |D , x ) : a conditioning term and a truncation term.
The former is computed exactly, while the latter, generally smaller term, requires approximation, as shown in Sec. 3.4.
[Figure] [Figure]
Fig.
3 shows the difference in log variance over p ( y |D , x ) resulting from conditioning (in blue) and truncation (in orange) for the scenario in Fig.
The overall reduction is largest close to the point of conditioning, and the truncation term mainly contributes at uncertain regions far away from the conditioned point.
Moreover, the magnitude of the conditioning term will rely on the prior variance at the conditioned point, as a larger prior variance will lead to a larger reduction in entropy from conditioning.
As we average over optimal pairs, many such entropy reduction terms accumulate.
3.3 Incorporating optimal pairs
To obtain samples ( x ∗ , f ∗ ) , we utilize an approximate variant of TS [42], originally proposed in
PES [15].
We utilize Bochner’s theorem [4], which, for any stationary kernel k , asserts the existence of its Fourier dual s ( w ) .
By normalizing s ( w ) , we obtain the spectral density p ( w ) =
s ( w ) /α , where α is a normalization constant.
We can then write the kernel as an expectation,
[Equation]
where b ∼ U ( 0 , 2 π I ) .
Following Rahimi and Recht [28], we sample b and w to obtain an unbiased estimate of the kernel k .
From this approximation, approximate sample paths can be drawn as a weighted sum of basis functions.
This form allows for fast and dense querying of the sample paths – the arg max and max of which is an approximate draw from p ( x ∗ , f ∗ ) .
In PES , each sample x ∗ ℓ along with its inverted Hessian is required for computing the acquisition function.
To obtain the
Hessian, each sample needs to be thoroughly optimized through gradient-based optimization.
JES on the other hand, only requires ( x ∗ , f ∗ ) .
As such, it can rely on cheap, approximate optimization of these samples, e.g., by densely querying sample points on a non-uniform grid.
After obtaining a set of optimal pairs { ( x ∗ ℓ , y ∗ ℓ ) } L ℓ =1 , JES computes the conditional entropy quantity over the output y .
Concretely, we generate L GPs, each modeling a posterior density { p ( y |D ∪ ( x ∗ ℓ , f ∗ ℓ ) , x ) } Lℓ =1 conditioned on an optimal pair and previously observed data D .
Since each optimal pair is drawn from the current GP hyperparameter set, we know that the current hyperparameter set is the correct one even after adding the optimal pair to the data.
By consequence, JES can compute the updated inverse Gram matrix, ( K + σ 2 ϵ ) − 1 , through a rank-1 update, instead of solving a linear system of equations.
Utilizing the Sherman–Morrison formula [35], we obtain updated Gram matrices in
O ( n 2 ) for each sample, as opposed to O ( n 3 ) for solving the linear system of equations.
3.4 Approximating the truncated entropy
As highlighted in the right panel of Fig. 2, conditioning on f ∗ yields a truncated normal distribution p ( f |D ∪ ( x ∗ , f ∗ ) , x , f ∗ ) after having locally enforced the inequality f ( x ) ≤ f ∗ .
The entropy, however, is computed with regard to the density over noisy observations, y = f
+ ϵ , which follows an Extended Skew distribution [27] and as such, does not have tractable entropy.
We approximate this quantity through moment matching [25] of the truncated Gaussian distribution over f , which yields a valid lower bound on the information gain [25].
Consequently, we obtain two Gaussian densities ˆ p ( f |D ∪ ( x ∗ , f ∗ ) , x , f ∗ ) ∼ N ( m f | f ∗ , σ 2 f | f ∗ ) and p ( ϵ ) ∼ N (0 , σ 2 ϵ ) , where m f | f ∗ and σ 2 f | f ∗ are the mean and variance of the truncated Gaussian posterior p ( f |D ∪ ( x ∗ , f ∗ ) , x , f ∗ ) .
Due to independence between f and ϵ and the linearity of Gaussian distributions, we can then compute the entropy of the approximate density ˆ p y
exactly as H [ˆ p ( y |D∪
( x ∗ , f ∗ ) , x , f ∗ )] =
log(
π ( σ 2 ϵ + σ 2 f | f ∗ )) .
Moreover, the variance of the truncated Gaussian σ 2 f | f ∗ is computed as
[Equation]
(11) where σ 2 T ( α ; µ, σ 2 ) is the variance of an upper truncated Gaussian distribution with parameters ( µ, σ 2 ) , truncated at α , and m ℓn ( x ) and s ℓn ( x ) are the mean and covariance functions of the GP which is conditioned on the optimal pair ( x ∗ ℓ , f ∗ ℓ ) .
The quality of the moment matching approximation is studied in greater detail in Appendix E.
3.5 Exploitative selection to guard against model misspecification
As with all information-theoretic approaches, JES aims to reduce the uncertainty over the location of the optimum.
With this strategy, the incentive to query the perceived optimum is often lower than for heuristic approaches, such as EI .
In cases where the surrogate model is misspecified, informationtheoretic approaches risk reducing the entropy based on a faulty belief of the optimum, which can drastically impact their performance.
As a remedy, we utilize γ -exploit approach: with probability γ ,
JES will query the arg max of the posterior mean to confirm its belief of the location of the optimum.
If the model is misspecified, these exploitative steps enable the algorithm to reconsider its beliefs, rather than continuing to act based on faulty ones.
In Appendix C.2, we show how this approach can substantially improve performance in cases of surrogate model misspecification, while having negligible impact on performance in the worst case.
3.6 Putting it all together: The JES algorithm
For a sampled set of size L , containing optimal pairs { ( x ∗ ℓ , y ∗ ℓ ) } Lℓ =1 and GPs with mean and covariance functions { m ℓn ( x ) , s ℓn ( x ) } Lℓ =1 , the expression for the JES acquisition function is
[Equation] [Equation]
The first term in 13 is simply the entropy of a Gaussian that can be computed in closed form.
The second term contains both the conditioning term, which is exact, and the truncation, which is approximated as described in Sec. 3.4.
Algorithm 1 outlines pseudocode for JES in its entirety.
Experimental evaluation
Benchmarks.
We now evaluate JES on a suite of diverse tasks.
We consider three different types of benchmarks: samples drawn from a GP prior, commonly used synthetic test functions [15], and a collection of classification tasks on tabular data using an MLP, provided through HPOBench [9].
For the GP prior tasks, the hyperparameters are known for all methods to evaluate the effect of the acquisition function in isolation.
Consequently, we do not use the γ -exploit approach from Sec. 3.
← arg
∈X
15: end in this case (i.e., we set γ = 0 in Algorithm 1).
For the synthetic and MLP tasks, we marginalize over the GP hyperparameters, and set γ = 0 . 1 .
The hyperparameters of the GP prior experiments can be found in Appendix B, and ablation studies on γ in Appendix C.
Evaluation criteria.
We use two types of evaluation criteria as in [46]: simple regret and inference regret .
The simple regret r n = max x ∈X f ( x ) − max t ∈ [1 ,n ] f ( x t ) measures the value of the best queried point so far.
After a query, we may infer an arg max of the function, which is chosen as x ∗ n = arg max x ∈X m n ( x ) [14, 15, 46].
We denote the inference regret as r n = max x ∈X f ( x ) − f ( x ∗ n ) .
Since information-theoretic approaches do not necessarily seek to query the optimum, but only to know its location, inference regret characterizes how satisfying our belief of the arg max is.
Notably, this metric is non-monotonic, meaning that the best guess can worsen with time.
We use this metric in the ideal model benchmarking setting, when we sample tasks from a GP with known hyperparameters.
We use simple regret for the synthetic test functions, as it constitutes a metric that is more robust to surrogate model misspecification.
Inference regret for these tasks can be found in Appendix F. For the HPOBench tasks, inference regret is unobtainable.
The experimental setup.
We compare against other state-of-the-art information-theoretic approaches: PES [15] and MES [46], as well as EI [20].
The acquisition functions are all run in the same framework written in MATLAB, created for the original PES implementation by Hernández-Lobato et al.
All synthetic experiments were run for 50 D iterations.
In the main paper, we fix the number of MC samples for MES, PES and JES to 100 each.
In Appendix D we assess the sensitivity of JES to this number and quantify the computational expense.
In Appendix B, we provide all details on our experimental setup, including the runtime tests.
4.1 GP prior samples
We consider samples from a GP prior for four different dimensionalities: 2D, 4D, 6D, and 12D, with a noise standard deviation of 0 .
1 for a range of outputs spanning roughly [ − 10 , 10] .
These tasks constitute an optimal setting for each algorithm, as the surrogate perfectly models the task at hand.
In
Fig.
5, JES demonstrates empirically the value of the additional source of information, substantially outperforming PES and MES on all tasks.
Fig.
4 compares JES (top left) against PES , MES and EI in terms of point selection for one repetition on a two-dimensional sample task, where all runs are initialized with D +1 identical random samples.
We observe that JES succeeds in finding all attractive regions of the search space, and queries the region around the optimum densely, which is sensible in a noisy setting.
We further notice that
EI (bottom right) fails to query the two circled local optima.
PES (bottom left) also ignores two local optima to various degrees, and tends to circle the (perceived) optimum densely, which is expensive in terms of number of evaluations.
We believe this showcases a shortcoming of only
[Figure] [Figure]
considering the density over the optimum: PES circles the optimum
[Figure]
but does not query its value.
Lastly, MES (top right) successfully queries all attractive regions of the space, but also samples regions that are evidently poor the most densely out of the four approaches, despite information given by earlier (brighter) samples.
Since JES considers the information conveyed by both MES and PES , it successfully excludes the apparent suboptimal regions of the space, finds all relevant optima
[Figure]
and queries these optima in a desirable manner
[Figure]
We additionally evaluate the performance of all approaches on GP sample tasks that have a substantial amount of noise - its standard deviation roughly accounting for 10% of the total output range.
We run these tasks with the GP hyperparameters fixed a priori for a larger number of iterations, 125D, to display the stagnation of some approaches.
While MES and PES slow down approximately at the halfway point for both tasks, JES steadily improves for the entire length of the run.
This robustness to large noise magnitudes highlights the importance of intrinsically handling noisy objectives in JES .
[Table]
In Table 1, we display the runtime of each acquisition function on these tasks when marginalizing over 10 sets hyperparameters, and sampling 10 optima per set.
We time each iteration from after hyperparameters have been sampled, up until (but excluding) the query of the black-box function.
Thus, acquisition function pre-computation and optimization are included.
The runtime of JES is only marginally slower than that of MES with Gumbel sampling, while being at least an order of magnitude faster than PES for all displayed dimensionalities.
4.2 Synthetic test functions
Next, in Fig.
[Figure]
we study the performance of JES on three optimization test functions: Branin (2D),
Hartmann (3D) and Hartmann (6D).
For these tasks, we follow convention [15, 30] and marginalize over GP hyperparameters.
On Branin, JES starts out slightly slower than MES but reaches the same performance in 100 iterations; and on the two Hartmann functions, JES performs amongst the best in the beginning and clearly best in the end.
We note that PES experienced numerical issues on Branin, and as such, we acknowledge that its performance should be better than what is reported.
[Figure]
4.3 MLP tasks
Lastly
[Figure]
we evaluate the performance of JES on the tuning an MLP model’s 4 hyperparameters for 20D iterations on three datasets.
These tasks are part of the OpenML 1 library of tasks, and the
HPO benchmark is provided through the HPOBench [9] suite.
The tasks, which are classification problems with tabular data, regard classifying: (1) vehicles into one of four different categories, (2) blood donors for a blood transfusion service center
[Figure]
and (3) anonymous credit card applications.
We measure the best observed classification accuracy.
Notably, these tasks have a large amount of noise, which causes the performance to fluctuate substantially between repetitions.
We observe that JES performs almost identically to MES and EI on vehicle silhouette and credit application, and substantially better on blood transfusion.
PES lags behind significantly on all tasks.
[Figure]
Conclusions
We have presented Joint Entropy Search, an information-theoretic acquisition function that considers an entirely new quantity, namely the joint density over the optimum and optimal value.
By utilizing the entropy reduction from fantasized optimal observations, JES obtains a simple form for the entropy reduction regarding the joint distribution.
As such, the additional information considered comes with minimal computational overhead, avoids restrictive assumptions on the objective, and yields state-of-the-art performance along with superior decision-making.
We believe JES to be a new go-to acquisition function for BO, and to establish a new standard for subsequent information-theoretic techniques.
Limitations and Future Work
The main contribution of this paper is to provide a novel information-theoretic acquisition function which, given a sufficiently accurate model, yields impressive results.
However, the non-myopic, speculative nature of information-theoretic approaches lend them to be susceptible to model misspecification, such as a poor choice of GP kernel or GP hyperparameters.
In our view, information-theoretic approaches are possibly more susceptible to this issue than their myopic counterparts ( EI , UCB , TS ).
While we propose a remedy to stabilize and improve the acquisition function under model misspecification with the γ -exploit approach, this technique only serves to discover misspecification and adjust accordingly, but not to inherently fix the misspecification.
We believe misspecification can only be remedied by altering the surrogate model.
It is thus very promising to combine advanced modelling techniques with information-theoretic acquisition functions, as already done with the additive GP approach utilized in conjunction with MES by Wang and Jegelka [46]; further promising additions would be to tackle heterogeneous noise and input warping as done by HEBO [8].
We also note that, since JES computes the entropy reduction from conditioning on the optimal pair, it relies on some level of noise in the objective.
A surrogate model with zero noise will result in an infinite information gain for every optimal pair, which (by utilizing some random tie-breaking strategy) would make JES equivalent to TS.
However, if JES is to be used in a completely noiseless setting, we argue that a small noise term should be added as a remedy.
As this is done by default in many prominent GP frameworks [12, 13, 44], we do not view this as a major limitation of our approach.
Nevertheless, improving upon this strategy would be interesting in future work.
For future work, we also envision work on the adaptation of JES to various different domains, such as multi-fidelity [47] and multi-objective optimization [1], as well as the integration of user prior knowledge over the location of the optimum [19] to accelerate optimization.
Acknowledgments and Disclosure of Funding
Luigi Nardi was supported in part by affiliate members and other supporters of the Stanford DAWN project — Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware.
Carl Hvarfner and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348.
Frank Hutter acknowledges support through TAILOR, a project funded by the EU Horizon 2020 research and innovation programme under GA No 952215, by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) under grant number 417962828, by the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963FUGG, and by the European Research Council (ERC)
Consolidator Grant “Deep Learning 2.0” (grant no. 101045765).
The computations were also enabled by resources provided by the Swedish National
Infrastructure for Computing (SNIC) at LUNARC partially funded by the Swedish Research Council through grant agreement no. 2018-05973.
Funded by the European Union.
Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European
Union or the ERC.
Neither the European Union nor the ERC can be held responsible for them.
A Broader impact
Our work proposes a novel acquisition function for Bayesian optimization.
The approach is foundational and does not have direct societal or ethical consequences.
However, JES will be used in the development of applications for a wide range of areas and thus indirectly contribute to their impacts on society.
As an algorithm that can be used for HPO, JES intends to cut resource expenditure associated with model training, while increasing their performance.
This can help reduce the environmental footprint of machine learning research.
B Experimental setup
Frameworks.
For all tasks and acquisition functions, we use the original PES implementation in
MATLAB by Hernández-Lobato et al.
[15], which uses the GPStuff [44] library.
The implementation optimizes the acquisition function, and the posterior mean, by sampling a dense grid of points, and uses a gradient-based optimizer to further optimize the single best point.
For better accuracy, we substantially increased the number of grid points.
The JES implementation can be found at https: //github.com/jointentropysearch/JointEntropySearch .
GP sample tasks.
To generate the GP sample tasks, we use a random Fourier feature [28] with weights drawn from the spectral density of a squared exponential kernel.
For dimension-wise length scale θ d , output scale σ 2 , and noise variance σ 2 ϵ , the hyperparameters per task are shown in Table 2.
The range in the last column is a rough approximation of the magnitude of the output spanned by
D
[Table]
[Table] [Table]
[Table] [Table]
[ −
[Table]
each GP sample.
The length scales of the samples are gradually increased with each dimensionality to maintain a reasonable level of difficulty for all tasks.
Since the optimal values for these tasks are unavailable, they are approximated through a dense random search, followed by local search on the most promising subset of points.
Runtime tests.
For the runtime tests, we run each acquisition function on 100 seeds on the 2,and 6-dimensional GP sample tasks for 100 iterations each.
To consider the runtime induced by each acquisition function in a realistic setting, we marginalize over 10 hyperparameter sets (including noise variance), and sample 10 optima each for JES , MES and PES .
For these experiments, the runtime of an acquisition function is considered to be the time the time from after GP hyperparameters are sampled and the GP covariance has been inverted, until the query has been selected.
Thus, only acquisition function setup and acquisition function optimization are considered as part of the runtime.
All acquisition functions are optimized using an identical budget of 10000 raw samples, and a subsequent gradient-based optimization around the single best point.
Synthetic test functions.
For the synthetic test functions, 100 sampled optimal pairs are used for each acquisition function.
GP hyperparameters are marginalized over for these tasks, so an equal number of optimal pairs are sampled for each hyperparameter set.
The hyperparameters are re-sampled on a fixed schedule throughout the run.
Naturally, the sampled maxima were updated at each iteration.
Moreover, each test function was also given a fixed amount of noise.
Regret was computed not from the noisy observed value, but from the true, noiseless function value.
MLP classification tasks.
All the classification tasks have a substantial amount of noise.
As noiseless objective values are unavailable, we report the observed classification accuracy.
5 hyperparameters are available in HPOBench for these tasks.
However, one of them (number of layers, depth ) is held fixed due to its small integer-valued domain and the lack of integer hyperparameter support in
[Table] [Table]
the MATLAB framework.
As seen in Tab. 4, the two other integer-valued hyperparameters batch size and width have orders of magnitude larger domains, and can therefore reasonably be treated as continuous hyperparameters.
All tasks are optimized in the [0, 1] range, and are scaled, transformed, and rounded to the nearest integer in the objective function where necessary.
All non-fixed parameters are evaluated in log scale.
The three tasks evaluated are Australian , Blood-transfusion-service-center , and Vehicle , with HPOBench task numbers
#146818, #10101, #53, respectively.
Compute resources.
All experiments are carried out on Intel Xeon Gold 6130 CPUs.
Each repetition is run on a single core.
In total, approximately 50 , 000 core hours are used for the experiments in the main paper, and an additional 20 , 000 for the appendix.
C Ablation studies and model misspecification
We provide ablation studies for the hyperparameter controlling the ratio of exploitative selections γ .
Moreover, we show how γ -exploit improves inference regret by recognizing regions believed to be optimal as poor.
Lastly, we display the robustness of JES to the noise variance σ 2 ϵ .
C.1 Ablation studies
We provide an ablation study on γ in terms of both the simple and inference regret of JES.
While simple regret may be a more practically relevant metric, inference regret helps understand the ability of the acquisition function to successfully locate the optimum.
Fig.
9 shows that γ > 0 improves simple regret, which is to be expected from its occasional greedy selection.
However, as is shown in Fig.
10, a moderate fraction γ ∈ { 0 . 05 , 0 .
1 } also yields comparable or even improved inference regret on all tasks.
Notably, γ = 0 .
2 yields slightly worse performance on Hartmann (6D), but yields marginally improved performance on Branin and Hartmann (3D).
As such, the γ -exploit approach not only improves performance in terms of simple regret, but yields improved inference as well.
C.2 Model misspecification
As mentioned in Sec. 3.5, the performance of information-theoretic methods can suffer substantially from model misspecification.
In Fig.
11, we show for JES , PES and MES how the γ -exploit approach helps stabilize inference and improve inference regret, and yields substantially better simple regret for all methods.
For Michalewicz (10D), we observe that the inference regret of MES gets substantially worse after iteration 150.
With the γ -exploit approach, this issue is severely reduced.
In Fig.
12, the corresponding simple regrets for MES deviate at iteration 150.
The same behavior can be observed for
JES on Levy (8D) around iteration 200.
Across all test functions in Fig.
11 and Fig.
12, an γ -exploit strategy yields comparable or improved inference regret, and strictly improved simple regret for all acquisition functions.
Moreover, we observe that MES is generally the top-performing acquisition
[Figure] [Figure]
[Figure]
function, implying that it is the most robust to model misspecification.
Using the γ -exploit approach, the acquisition function verifies whether the belief over the location of the optimum is correct under the current model hyperparameters.
If it is not, it re-calibrates its belief.
C.3 Ablation study on σ 2 ϵ
For the noise variance ablation study, we once again consider the GP sample tasks.
We fix the GP noise hyperparameter σ 2 ϵ to the correct value prior to the start of the experiment.
In Fig.
13, we show that the performance of JES is robust with respect to the noise level, while the performance of MES and PES decrease more drastically as the level of noise increases.
D Dependence on the number of MC samples
We show in Fig.
14 the dependence of JES on the number of MC samples for the GP sample tasks.
JES displays a slight dependence on the number of GP samples, as initial performance improves marginally for larger number of samples.
This is more prominent for higher-dimensional tasks, where a lower number of samples causes a substantially slower start.
This can be explained by the fact that a larger number of sampled optimal pairs are required to accurately model a higher-dimensional density.
Notably, the final regret on the 12-D benchmark is better for lower numbers of samples, such as JES -4.
One potential explanation for this is its inability to model the joint distribution accurately.
If all realized optimal pairs are close to the perceived optimum, JES is almost certainly going to sample there.
Moreover, since the information gain for each sample is relatively low for samples in a well-explored region, the information gain of a few samples in an unexplored region may outweight the information gain of a much larger number of samples in a well-explored region.
For JES -4 and
JES -20, it is more likely that all of the sampled optimal pairs are close to the optimum in this manner, while there is still small positive density on other parts of the search space.
As such, it is possible that
JES -4 and JES -20 over-exploit slightly in the 12-dimensional benchmark.
[Figure]
[Figure] [Figure]
E Approximation Quality
As stated in Sec. 3.
[Figure]
we approximate the entropy of the posterior over observations conditioned on the data and the optimal pair H [ p ( y |D ∪ ( x ∗ , f ∗ ) , x , f ∗ )] – the entropy of the sum of a Gaussian and a truncated Gaussian variable – by moment matching of the truncated distribution.
We now show the quality of this approximation, and what impact it can potentially have on point selection.
To do so, we utilize the results from Nguyen et al.
[27] regarding the density of p ( y |D , x , f ∗ ) .
We note that the approximation regards the truncation from knowing the optimal value f ∗ , which constitutes an additional reduction in entropy after having conditioned the GP on the new observation as in Sec. 3.3.
As such, the approximation considers only a fraction of the total entropy reduction, as visualized in
Fig.
To establish the quality of the approximation, we compare our approach to approximating the entropy by MC.
Naturally, the MC approach is more computationally expensive than moment matching, but
[Figure]
[Figure] [Figure]
i
[Figure]
f e r e n c e r
[Figure]
g r e t
[Figure]
yields an asymptotically correct result.
In Fig.
15, we show for varying noise levels, expressed as the noise variance ratio of the total variance, and truncation quantiles Φ − 1 ( α ) , the difference in entropy between an MC and a moment matching approach.
For example, a truncation quantile of 10 − 2 means that the upper 99% of the density of the posterior distribution is removed as a result of truncation.
We see that the approximate entropy reduction from moment matching is consistently lower than for the asymptotically exact MC approach.
Moreover, the approximation error seemingly increases logarithmically with the truncation quantile.
As such, we can expect modest underestimates of the entropy reduction when we truncate the posterior to an extreme degree, and the level of noise is low.
In the right image of Fig 2, the posterior is severely truncated left of the conditioned location.
However, the noise variance constitutes a large fraction of the total variance at this location, which means that the approximation is still accurate with respect to the true entropy reduction.
The scenario represented in the bottom left corner of the grid, where we severely truncate the posterior and the noise variance is low, does not reasonably occur in practice.
The aforementioned scenario would entail sampling an optimal pair which is several orders of magnitudes worse than the mean of the GP at uncertain (in the sense that σ >> σ ϵ ) locations.
Lastly, the blue region in the upper left corner of
[Figure] [Figure]
[Figure]
the grid represents a region where we have less truncation, and the noise is a relatively insignificant part of the total variance.
Fig.
15 shows that the entropy reduction from truncation in this region is underestimated by approximately 15%.
As such, the approximation error leads to a slightly less explorative strategy than what a strategy with exact computation of the truncation term would provide.
F Regret Measures
We display the regret measures for the GP sample and synthetic test functions.
F.1 GP sample tasks
In Fig.
16, we show the simple regret for the GP sample tasks.
We note that the simple and inference regrets for MES are approximately equal, while there is a substantial difference for JES .
For PES , the difference between simple and inference regret is the most pronounced at approximately two order of magnitude at the most.
[Figure]
F.2 Synthetic test functions
Next, we show the inference regret for the synthetic test functions in Fig.
We note that he simple and inference regrets for JES are approximately equal.
For PES , the difference between simple and inference regret is once again very pronounced.
Notably, the simple regret is significantly better than the inference regret for MES on Branin, implying that it does not yet have full knowledge of where the optimum is.
We once again note the numerical issues of PES on Branin.
Moreover, the inference regret of MES gets marginally worse for Hartmann (6D) from approximately iteration 100 until the end of the run, implying that its knowledge about the location of the optimum gets worse.