Joint Entropy Search For Maximally-Informed Bayesian Optimization
Anonymous Author(s)
Abstract
Information-theoretic Bayesian optimization techniques have become popular foroptimizing expensive-to-evaluate black-box functions due to their non-myopic qualities.
Entropy Search and Predictive Entropy Search both consider the entropy overthe optimum in the input space, while the recent Max-value Entropy Search considers the entropy over the optimal value in the output space.
We propose Joint Entropy
Search (JES), a novel information-theoretic acquisition function that considers anentirely new quantity, namely the entropy over the joint optimal probability densityover both input and output space.
To incorporate this information, we consider thereduction in entropy from conditioning on fantasized optimal input/output pairs.
The resulting approach primarily relies on standard GP machinery and removescomplex approximations typically associated with information-theoretic methods.
With minimal computational overhead, JES shows superior decision-making, andyields state-of-the-art performance for information-theoretic approaches across awide suite of tasks.
As a light-weight approach with superior results, JES providesa new go-to acquisition function for Bayesian optimization.
Introduction
The optimization of expensive black-box functions is a prominent task, arising across a wide rangeof applications.
Bayesian optimization (BO) [24, 34] is a sample-efficient approach and has beensuccessfully applied to various problems, including machine learning hyperparameter optimization [2,19 , 32, 36], robotics [3, 6, 23], hardware design [10, 26] and tuning reinforcement learning agents like
AlphaGo [7].
In BO, a probabilistic surrogate model is used for modeling the (unknown) objective.
The selection policy employed by the BO algorithm is dictated by an acquisition function, whichdraws on the uncertainty of the surrogate to guide the selection of the next query.
The choice of acquisition function is significant for the success of the BO algorithm.
A popular lineof acquisition functions takes an information-theoretic angle and considers the expected informationgain regarding the location of the optimum that is obtained from an upcoming query.
Entropy Search(ES)
[14], Predictive Entropy Search ( PES ) [15] and the earlier work of IAGO [44] select queriesby maximizing this quantity.
While ES and PES are efficient in the number of queries to optimizethe objective, they both require significant computational effort and complex approximations of theexpected information gain, which impacts their performance and practical use [15, 45].
To addressthis computational expense, efficient parameterizations of the objective have been proposed [30].
However, this solution imposes constraints in the ability to accurately model the objective.
A related information-theoretic family of approaches considers the information gain on the optimalobjective value [17, 45].
Max-value Entropy Search ( MES ) [45] was the first information-theoreticapproach to have a proven convergence rate, albeit only in a noiseless problem setting.
Moreover,its consideration of a one-dimensional density over the output space as opposed to a D -dimensional input space and a reduction in intricate approximations yielded a computationally efficient alternativeto the ES / PES line of approaches.
Despite its empirical success, some crucial shortcomings of MEShave been highlighted in recent works.
It’s convergence rate has been disputed [41], and crucially, itdoes not differentiate between the (unobserved) maximal objective value f ∗ and the observed noisymaximum y max [25, 27, 40, 41].
As such, its assumption on the posterior distribution of the outputp ( y |D , x ) does not hold in any setting where noise is present, and several follow-ups have beenproposed to address the noisy problem setting [25, 27, 40, 41].
We propose an approach which merges the ES / PES and MES lines of work, and provides an allencompassing perspective on information gain regarding optimality.
We introduce Joint Entropy
Search (JES), a novel acquisition function which has the following advantages over existinginfomation-theoretic approaches:
It utilizes two sources of information, by considering the entropy over both the optimum
and the noiseless optimal value;
It utilizes the full optimal observation, allowing it to rely primarily on exact computation
through standard GP machinery instead of complex approximations; and3.
It is computationally light-weight, requiring minimal pre-computation relative to otherinformation-theoretic approaches which consider the input space
The code for reproducing the experiments is available athttps://github.com/jointentropysearch
JointEntropySearch
Background and related work
Bayesian optimization.
We consider the problem of optimizing a black-box function f across a setof feasible inputs X ⊂ R d :
[Equation]
We assume that f ( x ) is expensive to evaluate, and can potentially only be observed through anoise-corrupted estimate, y , where
y = f ( x ) + ϵ, ϵ ∼ N (0 , σ 2 ϵ ) for some noise level σ 2 ϵ .
In thissetting, we wish to maximize f in an efficient manner, typically while adhering to a budget whichsets a cap on the number of points that can be evaluated.
BO aims to globally maximize f by aninitial design and thereafter sequentially choosing new points x n for some iteration n , creating thedata D n = D n − 1 ∪ { ( x n , y n ) } .
After each new observation, BO constructs a probabilistic surrogatemodel p ( f |D n ) and uses that surrogate to build an acquisition function α ( x , D n ) .
The combinationof surrogate model and acquisition function encodes the strategy for selecting the next point x n +1 .
After the full budget of N iterations is exhausted, a best configuration x ∗ N is returned as either thearg max of the observed values, or the optimum as predicted by the surrogate model.
Gaussian processes.
When constructing the surrogate, the most common choice is Gaussianprocesses (GPs) [29].
Formally, a GP is an infinite collection of random variables, such thatevery finite subset of those variables follows a multivariate Gaussian distribution.
The GP utilizesa covariance (or kernel) function k , which encodes a prior belief for the smoothness of f , anddetermines how previous observations influence prediction.
Typical choices for the kernel are thesquared exponential and Matérn-5/2 [22] kernel.
Given the observations D n at iteration n , the densityp ( f |D n ) over the objective is characterized by the posterior mean m n and variance s n of the GP:
[Equation]
where ( K n ) ij = k
( x i , x j ) , k n ( x ) =
[ k ( x , x 1 ) , . .
k ( x , x n )]
⊤ and σ 2 ϵ is the noise variance.
Alternative surrogate models include random forests [18] and Bayesian neural networks [37, 38].
Acquisition functions.
The acquisition function acts on the surrogate model to quantify theattractiveness of a point in the search space.
Acquisition functions employ a trade-off betweenexploration and exploitation, typically using a greedy heuristic to do so.
Simple, computationallycheap heuristics are Expected Improvement ( EI ) [5, 20].
For a noiseless function, EI selects the nextpoint x n +1 as
[Equation]
where Z = ( y ∗ n − m n ( x ))
/s n ( x ) .
Other acquisition functions which use similar heuristics are the
Upper Confidence Bound ( UCB ) [39], and Probability of Improvement (PI) [21].
A more sophisticatedapproach related to EI is Knowledge Gradient (KG) [11].
Information-theoretic acquisition functions.
Information-theoretic acquisition functions [14, 15,31, 45] and their various adaptations [1, 16, 33] seek to maximize the expected information gain Ifrom observing a subsequent query ( x , y ) regarding the optimum, x ∗ .
This equates to reducing theuncertainty of the density over the optimum, p ( x ∗ |D )
=
P ( x = arg max x ′ ∈X f ( x ′ ) |D ) , using theinformation obtained through ( x , y ) .
By quantifying uncertainty through the differential entropy H,design points are selected based on the expected reduction in this quantity over p ( x ∗ |D ) .
Formally,this is expressed as the difference between the current entropy over p ( x ∗ |D ) , and the expected entropyof that density after observing the next query:
[Equation]
By utilizing the symmetric property of the mutual information, one can arrive at an equivalentexpression, where the entropy is computed with regard to the density over the output y ,
[Equation]
Eq. 4 is the original formulation used in ES [14] and Eq.
5 is the formulation introduced with PES [15].
Both formulations require a series of approximations and expensive computational steps to computethe entropy in the second term.
For PES specifically, with n data points of dimension d , the secondterm is estimated through Monte Carlo (MC) methods by computing Cholesky decompositions ofsize O ( n + d 2 / 2) 3 , and approximating the Hessian at the optimum for each MC sample.
MES [45] avoids this computational hurdle by considering the information gain I (( x , y ); y ∗ |D )regarding the optimal value y ∗ .
As such, it computes the entropy reduction for a one-dimensionaldensity:
[Equation]
Here, it is assumed that the posterior predictive distribution p ( y |D , x , y ∗ ) is a truncated Gaussiandistribution, for which the entropy can be computed in closed form.
However, p ( y |D , x , y ∗ ) takesthis form only in a strictly noiseless setting [27, 40], where it holds true that f ∗ =
y max , i.e. when themaximal observation and the optimal value of the objective function coincide.
For noisy applications,this assumption leads to an overestimation of the entropy reduction [27].
Joint Entropy Search
We now present Joint Entropy Search ( JES ), a novel information-theoretic approach for Bayesianoptimization.
As for other information-theoretic acquisition functions, JES considers a mutualinformation quantity.
However, unlike its predecessors, JES adds an additional piece of information:compared to ES / PES , it adds the density over the noiseless optimal value f ∗ , and compared to MES ,it adds the density over x ∗ .
It utilizes a novel two-step reduction in the predictive entropy fromconditioning on sampled optima and their associated values.
Throughout the section, we will refer toa sampled optimum and its associated value, ( x ∗ , f ∗ ) , as an optimal pair .
3.1 Joint density over the
optimum
and
optimal
value:
pictorial
JES considers the joint probability density p ( x ∗ , f ∗ ) over both the optimum x ∗ and the true, noiselessoptimal value f ∗ .
Fig.
1 visualizes the densities
Fig.
[Figure] [Figure]
3.2 The Joint Entropy Search acquisition function
We consider the mutual information between the random variables ( x ∗ , f ∗ ) and a future query ( x , y ) :
[Equation]
[Equation]
Eq. 8 is similar to Eq.
but with the addition of x ∗ and the replacement of y ∗ with f ∗ in theconditioning of the second term.
The expectation is computed with respect to a D + 1 -dimensionaljoint probability density over x ∗ and f ∗ .
In Eq.
9, we make it explicit that the conditional densityinside the expectation is obtained after 1.
conditioning the GP on the previous data D , plus oneadditional noiseless optimal pair ( x ∗ , f ∗ ) , and 2. knowing that the noiseless optimal value is in factf ∗ .
By utilizing the complete observation ( x ∗ , f ∗ ) , we can treat it like any (noiseless) observation.
Assuch, we quantify much of the entropy reduction by utilizing standard GP conditioning functionality.
For 2., we gain knowledge of the maximum of the noiseless objective, globally.
Following [25, 27,41, 45], the resulting effect is to truncate the GP’s posterior over f , upper bounding it to f ∗ .
Notably,utilizing the fantasized observation ( x ∗ , f ∗ ) guarantees that the conditioned optimal value f ∗ in JESis actually obtained, rather than serving as a possibly unattained upper bound, which is typical inthe MES family of acquisition functions.
The expectation in Eq.
9 is approximated through MC bysampling L optimal pairs { ( x ∗ ℓ , f ∗ ℓ ) } Lℓ =1 from p ( x ∗ , f ∗ ) using an approximate version of Thompson
Sampling ( TS ) [42], as explained in Sec. 3.3.
In Fig.
2, the resulting posterior distribution of thetwo-step conditioning is shown in greater detail.
As pointed out in [27, 40], after conditioning on f ∗ ,the posterior predictive density over y is a sum of a truncated Gaussian distribution over f and the
Gaussian noise ϵ .
The entropy reduction from the two-step conditioning yields two separate variancereduction steps over p ( y |D , x ) : a conditioning term and a truncation term.
The former is computedexactly, while the latter, generally smaller term, requires approximation, as shown in Sec. 3.4.
[Figure]
Fig.
3 shows the difference in log variance overp ( y |D , x ) resulting from conditioning (in blue) andtruncation (in orange) for the scenario in Fig.
The overall reduction is largest close to the pointof conditioning, and the truncation term mainlycontributes at uncertain regions far away from theconditioned point.
Moreover, the magnitude of theconditioning term will rely on the prior varianceat the conditioned point, as a larger prior variancewill lead to a larger reduction in entropy from conditioning.
As we average over optimal pairs, manysuch entropy reduction terms accumulate.
[Figure]
3.3 Incorporating optimal pairs
To obtain samples ( x ∗ , f ∗ ) , we utilize an approximate variant of TS [42], originally proposed in
PES [15].
We utilize Bochner’s theorem [4], which, for any stationary kernel k , asserts the existenceof its Fourier dual s ( w ) .
By normalizing s ( w ) , we obtain the spectral density p ( w ) =
s ( w ) /α ,where α is a normalization constant.
We can then write the kernel as an expectation,
[Equation]
(10) where b ∼ U ( 0 , 2 π I ) .
Following Rahimi and Recht [28], we sample b and w to obtain an unbiasedestimate of the kernel k .
From this approximation, approximate sample paths can be drawn as aweighted sum of basis functions.
This form allows for fast and dense querying of the sample paths– the arg max and max of which is an approximate draw from p ( x ∗ , f ∗ ) .
In PES , each sample x ∗ ℓalong with its inverted Hessian is required for computing the acquisition function.
To obtain the
Hessian, each sample needs to be thoroughly optimized through gradient-based optimization.
JES onthe other hand, only requires ( x ∗ , f ∗ ) .
As such, it can rely on cheap, approximate optimization ofthese samples, e.g., by densely querying sample points on a non-uniform grid.
After obtaining a set of optimal pairs { ( x ∗ ℓ , y ∗ ℓ ) } Lℓ =1 , JES computes the conditional entropy quantityover the output y .
Concretely, we generate L GPs, each modeling a posterior density { p ( y |D ∪( x ∗ ℓ , f ∗ ℓ ) , x ) } Lℓ =1 conditioned on an optimal pair and previously observed data D .
Since each optimalpair is drawn from the current GP hyperparameter set, we know that the current hyperparameter set isthe correct one even after adding the optimal pair to the data.
By consequence, JES can compute theupdated inverse Gram matrix, ( K + σ 2 ϵ ) − 1 , through a rank-1 update, instead of solving a linear systemof equations.
Utilizing the Sherman–Morrison formula [35], we obtain updated Gram matrices in
O ( n 2 ) for each sample, as opposed to O ( n 3 ) for solving the linear system of equations.
3.4 Approximating the truncated entropy
As highlighted in the right panel of Fig. 2, conditioning on f ∗ yields a truncated normal distributionp ( f |D ∪ ( x ∗ , f ∗ ) , x , f ∗ ) , but the entropy is computed with regard to the density over noisy observations, y = f
+ ϵ , which leads to an intractable entropy.
We approximate this quantity through momentmatching of the truncated Gaussian distribution over f .
Consequently, we obtain two Gaussiandensities ˆ p ( f |D ∪ ( x ∗ , f ∗ ) , x , f ∗ ) ∼ N ( m f | f ∗ , σ 2 f | f ∗ ) and p ( ϵ ) ∼ N (0 , σ 2 ϵ ) , where m f | f ∗ andσ 2 f | f ∗ are the mean and variance of the truncated Gaussian posterior p ( f |D ∪ ( x ∗ , f ∗ ) , x , f ∗ ) .
Dueto independence between f and ϵ and the linearity of Gaussian distributions, we can then compute theentropy of the approximate density ˆ p y exactly as H [ˆ p ( y |D∪
( x ∗ , f ∗ ) , x , f ∗ )] =
log(
π ( σ 2 ϵ + σ 2 f | f ∗ )) .
←
∈X
Moreover, the variance of the truncated Gaussian σ 2 f | f ∗ is computed asσ
[Equation]
( x
[Equation]
D
[Equation]
=
[Equation] [Equation]
)) (11) where σ 2 T ( α ; µ, σ 2 ) is the variance of an upper truncated Gaussian distribution with parameters( µ, σ 2 ) , truncated at α , and m ℓn ( x ) and s ℓn ( x ) are the mean and covariance functions of the GP whichis conditioned on the optimal pair ( x ∗ ℓ , f ∗ ℓ ) .
The quality of the moment matching approximation isstudied in greater detail in Appendix E.
3.5 Greedy selection to guard against model misspecification
As with all information-theoretic approaches, JES aims to reduce the uncertainty over the location ofthe optimum.
With this strategy, the incentive to query the perceived optimum is often lower than forheuristic approaches, such as EI .
In cases where the surrogate model is misspecified, informationtheoretic approaches risk reducing the entropy based on a faulty belief of the optimum, which candrastically impact their performance.
As a remedy, we utilize an inverse- γ -greedy approach: withprobability γ , JES will query the arg max of the posterior mean to confirm its belief of the locationof the optimum.
If the model is misspecified, these greedy steps enable the algorithm to reconsiderits beliefs, rather than continuing to act based on faulty ones.
In Appendix C.2, we show how thisapproach can substantially improve performance in cases of surrogate model misspecification, whilehaving negligible impact on performance in the worst case.
3.6 Putting it all together: The JES algorithm
For a sampled set of size L , containing optimal pairs { ( x ∗ ℓ , y ∗ ℓ ) } Lℓ =1 and GPs with mean and covariance functions { m ℓn ( x ) , s ℓn ( x ) } Lℓ =1 , the expression for the JES acquisition function is
[Equation] [Equation]
The first term in 13 is simply the entropy of a Gaussian that can be computed in closed form.
The second term contains both the conditioning term, which is exact, and the truncation, which isapproximated as described in Sec. 3.4.
Algorithm 1 outlines pseudocode for JES in its entirety.
Experimental evaluation
Benchmarks.
We now evaluate JES on a suite of diverse tasks.
We consider three different typesof benchmarks: samples drawn from a GP prior, commonly used synthetic test functions [15], and
Joint Entropy Search
[Figure]
[Figure]
a collection of classification tasks on tabular data using an MLP, provided through HPOBench [9].
For the GP prior tasks, the hyperparameters are known for all methods to evaluate the effect of theacquisition function in isolation.
Consequently, we do not use the inverse greedy approach from
Sec. 3.5 in this case (i.e., we set γ = 0 in Algorithm 1).
For the synthetic and MLP tasks, wemarginalize over the GP hyperparameters, and set γ = 0 .
The hyperparameters of the GP priorexperiments can be found in Appendix B, and ablation studies on γ in Appendix C.
Evaluation criteria.
We use two types of evaluation criteria as in [45]: simple regret and inferenceregret .
The simple regret r n = max x ∈X f ( x ) − max t ∈ [1 ,n ] f ( x t ) measures the value of the bestqueried point so far.
After a query, we may infer an arg max of the function, which is chosen as x ∗ n =arg max x ∈X m n ( x ) [14, 15, 45].
We denote the inference regret as r n = max x ∈X f ( x ) − f ( x ∗ n ) .
Since information-theoretic approaches do not necessarily seek to query the optimum, but only toknow its location, inference regret characterizes how satisfying our belief of the arg max is.
Notably,this metric is non-monotonic, meaning that the best guess can worsen with time.
We use this metric inthe ideal model benchmarking setting, when we sample tasks from a GP with known hyperparameters.
We use simple regret for the synthetic test functions, as it constitutes a metric that is more robust tosurrogate model misspecification.
Inference regret for these tasks can be found in Appendix F. Forthe HPOBench tasks, inference regret is unobtainable.
The experimental setup.
We compare against other state-of-the-art information-theoretic approaches: PES [15] and MES [45], as well as EI [20].
The acquisition functions are all run in the sameframework written in MATLAB, created for the original PES implementation by Hernández-Lobatoet al.
All synthetic experiments were run for 50 D iterations.
In the main paper, we fix thenumber of MC samples for MES, PES and JES to 100 each.
In Appendix D we assess the sensitivityof JES to this number and quantify the computational expense.
In Appendix B, we provide all detailson our experimental setup.
4.1 GP prior samples
We consider samples from a GP prior for four different dimensionalities: 2D, 4D, 6D, and 12D, witha noise standard deviation of 0 .
1 for a range of outputs spanning roughly [ − 10 , 10] .
These tasksconstitute an optimal setting for each algorithm, as the surrogate perfectly models the task at hand. In
Fig. 5, JES demonstrates empirically the value of the additional source of information, substantiallyoutperforming PES and MES on all tasks.
[Figure]
Hartmann-
[Figure]
Fig.
4 compares JES (top left) against PES , MES and EI in terms of point selection for one repetition ona two-dimensional sample task, where all runs are initialized with D +1 identical random samples.
Weobserve that JES succeeds in finding all attractive regions of the search space, and queries the regionaround the optimum densely, which is sensible in a noisy setting.
We further notice that EI (bottomright) fails to query the two circled local optima.
PES (bottom left) also ignores two local optima to various degrees, and tends to circle the (perceived) optimum densely, which is expensive in terms of number of evaluations.
We believe this showcases a shortcoming of only considering the density over theoptimum: PES circles the optimum, but does not query its value.
Lastly, MES (top right) successfullyqueries all attractive regions of the space, but also samples regions that are evidently poor the mostdensely out of the four approaches, despite information given by earlier (brighter) samples.
Since JESconsiders the information conveyed by both MES and PES , it successfully excludes the apparent suboptimal regions of the space, finds all relevant optima, and queries these optima in a desirable manner.
[Figure]
We additionally evaluate the performance of allapproaches on GP sample tasks that have a substantial amount of noise - its standard deviationroughly accounting for 10% of the total outputrange.
We run these tasks for a larger numberof iterations, 125D, to display the stagnationof some approaches.
While MES and PES slowdown approximately at the halfway point forboth tasks, JES steadily improves for the entirelength of the run.
This robustness to large noisemagnitudes highlights the importance of intrinsically handling noisy objectives in JES .
4.2 Synthetic test functions
Next, in Fig.
7, we study the performance of JES on three optimization test functions: Branin (2D),
Hartmann (3D) and Hartmann (6D).
For these tasks, we follow convention [15, 30] and marginalizeover GP hyperparameters.
On Branin, JES starts out slightly slower than MES but reaches the sameperformance in 100 iterations; and on the two Hartmann functions, JES performs amongst the best inthe beginning and clearly best in the end.
We note that PES experienced numerical issues on Branin,and as such, we acknowledge that its performance should be better than what is reported.
4.3 MLP tasks
Lastly, we evaluate the performance of JES on the tuning an MLP model’s 4 hyperparameters
for20 D iterations on three datasets.
These tasks are part of the OpenML 1 library of tasks, and the
HPO benchmark is provided through the HPOBench [9] suite.
The tasks, which are classificationproblems with tabular data, regard classifying: (1) vehicles into one of four different categories, (2)blood donors for a blood transfusion service center, and (3) anonymous credit card applications.
[Figure]
i f i c a t i o n
A cc u r a c y
Vehicle Silhouette
[Figure]
We measure the best observed classification accuracy.
Notably, these tasks have a large amountof noise, which causes the performance to fluctuate substantially between repetitions.
We observethat
JES performs almost identically to MES and EI on vehicle silhouette and credit application, andsubstantially better on blood transfusion.
PES lags behind significantly on all tasks.
Conclusions
We have presented Joint Entropy Search, an information-theoretic acquisition function that considersan entirely new quantity, namely the joint density over the optimum and optimal value.
By utilizingthe entropy reduction from fantasized optimal observations, JES obtains a simple form for the entropyreduction regarding the joint distribution.
As such, the additional information considered comeswith minimal computational overhead, avoids restrictive assumptions on the objective, and yieldsstate-of-the-art performance along with superior decision-making.
We believe JES to be a new go-toacquisition function for BO, and to establish a new standard for subsequent information-theoretictechniques.
Limitations and Future Work
The main contribution of this paper is to provide a novel information-theoretic acquisition functionwhich, given a sufficiently accurate model, yields impressive results.
However, the non-myopic,speculative nature of information-theoretic approaches lend them to be susceptible to model misspecification, such as a poor choice of GP kernel or GP hyperparameters.
In our view, information-theoreticapproaches are possibly more susceptible to this issue than their myopic counterparts ( EI , UCB , TS ).
While we propose a remedy to stabilize and improve the acquisition function under model misspecification with the inverse γ -greedy approach, this technique only serves to discover misspecificationand adjust accordingly, but not to inherently fix the misspecification.
We believe misspecification canonly be remedied by altering the surrogate model.
It is thus very promising to combine advancedmodelling techniques with information-theoretic acquisition functions, as already done with theadditive GP approach utilized in conjunction with MES by Wang and Jegelka [45]; further promisingadditions would be to tackle heterogeneous noise and input warping as done by HEBO [8].
We also note that, since JES computes the entropy reduction from conditioning on the optimal pair,it relies on some level of noise in the objective.
A surrogate model with zero noise will result inan infinite information gain for every optimal pair, which (by utilizing some random tie-breakingstrategy) would make JES equivalent to TS.
However, if JES is to be used in a completely noiselesssetting, we argue that a small noise term should be added as a remedy.
As this is done by defaultin many prominent GP frameworks [12, 13, 43], we do not view this as a major limitation of ourapproach.
Nevertheless, improving upon this strategy would be interesting in future work.
For future work, we also envision work on the adaptation of JES to various different domains, suchas multi-fidelity [46] and multi-objective optimization [1], as well as the integration of user priorknowledge over the location of the optimum [19] to accelerate optimization.
A Broader impact
Our work proposes a novel acquisition function for Bayesian optimization.
The approach is foundational and does not have direct societal or ethical consequences.
However, JES will be used in thedevelopment of applications for a wide range of areas and thus indirectly contribute to their impacts onsociety.
As an algorithm that can be used for HPO, JES intends to cut resource expenditure associatedwith model training, while increasing their performance.
This can help reduce the environmentalfootprint of machine learning research.
B Experimental setup
Frameworks.
For all tasks and acquisition functions, we use the original PES implementation in
MATLAB by Hernández-Lobato et al.
[15], which uses the GPStuff [43] library.
The implementationoptimizes the acquisition function, and the posterior mean, by sampling a dense grid of points, anduses a gradient-based optimizer to further optimize the single best point.
For better accuracy, wesubstantially increased the number of grid points.
The JES implementation can be found at https://github.com/jointentropysearch/JointEntropySearch .
GP Sample tasks.
To generate the GP sample tasks, we use a random Fourier feature [28] withweights drawn from the spectral density of a squared exponential kernel.
For dimension-wise lengthscale θ d , output scale σ 2 , and noise variance σ 2 ϵ , the hyperparameters per task are shown in Table 1.
The range in the last column is a rough approximation of the magnitude of the output spanned byeach GP sample.
The length scales of the samples are gradually increased with each dimensionalityto maintain a reasonable level of difficulty for all tasks.
Since the optimal values for these tasks areunavailable, they are approximated through a dense random search, followed by local search on themost promising subset of points.
Synthetic test functions.
For the synthetic test functions, 100 sampled optimal pairs are usedfor each acquisition function.
GP hyperparameters are marginalized over for these tasks, so anequal number of optimal pairs are sampled for each hyperparameter set.
The hyperparameters arere-sampled on a fixed schedule throughout the run.
Naturally, the sampled maxima were updatedat each iteration.
Moreover, each test function was also given a fixed amount of noise.
Regret wascomputed not from the noisy observed value, but from the true, noiseless function value.
MLP classification tasks.
All the classification tasks have a substantial amount of noise.
Asnoiseless objective values are unavailable, we report the observed classification accuracy.
5 hyperparameters are available in HPOBench for these tasks.
However, one of them (number of layers, depth )is held fixed due to its small integer-valued domain and the lack of integer hyperparameter support inthe MATLAB framework.
As seen in Tab. 3, the two other integer-valued hyperparameters batchsize and width have orders of magnitude larger domains, and can therefore reasonably be treated as
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
g s i m p l e r e g r e t
[Figure]
continuous hyperparameters.
All tasks are optimized in the [0, 1] range, and are scaled, transformed,and rounded to the nearest integer in the objective function where necessary.
All non-fixed parametersare evaluated in log scale.
The three tasks evaluated are Australian , Blood-transfusion-service-center ,and Vehicle , with HPOBench task numbers
#146818, #10101, #53, respectively.
Compute resources.
All experiments are carried out on Intel Xeon Gold 6130 CPUs.
Eachrepetition is run on a single core.
In total, approximately 50 , 000 core hours are used for theexperiments in the main paper, and an additional 20 , 000 for the appendix.
C Ablation studies and model misspecification
We provide ablation studies for the hyperparameter controlling the ratio of inverse greedy selectionγ .
Moreover, we show how inverse γ -greediness improves inference regret by recognizing regionsbelieved to be optimal as poor.
Lastly, we display the robustness of JES to the noise variance σ 2 ϵ .
C.1 Ablation studies
We provide an ablation study on γ in terms of both the simple and inference regret of JES.
Whilesimple regret may be a more practically relevant metric, inference regret helps understand the abilityof the acquisition function to successfully locate the optimum.
Fig.
9 shows that γ > 0 improvessimple regret, which is to be expected from its occasional greedy selection.
However, as is shownin Fig.
10, a moderate fraction γ ∈ { 0 . 05 , 0 .
1 } also yields comparable or even improved inferenceregret on all tasks.
Notably, γ = 0 .
2 yields slightly worse performance on Hartmann (6D), but yieldsmarginally improved performance on Branin and Hartmann (3D).
As such, the inverse γ -greedyapproach not only improves performance in terms of simple regret, but yields improved inference aswell.
C.2 Model misspecification
As mentioned in Sec. 3.5, the performance of information-theoretic methods can suffer substantiallyfrom model misspecification.
In Fig.
11, we show for JES , PES and MES how the inverse γ -greedyapproach helps stabilize inference and improve inference regret, and yields substantially better simpleregret for all methods.
For Michalewicz (10D), we observe that the inference regret of MES gets
[Figure]
Hartmann
[Figure]
Hartmann-
[Figure]
substantially worse after iteration 150.
With the inverse γ -greedy approach, this issue is severelyreduced.
In Fig.
12, the corresponding simple regrets for MES deviate at iteration 150.
The samebehavior can be observed for JES on Levy (8D) around iteration 200.
Across all test functions in
Fig.
11 and Fig.
12, an inverse γ -greedy strategy yields comparable or improved inference regret,and strictly improved simple regret for all acquisition functions.
Moreover, we observe that MESis generally the top-performing acquisition function, implying that it is the most robust to modelmisspecification.
Using the inverse γ -greedy approach, the acquisition function verifies whether thebelief over the location of the optimum is correct under the current model hyperparameters.
If it isnot, it re-calibrates its belief.
C.3 Ablation study on σ 2 ϵ
For the noise variance ablation study, we once again consider the GP sample tasks.
We fix the GPnoise hyperparameter σ 2 ϵ to the correct value prior to the start of the experiment.
In Fig.
13, we showthat the performance of JES is robust with respect to the noise level, while the performance of MESand PES decrease more drastically as the level of noise increases.
[Figure]
[Figure]
D Dependence on the number of MC samples
We show in Fig.
14 the dependence of JES on the number of MC samples for the GP sample tasks.
JES displays a slight dependence on the number of GP samples, as initial performance improvesmarginally for larger number of samples.
This is more prominent for higher-dimensional tasks, wherea lower number of samples causes a substantially slower start.
This can be explained by the factthat a larger number of sampled optimal pairs are required to accurately model a higher-dimensionaldensity.
Notably, the final regret on the 12-D benchmark is better for lower numbers of samples, suchas JES -4.
One potential explanation for this is its inability to model the joint distribution accurately.
[Figure]
If all realized optimal pairs are close to the perceived optimum, JES is almost certainly going tosample there.
Moreover, since the information gain for each sample is relatively low for samples in awell-explored region, the information gain of a few samples in an unexplored region may outweightthe information gain of a much larger number of samples in a well-explored region.
For JES -4 and
JES -20, it is more likely that all of the sampled optimal pairs are close to the optimum in this manner,while there is still small positive density on other parts of the search space.
As such, it is possible that
JES -4 and JES -20 over-exploit slightly in the 12-dimensional benchmark.
E Approximation Quality
As stated in Sec. 3.4, we approximate the entropy of the posterior over observations conditioned onthe data and the optimal pair H [ p ( y |D ∪ ( x ∗ , f ∗ ) , x , f ∗ )] – the entropy of the sum of a Gaussian anda truncated Gaussian variable – by moment matching of the truncated distribution.
We now show thequality of this approximation, and what impact it can potentially have on point selection.
To do so,we utilize the results from Nguyen et al.
[27] regarding the density of p ( y |D , x , f ∗ ) .
We note thatthe approximation regards the truncation from knowing the optimal value f ∗ , which constitutes anadditional reduction in entropy after having conditioned the GP on the new observation as in Sec. 3.3.
As such, the approximation considers only a fraction of the total entropy reduction, as visualized in
Fig.
To establish the quality of the approximation, we compare our approach to approximating the entropyby MC.
Naturally, the MC approach is more computationally expensive than moment matching, butyields an asymptotically correct result.
In Fig.
15, we show for varying noise levels, expressed as thenoise variance ratio of the total variance, and truncation quantiles Φ − 1 ( α ) , the difference in entropybetween an MC and a moment matching approach.
For example, a truncation quantile of 10 − 2 meansthat the upper 99% of the density of the posterior distribution is removed as a result of truncation.
We see that the approximate entropy reduction from moment matching is consistently lower thanfor the asymptotically exact MC approach.
Moreover, the approximation error seemingly increaseslogarithmically with the truncation quantile.
As such, we can expect modest underestimates of theentropy reduction when we truncate the posterior to an extreme degree, and the level of noise islow.
In the right image of Fig 2, the posterior is severely truncated left of the conditioned location.
However, the noise variance constitutes a large fraction of the total variance at this location, whichmeans that the approximation is still accurate with respect to the true entropy reduction.
The scenariorepresented in the bottom left corner of the grid, where we severely truncate the posterior and thenoise variance is low, does not reasonably occur in practice.
The aforementioned scenario wouldentail sampling an optimal pair which is several orders of magnitudes worse than the mean of the GPat uncertain (in the sense that σ >> σ ϵ ) locations.
Lastly, the blue region in the upper left corner ofthe grid represents a region where we have less truncation, and the noise is a relatively insignificantpart of the total variance.
Fig.
15 shows that the entropy reduction from truncation in this regionis underestimated by approximately 15%.
As such, the approximation error leads to a slightly lessexplorative strategy than what a strategy with exact computation of the truncation term would provide.
[Figure]
[Figure] [Figure]
F Regret Measures
We display the regret measures for the GP sample and synthetic test functions.
F.1 GP sample tasks
In Fig.
[Figure]
we show the simple regret for the GP sample tasks.
We note that the simple and inferenceregrets for MES are approximately equal, while there is a substantial difference for JES .
For PES , thedifference between simple and inference regret is the most pronounced at approximately two order ofmagnitude at the most.
F.2 Synthetic test functions
Next, we show the inference regret for the synthetic test functions in Fig.
We note that he simpleand inference regrets for JES are approximately equal.
For PES , the difference between simple andinference regret is once again very pronounced.
Notably, the simple regret is significantly better thanthe inference regret for MES on Branin, implying that it does not yet have full knowledge of wherethe optimum is.
We once again note the numerical issues of PES on Branin.
Moreover, the inference
[Figure]
[Figure] [Figure] regret of MES gets marginally worse for Hartmann (6D) from approximately iteration 100 until theend of the run, implying that its knowledge about the location of the optimum gets worse [Figure]
G Additional MLP tuning experiments
We provide additional results on the MLP tuning tasks provided through HPOBench [9].
We observethat JES outperforms MES on all tasks, and is competitive with EI.
[Figure]