Generalizing Bayesian Optimization with Decision-theoretic Entropies
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Bayesian optimization (BO) is a popular method for efﬁciently inferring optima ofan expensive black-box function via a sequence of queries.
Existing informationtheoretic BO procedures aim to make queries that most reduce the uncertaintyabout optima, where the uncertainty is captured by Shannon entropy.
However,an optimal measure of uncertainty would, ideally, factor in how we intend to usethe inferred quantity in some downstream procedure.
In this paper, we insteadconsider a generalization of Shannon entropy from work in statistical decisiontheory [7, 22], which contains a broad class of uncertainty measures parameterizedby a problem-speciﬁc loss function corresponding to a downstream task.
Weﬁrst show that special cases of this entropy lead to popular acquisition functionsused in BO procedures such as knowledge gradient, expected improvement, andentropy search.
We then show how alternative choices for the loss yield a ﬂexiblefamily of acquisition functions that can be customized for use in novel optimizationsettings.
Additionally, we develop gradient-based methods to efﬁciently optimizeour proposed family of acquisition functions, and demonstrate strong empiricalperformance on a diverse set of sequential decision making tasks, including variantsof top-
k optimization, multi-level set estimation, and sequence search.
Introduction
Bayesian optimization (BO) is a popular method for efﬁcient global optimization of an expensiveblack-box function, which leverages a probabilistic model to judiciously choose a sequence offunction queries.
In BO, there are a few key paradigms that motivate existing methodologies.
Oneparadigm is decision-theoretic BO, which includes methods such as knowledge gradient [9] andexpected improvement [18, 15].
At each iteration of BO, these methods aim to make a query thatmaximally increases the expected value, under the posterior, of a ﬁnal estimate of the optima (sometimes referred to as a terminal action ).
Another common paradigm is based on maximal uncertaintyreduction and includes information-based BO methods such as the family of entropy search methods[13, 14, 27].
At each iteration of BO, these methods aim to make a query that most reduces theuncertainty, under the posterior, about a quantity of interest (such as the location of the optima).
In the uncertainty-reduction paradigm, the information-based methods have predominantly used
Shannon entropy as the measure of uncertainty.
While Shannon entropy is one measure of uncertaintythat we could aim to reduce at each iteration of BO, it is not the only measure, and it is not necessarilythe most ideal measure for every optimization task.
For instance, an optimal uncertainty functionwould, ideally, factor in how we intend to use the ﬁnal uncertainty about an inferred quantity in somedownstream procedure.
In this paper, we develop a framework that aims to ﬁrst unify and then extend these two paradigms.
Speciﬁcally, we adopt a generalized deﬁnition of entropy from past work in Bayesian decisiontheory [7, 22, 12], which proposes a family of decision-theoretic entropies parameterized by aproblem-speciﬁc loss function and action set.
This family includes Shannon entropy as a special case.
Using this generalized entropy, we can view information-based BO methods as instances of decisiontheoretic BO, with a terminal action chosen from a different type of action set.
Similarly, this framework also includes as special cases the decision-theoretic methods such as expected improvement andknowledge gradient, which yields an uncertainty-reduction view of these methods.
Beyond this uniﬁed view, our framework can be easily adapted to novel problem settings by choosing an appropriateloss and action set tailored to a given downstream use case.
This allows for handling new optimizationscenarios that have not previously been studied and where no BO procedure currently exists.
As an example, there are many real-world problems where we want to estimate a set of optimal points,rather than a single global optimum.
Use cases include when we wish to ﬁnd a set of highest-valuepoints subject to some constraints on the similarity between these points (e.g. to produce a diverseset of candidates in drug or materials design [21, 25, 26]), or points which satisfy some sequentialrelation (e.g. to construct a library of molecules that attains a sequence of desired measurements [8]).
Further, we may wish to estimate other properties of a black-box function, such as certain curves,surfaces, or subsets of the domain [33, 16, 23].
Due to the vast number of possibilities, most customproblem settings have not been explicitly studied in the literature.
A key advantage of our frameworkis that it provides a way to approach these problems where no suitable methods have been developed.
Additionally, since we deﬁne this family of generalized entropies in a standardized way, we candevelop a common acquisition optimization procedure, which applies generically to many membersof this family (where each member is induced by a speciﬁc loss function and action set).
In particular,we develop a fully differentiable acquisition optimization method inspired by recent work on one-shotknowledge gradient procedures [3].
This yields an effective and computationally efﬁcient algorithmfor many optimization and sequential decision making tasks, as long as the problem-speciﬁc lossfunction is differentiable.
In summary, our main contributions are the following:
• We propose an acquisition function based on a family of decision-theoretic entropies parameterized by a loss function (cid:96) and action set A .
Under certain choices of (cid:96) and A , we can viewmultiple BO acquisition functions in a single decision-theoretic perspective, which sheds lighton the settings for which each is best suited.•
By selecting a suitable (cid:96) and A , we can produce a problem-speciﬁc acquisition function, whichis tailored to a given downstream use case.
This yields a customizable BO method that can beapplied to new optimization problems and other sequential decision making tasks, where noapplicable methods currently exist.•
We develop an acquisition optimization procedure that applies generically to many instance ofour framework.
This procedure is computationally efﬁcient, using a gradient-based approach.• We demonstrate that our method shows strong empirical performance on a diverse set of tasksincluding top-
k optimization with diversity, multi-level set estimation, and sequence search.
Setup
Let f : X → Y ⊂ R denote an expensive black-box function that maps from an input search space
X to an output space Y , and f ∈ F .
We assume that we can evaluate f at an input x ∈ X , and will 76 observe a noisy function value y x
= f ( x ) + (cid:15) , where (cid:15)
[Equation]
We also assume that our uncertainty about f is captured by a probabilistic model with prior distributionp ( f ) , which reﬂects our prior beliefs about f .
Given a dataset of observed function evaluations
D t = { ( x i , y x i ) } t − 1 i =1 , our model gives a posterior distribution over F , denoted by p ( f |D t ) .
Suppose that, after a given BO procedure is complete, we intend to choose a terminal action a fromsome set of actions A , and then incur a loss based on both this action a and the function f .
We denotethis loss as (cid:96) :
F × A → R .
As one example, after the BO procedure, suppose we make a single 83 guess for the function maximizer, and then incur a loss based on the value of the function at thisguess.
In this case, the action set is A = X and the loss is (cid:96) ( f , a ) =
− f ( a ) .
Decision-theoretic Entropy Search
In this section, we ﬁrst describe a family of decision-theoretic entropies from past work in Bayesiandecision theory [7, 22, 12], which are parameterized by a problem-speciﬁc action set A and lossfunction (cid:96) .
This family includes Shannon entropy as a special case.
We denote this family usingthe symbol H (cid:96), A , and refer to it as the H (cid:96), A -entropy .
Deﬁnition 3.1.
( H (cid:96), A -entropy of f ).
Given a prior distribution p ( f ) on functions, and a dataset D ofobserved function evaluations, the posterior H (cid:96), A -entropy with loss (cid:96) and action set A is deﬁned as
[Equation] [Equation]
[Equation]
Intuitively, after expending our budget of function queries, suppose that we must make a terminalaction a ∗ ∈ A , where this action incurs a loss (cid:96) ( f, a ∗ ) deﬁned by the loss (cid:96) and function f .
Givena posterior p ( f |D ) that describes our belief about f after observing D , we take the terminal 95 action a ∗ to be the Bayes action , i.e. the action that minimizes the posterior expected loss,a ∗ = arg inf a ∈A E p ( f |D )
[ (cid:96) ( f, a )] .
The H (cid:96), A -entropy can then be viewed as the posterior expectedloss of the Bayes action.
We next describe how this generalizes Shannon entropy, and why it is areasonable deﬁnition for an uncertainty measure.
Example: Shannon entropy Let P ( F ) denote a set of probability distributions on a functionspace F , which we assume contains the posterior distribution p ( f | D ) ∈ P ( F ) .
Suppose, for the 101 H (cid:96), A -entropy, that we let the action set A = P ( F ) , and loss function (cid:96) ( f, a ) =
− log a ( f ) , fora ∈ P ( F ) .
Unlike the previous examples, note that the action set is now a set of distributions.
Then, the Bayes action will be a ∗ =
p ( f | D ) (this can be shown by writing out the deﬁnition of the
Bayes action as a cross entropy, see Appendix A.1), and thus
[Equation]
where H [ f | D ] = − (cid:82) p ( f | D ) log p ( f | D ) is the Shannon differential entropy.
Thus, the 106 H (cid:96), A -entropy using the above ( (cid:96), A ) is equal to the Shannon differential entropy.
Note that we have focused here on the Shannon entropy of the posterior over functions p ( f | D ) .
In
Section 4 we show how this example can be extended to the Shannon entropy of the posterior overproperties of f , such as the location (or values) of optima, which will provide a direct equivalence toentropy search methods in BO.
Why is this a reasonable measure of uncertainty?
The H (cid:96), A -entropy has been interpreted asa measurement of uncertainty in the literature because it satisﬁes a few intuitive properties.
First,similar to Shannon differential entropy, the H (cid:96), A -entropy is a concave uncertainty measure [7, 12].
Intuitively, if we have two distributions p 1 and p 2 , and ﬂip a coin to sample from p 1 or p 2 , then weshould have less uncertainty if we were told the outcome of the coin ﬂip than if we weren’t.
In otherwords, the average uncertainty of p 1 and p 2 (i.e. coin ﬂip outcome known ) should be less than theuncertainty of 0 .
5 p 1 +0 .
5 p 2 (coin ﬂip outcome unknown ).
Since H (cid:96), A is concave, it has this property.
As a consequence—also similar to Shannon differential entropy—the H (cid:96), A -entropy of the posterioris less than the H (cid:96), A -entropy of the prior, in expectation.
Intuitively, whenever we make additionalobservations (i.e. gain more information), the posterior entropy is expected to decrease.
Acquisition function We propose a family of acquisition functions for BO based on the H (cid:96), Aentropy, which are similar in structure to information-theoretic acquisition functions in the entropysearch family.
Like these, our acquisition function selects the query x t ∈ X that maximally reduces 124 the uncertainty, as characterized by the H (cid:96), A -entropy, in expectation.
We refer to this quantity as theexpected H (cid:96), A -information gain (EHIG).
Deﬁnition 3.2.
(Expected H (cid:96), A -information gain).
Given a prior p ( f ) on functions and a datasetof observed function evaluations D t , the expected H (cid:96), A -information gain (EHIG), with loss (cid:96) andaction set A , is deﬁned as
[Equation]
There are multiple beneﬁts to developing this acquisition function.
Though similar in form to entropysearch acquisition functions, the EHIG yields (based on the deﬁnition of H (cid:96), A ) the one-step Bayesoptimal query for the associated decision problem speciﬁed by the given loss (cid:96) and action set A .
Weprove in Section 4 that the EHIG casts both uncertainty-reduction and decision-theoretic acquisitionfunctions under a common umbrella, using different choices of (cid:96) and A ; this standardization providesguidance on which acquisition function is optimal for a given use case, based on details of theassociated terminal action.
More interestingly, in Section 5 we show how the EHIG allows us toderive problem-speciﬁc acquisition functions tailored to novel optimization and sequential decision making tasks.
And importantly, since we frame acquisition optimization of this family in a commonway—as a bilevel optimization problem over the sample space and action space—we can develop asingle acquisition optimization method that can generically apply to many custom tasks (Section 6).
In Algorithm 1, we present H (cid:96), A -E NTROPY S EARCH , our full Bayesian optimization procedure usingthe EHIG acquisition function.
This procedure takes as input a loss (cid:96) , action set A , and prior modelp ( f ) .
At each iteration, the procedure optimizes EHIG t ( x ; (cid:96), A ) to select a design x t ∈ X to query, 143 and then evaluates the black-box function on this design to observe an outcome y x t ∼ f
( x t )
+ (cid:15) .
In Section 6 we describe methods for optimizing the EHIG acquisition function via gradient-basedprocedures, which provide a computationally efﬁcient algorithm for many A and (cid:96) .
A Uniﬁed View of Information-based and Decision-theoretic Acquisitions
In this section, we aim to show how acquisition functions commonly used in BO are special casesof
the proposed EHIG family, for particular choices of (cid:96) and A .
This will allow us to view eachacquisition function (including information-based ones) from the perspective of a common decisionproblem: after the BO procedure is complete, we choose a terminal action from action set A andthen incur a loss deﬁned by (cid:96) .
Each acquisition function can be viewed as reducing the posterioruncertainty over f in a way that yields a terminal action with lowest expected loss.
This uniﬁed view provides two main beneﬁts.
First, it sheds light on the particular scenarios in whichone of the existing acquisition functions is optimal over the others (which we focus on in this section).
Second, it shows how using the EHIG with other choices for (cid:96) and A provides new acquisitionfunctions for a broader set of optimization scenarios and related tasks (which is the focus of Section 5).
Information-based acquisition functions We state the family of entropy search acquisitions function in a general way that includes the entropy search (ES) [13], predictive entropy search (PES)
[14],and max-value entropy search (MES) [27] algorithms.
Let θ f ∈ Θ denote a property of f we wouldlike to infer.
For example, we could set θ f = arg max x ∈X f ( x ) = x ∗ ∈ X , i.e. the location of theglobal maximizer of f , or θ f = max x ∈X f ( x ) ∈ R , i.e. the maximum value achieved by f in X .
This family of entropy search acquisition function can then be written as:
[Equation]
We can view this acquisition function as a special case of the EHIG in the following way.
Suppose,after the BO procedure is complete, we choose a distribution q from a set of distributions P (Θ) andthen incur a loss equal to the negative log-likelihood of q for the true value of θ f .
In this case, weview the action set as A = P (Θ) and the loss function as (cid:96) ( f, a ) =
− log a ( θ f ) , where a ∈ A .
To 167 visualize this, in the case where θ f
= x ∗ , see Figure
Under this choice, the H (cid:96), A -entropy of f will be equal to the Shannon entropy of θ f , and thus the
EHIG t will be equal to ES t .
We formalize this in the following proposition.
Proposition 1 .
If we choose A = P (Θ) and (cid:96) ( f, q ) =
− log q ( θ f ) , then the EHIG is equivalent tothe entropy search acquisition function, i.e. EHIG t ( x ; (cid:96), A ) =
ES t ( x ) .
Proof of Proposition 1.
See Appendix A.1.
Decision-theoretic acquisition functions We next describe how the EHIG generalizes decisiontheoretic acquisition functions such as knowledge gradient (KG) and expected improvement (EI).
Since these acquisition functions are often motivated from a perspective of a terminal decision, it isstraightforward to show how they are a special case of the EHIG .
However, the choice of A and (cid:96)here is insightful to review before extending EHIG to other scenarios.
[Figure] [Figure]
Knowledge Gradient
[Figure]
Expected Improvement
[Figure]
k -Guesses
First, the KG acquisition function can be written
[Equation]
where µ ∗ t = sup x
[Equation]
∈X E p ( f |D t )
[ f ( x (cid:48) )] is the max value of the posterior mean of f given data D t , andµ ∗ t +1 ( x, y x ) =
sup x (cid:48) ∈X E p ( f |D t ∪{ ( x,y x ) } )
[ f ( x (cid:48) )] is the max value of the posterior mean, given bothdata D t and observation ( x, y x ) .
Second, the EI acquisition function can be written
[Equation]
where f ∗ t = max { ˆ f ( x i ) } t − 1 i =1 , for x i ∈ D t and ˆ f ( x i ) is the posterior expected value of f at x i .
Thisdeﬁnition is equal to the standard formulation of EI in the noiseless setting (i.e. when y x = f ( x ) forqueried x ) and the plug-in formulation of EI in the noisy setting, when y x = f ( x ) + (cid:15) [20, 5].
To view these decision-theoretic acquisition functions as special cases of the EHIG, suppose that after
BO is complete, we aim to make a single guess x ∗
∈
A for the maximizer of f , and then incur a lossequal to the value of the function at x ∗ , i.e. (cid:96) ( f, x ∗ ) = − f ( x ∗ ) .
For KG, we let A =
X , in whichcase the Bayes action a ∗ is the maximizer of the posterior mean, and for EI, we let A
= { x i } t − 1 i =1 , inwhich case a ∗ is the best queried point.
We visualize these as gold vertical lines in Figure 1 (centerpanels).
In Appendix A, we prove this equivalence with EHIG t , for KG t and EI t .
We thus see two key differences here, in comparison with information-based BO: (i) the terminalaction a ∗ is a point estimate of the optimizer x ∗ rather than a distribution over X , and (ii) the lossdoes not depend on the particular value of the true optima x ∗ (nor on how accurately a ∗ provides anestimate of x ∗ ), but rather only depends on the function value of the terminal action, f ( a ∗ ) .
A Framework to Derive New Acquisition Functions for Custom Tasks
There are many real-world problems that go beyond simple black-box optimization, which havenot been explicitly studied in the literature, and for which there does not exist a suitable acquisitionfunction.
For these use cases, we can deﬁne an action set and loss based on details of the problem,and use the EHIG to provide a corresponding problem-speciﬁc acquisiton function.
As examplesof this, below we apply the EHIG to a number of relevant problems where, as far as we are aware,no corresponding acquisition function has been developed by prior work.
Illustrative example: k -guesses As a simple example to illustrate our framework, suppose, afteroptimization is complete, we are allowed to make a batch of k guesses for the function maximizer x ∗ ,and then recieve a ﬁnal reward based on the best guess.
This setting appears in settings where, after
BO is complete, we can make a batch of ﬁnal designs (e.g. synthesize a ﬁnal set of materials [25] ortrain a ﬁnal set of models [29, 24]), and only care about the single best design of the batch.
We canthus view the action set as A = X k , and loss as (cid:96) ( f, a ) = − max
( f ( a 1 ) , . . .
, f ( a k )) .
Figure 1 (far right) provides a visualization of this for k = 2 .
In this scenario, one of the Bayesactions ( a ∗ 2 ) is near the maximizer of the posterior mean (similar to KG), while the other ( a ∗ 1 ) isseparated from the ﬁrst.
Intuitively, to minimize the expected loss, the second guess should have botha high posterior mean, and also a low correlation to the ﬁrst guess—the ﬁrst guess has a better chanceof a low loss, but in cases where it fails, we want the second guess to not fail (i.e. not match the ﬁrstguess), while also achieving a low loss.
In practice, this yields an EHIG acquisition function that hasa similar but distinct exploration strategy from KG.
It spends a small portion of the budget on queriesthat give some information about not just the ﬁrst guess, but the other k - 1 guesses as well.
Top- k optimization with diversity Instead of a single optimal point in X , there are applicationswhere we wish to estimate a set of top- k optima, i.e. the subset of X of size k
that has the highest sumof values under f .
Examples of this can be found in materials discovery [17, 25], sensor networks[2, 11], and medicine [30].
Note that the goal of this problem is distinct from the k -guesses exampledescribed above.
When X is continuous, to avoid redundant solutions, we may wish to carry out thetask of top- k optimization with diversity , which aims to ﬁnd the top- k optima such that d ( x i , x j ) ≥ c ,∀ i, j ∈ { 1 , . .
. , k } , for a problem-speciﬁc distance d .
As one example of our EHIG framework,suppose that we choose A = X k (where a = ( a 1 , . .
. , a k ) ∈
A denotes a set of top- k points) and 225 incur the loss
[Equation]
Note that we can select the distance function d here to match the problem-speciﬁc constraint.
Intuitively speaking, this choice of ( (cid:96), A ) yields an EHIG acquisition function that makes a sequenceof queries which rotate focus between multiple diverse optimal points in the domain.
Multi-level set estimation The goal of level set estimation (LSE) is to estimate a subset of thedesign space X , where function values are larger than a given threshold c ,
S c = { x ∈ X : f ( x ) > c } .
This task appears in a number of applications, including catalyst design [33], interactive learning [4],and environmental monitoring [23].
While many prior works have studied standard LSE, here weconsider the task of multi-level set estimation (MLSE), where we are given m thresholds satisfyingc 1 < . . .
<
c m and want to estimate m + 1 sets: S i = { x ∈ X : c i < f ( x ) <
c i +1 } for 235 i = { 0 , . . .
, m } (where c 0 :=
−∞ and c m +1 := + ∞ ).
This is useful in the above applications when 236 we have more than one threshold of interest—for example, public health policy makers must estimateregions where disease prevalence exceeds 1%, 2%, etc., for graded reopening decisions [19, 31].
As one approach to MLSE using the EHIG, we focus on settings with a discrete set of designpoints X 0 ⊂ X , |X
| =
J .
Suppose, after querying is complete, we must choose a set of values 240 a ∈ A =
[0 , 1] J × m (one for each x ∈ X
and i ∈ { 0 , . . .
, m } ), which represent level set identity 241 variables.
Suppose we then incur a loss with the following form, that depends on these identityvariables a , as well as on a ﬂexible relation r ( f ( x ) , c i ) between function values f ( x ) and thresholdsc i , i.e.
[Equation]
For instance, if r ( f ( x ) , c i ) =
f
( x ) − c i , then the optimal a i ( x ) should specify the c i -super levelset for each i ∈ { 1 , . . .
, m } , i.e. a i ( x ) = 1 for each x ∈ X 0 with f ( x ) >
c i and a i ( x ) = 0 246 otherwise.
This example loss yields an acquisition function that, empirically, focuses samples aroundthe boundaries of multiple level sets of a black-box function simultaneously.
Sequence search We deﬁne sequence search as the task of estimating a sequence of inputs( x 1 , . . .
, x m ) ∈
X m with outputs values matching a set of problem-speciﬁc criteria.
For example, we may wish to estimate a sequence of inputs corresponding to a predeﬁned set of functionvalues ( y (cid:126) 1 , . . .
, y (cid:126) m ) .
This ﬁnds applications in materials science, such as in the task of synthesizinga nanoparticle library [8] (i.e. ﬁnding a set of input conditions that yield a set of nanoparticles ofdifferent pre-deﬁned sizes).
As another example, in the context of public health, we may be interestedin a set of locations where vaccination rates equal some pre-speciﬁed values (e.g. (20% , . . .
, 80%) )when making decisions involving vaccine allocations, as we describe in Section 7.
As an example ofthese applications in our EHIG framework, we might have the action set A = X m and loss
[Equation]
These examples all aim to show that the EHIG can be used to deﬁne a problem-speciﬁc acquisitionfunction, which can be tailored to the details of a particular use case.
As a result, when used in
Algorithm 1, we gain a customizable optimization framework that can be applied to a variety of novelproblem settings with special-purpose losses.
Gradient-based Acquisition Optimization
At each iteration of H (cid:96), A -E NTROPY S EARCH (Algorithm 1), we optimize the acquisition function toselect the next query x t = arg max x ∈X EHIG t ( x ; (cid:96), A ) .
Classically, zeroth order optimization routin- ues have been used for acquisition optimization in BO.
However, recent work has developed gradientbased methods for optimizing certain acquisition functions [28, 3], which can allow for efﬁcient acquisition optimization over X .
We work on similar methodology here—namely, we develop a gradientbased acquisition optimization procedure for appropriate settings (i.e. assuming continuous X and A ,and certain conditions on (cid:96) ).
We can, for example, apply this gradient-based optimization to each ofthe acquisition functions described in Section 5, for which we show experimental results in Section 7.
Similar to related work [28, 3], we give the following derivation with a focus on Gaussian process(GP) models, though the methodology can be extended to other models in which we can apply thereparameterization procedure described below to differentiate through posterior model parameters.
Differentiable loss function We ﬁrst describe a few assumptions that must be satisﬁed to carryout the gradient-based optimization procedure.
Denote the posterior expected loss given D by
L ( D , a ) :=
E p ( f |D )
[ (cid:96) ( f, a )] .
We assume that this loss function depends only on the functionvalue of f at a ﬁnite number of points, i.e. there exists functions x 1 ( a ) , · · · , x K ( a ) , and a function(cid:96) (cid:48) : R K ×
A → R , for K ∈ N , such that
[Equation]
This requirement is satisﬁed by the loss functions in Section 5.
For brevity, denote the sequencex 1 ( a ) , · · · , x K ( a ) by x 1: K ( a ) and f ( x 1 ( a )) , · · · , f ( x K ( a )) by f ( x 1: K ( a )) .
We assume that the func- 280 tions x k and (cid:96) (cid:48) are differentiable with respect to all arguments.
Given a dataset D and GP prior, theposterior distribution of f ( x K ( a )) is also Gaussian.
In particular, there exist functions
[Equation]
and
[Equation]
such that f ( x 1: K ( a )) =
µ ( x 1: K ( a ); D ) + U ( x 1: K ( a ); D ) (cid:15) where (cid:15) is drawn from a K -dimensionalstandard normal distribution.
We can combine the above results to get
[Equation]
A key property is that we can compute unbiased gradients of this with respect to both D and a , as
[Equation]
Differentiable acquisition function For a given input x ∈ X , let y ( x, D ) denote the posterior 286 predictive distribution of our model.
Note that there exists a deterministic function ¯ y ( x,
D , λ ) such 287 that y ( x,
D ) =
¯ y ( x, D , λ ) , where λ is drawn from a standard normal distribution.
Hence, if (cid:96) satisﬁes
Eq.
(7), then we can optimize EHIG t with gradient descent.
In particular, we can write
[Equation] [Equation]
where in Eq.
(8), to avoid clutter, we use the shorthand ˆ µ ( x, a ( λ )) :
= µ ( x 1: K ( a ( λ )); D ∪ ¯ y ( x, D , λ )) ,and ˆ U ( x, a ( λ )) :
= U ( x 1: K ( a ( λ )); D ∪ ¯ y ( x, D , λ )) .
Importantly, we can compute the unbiased gradient of the quantity E λ,(cid:15)
[ (cid:96) (cid:48)
(ˆ µ ( x, a ( λ ))
+ ˆ U ( x, a ( λ ))
(cid:15), a ( λ ))] .
In practice, we can also take gradientsof a Monte Carlo estimate of Eq.
(8) [3], by ﬁxing samples of λ, (cid:15) throughout the optimization.
Speciﬁcally, we can sample λ 1 , · · · , λ M and (cid:15) 1 , · · · , (cid:15) N and approximate Eq.
(8) via
[Equation]
where we use a m = a ( λ m ) for brevity.
Under the assumptions above, we can compute the unbiasedgradient of this quantity, and by using systems such as GPyTorch [10] and BoTorch [3] we cancompute this gradient efﬁciently via automatic differentiation.
Experiments
We evaluate our proposed method on the example tasks described in Section 5: top- k optimizationwith diversity, multi-level set estimation, and sequence search.
For these applications, we showcomparisons against a set of baselines on real and synthetic black-box functions.
Comparison methods.
In our experiments, we compare the following set of acquisition strategies:
• H (cid:96), A -E NTROPY S EARCH (HES)
We follow Algorithm 1, using the loss and action set for eachtask as described in Section 5, and the gradient-based procedure outlined in Section 6.
40 Iteration 13 .14 .14 .14 .14 .
N e ga t i v e
L o ss
Alpine-3, k =
RandomSearchKnowledgeGradientUncertaintySampling
H ‘, A -EntropySearch     50 Iteration 15 .
N e ga t i v e
L o ss
Alpine-5, k =      60 Iteration 8 .9 .9 .9 .
N e ga t i v e
L o ss
Vaccination, k =     10 x 1     x
RandomSearch     10 x 1     x
UncertaintySampling     10 x 1     x
KnowledgeGradient     10 x 1     x
H ‘, A -EntropySearch
• R ANDOM S
EARCH (RS).
At each iteration, we draw a sample x t uniformly at random from X .
• U NCERTAINTY S AMPLING (US).
At each iteration, we select the point that maximizes theposterior predictive variance, i.e. x t = arg max x ∈X Var[ p ( y x | D t )] .•
K NOWLEDGE G RADIENT (KG).
We show KG as a representative method for standard BO.
KGallows us to carry out a similar gradient-based procedure as in HES.
• P ROBABILITY OF M ISCLASSIFICATION (POM).
This is a common acquisition function for levelset estimation [6].
We predict whether a point is above a threshold, represented by a binary variablez , and select the design with maximal label uncertainty x t = arg min x ∈X max z ∈{ 0 , 1 } p ( z | x ) .
Note that we are restricted to comparing against relatively general-purpose baseline methods, asmore-speciﬁc acquisition functions have not previously been developed for the tasks below.
Top- k Optimization with Diversity In our ﬁrst task, the goal is to ﬁnd a set of k diverse elementsin X , each with a high value of f .
To assess each method, at each iteration we record the negativeloss − (cid:96) ( f , a ∗ ) using Eq.
(4)—i.e.
the negative top- k with diversity loss of the Bayes action a ∗ =arg inf a ∈A E p ( f |D t )
[ (cid:96) ( f, a )] on the true function f —using the set of queries D t produced by the 318 given method.
Intuitively, if a method makes a set of queries that yield a good estimate of diversetop- k elements, it will score a high value under this metric.
In Figure 2 ( bottom row ) we visualize results on the multimodal Alpine- d benchmark function (seeappendix for details).
Here, we can see that HES concentrates queries over ﬁve local optima of thisfunction, while KG allocates a majority of samples on only the highest peak, and both US and RSdistribute their queries over the full domain X .
In Figure 2 ( top row ), we compare performance byplotting the negative loss versus iteration on two higher dimensional examples.
We also compareeach method on the Vaccination function (provided by [32]), which returns the vaccination rate forlocations in the continential United States, given an input (latitude, longitude) .
The goal of this taskis to efﬁciently ﬁnd a set of ﬁve diverse locations with highest vaccination rates.
We show results in
Figure 2 ( top row, right ), and see a similar advantage of HES over comparison methods.
Multi-level Set Estimation
In our second task, the goal is to carry out multi-level set estimation.
Here, we can assess each method using a more conventional metric: we produce an estimateof the level for every x ∈ X 0 , using the model’s posterior mean (given the queries selected bya particular method), and then record the accuracy of this estimate averaged across all level setthresholds.
Intuitively, a method will achieve a higher accuracy if it chooses queries that yield abetter estimate of the function near the threshold boundaries of the level sets.
In Figure 3 ( bottomrow ), we visualize results for a two-level set task on the Multihills function, deﬁned as a mixture density (details given in appendix).
We see that HES concentrates queries along both of theboundaries, which are drawn as blue and red dashed lines.
In the top row , we compare the performance of all methods, showing the accuracy vs. iteration.
Here, the Pennsylvania Night Lightfunction [1] released by NASA (additional details in the appendix), returns the relative level oflight at a location in Pennsylvania, as queried by a satellite image.
The goal of this experiment
[Figure]
A cc u r a c y
Pennsylvania Night Light
RandomSearchUncertaintySamplingKnowledgeGradient
H ‘, A -EntropySearch ProbabilityofMisclassiﬁcation     100 Iteration 0 .0 .0 .0 .1 .
A cc u r a c y
Alpine10 15 20 25 30 35 40 Iteration 0 .0 .0 .1 .
A cc u r a c y
Multihills 0 . 0 0 .
[Figure]
[Figure] [Figure]
x
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
UncertaintySampling 0 . 0 0 .
[Figure] [Figure]
[Figure]
8 1 . 0 x 1 0 .0 .0 .0 .0 .1 .x
KnowledgeGradient 0 . 0 0 .
8 1 . 0 x 1 0 .0 .0 .0 .0 .1 .x
H ‘, A -EntropySearch is to determine the portion of land at which night light is above a speciﬁed threshold value.
In
Appendix B, we show additional experiments, including a visualization of results on this function.
[Figure]
344 Sequence Search
In our third task, the goalis to ﬁnd a sequence of elements whose valueunder the black-box function matches a set ofpre-speciﬁed function values ( y (cid:126) 1 , . . .
, y (cid:126) m ) .
Toassess each method, at each iteration we recordthe negative loss − (cid:96) ( f , a ∗ ) from Eq.
(6)—i.e.the negative sequence search loss of the Bayes
[Equation] [Equation]
using the set of queries D t produced by 353 the given method.
Intuitively, if a methodmakes a set of queries that yield a good estimate of a sequence of ( x (cid:126) 1 , . .
, x (cid:126) m ) such that( f ( x (cid:126) 1 ) , . . .
, f ( x (cid:126) m ))
≈ ( y (cid:126) 1 , . . .
, y (cid:126) m ) , it willscore a high value on this metric.
In Figure 4 ( right ) we visualize results on the Vaccination function (described above).
Here, ourgoal is to ﬁnd a sequence of ﬁve (latitude, longitude) coordinates with vaccination rates equal to( y (cid:126) 1 , . . .
, y (cid:126) m ) =
Estimates of locations that match these functionvalues can be useful when making policy decisions involving a vaccine response or allocation.
In thiscase, we see that HES concentrates queries along a route from the relatively highly vaccinated regionin the East to the relatively lowly vaccinated region in the North.
The left plots in Figure 4 provides aquantitive comparison of methods on the Vaccination function (results on additional functions areshown in the appendix), plotting the negative loss vs. iteration.
These again show the beneﬁts ofquery selection performed by HES relative to the comparison strategies.
Conclusion
In this paper, we take a decision making perspective on information-based acquisition functions: afterquerying is complete, we assume that we must make some decision a ∗ and then incur a loss (cid:96) ( f , a ∗ ) .
Our goal is thus to make a sequence of queries that reduce the uncertainty of the posterior distributionp ( f | D t ) in a way to best help make this decision with low loss.
Using H (cid:96), A -entropy [7, 22], wecan deﬁne an EHIG acquisition function which carries this out directly: it selects a point that isexpected to maximally reduce the posterior expected loss of the Bayes action a ∗ .
We incorporate thisacquisition function into a procedure called H (cid:96), A -E NTROPY S EARCH , and show, in many cases, thatwe can perform efﬁcient gradient-based optimization of this acquisition function.
There are multiple interesting avenues for future work.
First, we hope to develop acquisitionoptimization methods for additional settings, such as for non-continuous action sets A or designspaces X .
One interesting avenue is hybrid optimization settings, where we can only take gradientsteps with respect to either the design or action variables.
We also wish to study how the proposed
EHIG framework could be applied to solve various problems in scientiﬁc discovery.
NightLights .
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4and appendix Section A.
(b) Did you include complete proofs of all theoretical results?
[Yes] See appendixSection A.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] All code and
instructions are included in supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] All training details are speciﬁed in the paper and included
code.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 7.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See appendix B.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] All existing assetswere cited.
(b) Did you mention the license of the assets?
[Yes] Licences of all assets are properly
attributed.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
New assets are included in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]