Generalizing Bayesian Optimization with Decision-theoretic Entropies
Abstract
Bayesian optimization (BO) is a popular method for efﬁciently inferring optima of an expensive black-box function via a sequence of queries.
Existing informationtheoretic BO procedures aim to make queries that most reduce the uncertainty about optima, where the uncertainty is captured by Shannon entropy.
However, an optimal measure of uncertainty would, ideally, factor in how we intend to use the inferred quantity in some downstream procedure.
In this paper, we instead consider a generalization of Shannon entropy from work in statistical decision theory [13, 39], which contains a broad class of uncertainty measures parameterized by a problem-speciﬁc loss function corresponding to a downstream task.
We ﬁrst show that special cases of this entropy lead to popular acquisition functions used in BO procedures such as knowledge gradient, expected improvement, and entropy search.
We then show how alternative choices for the loss yield a ﬂexible family of acquisition functions that can be customized for use in novel optimization settings.
Additionally, we develop gradient-based methods to efﬁciently optimize our proposed family of acquisition functions, and demonstrate strong empirical performance on a diverse set of sequential decision making tasks, including variants of top- k optimization, multi-level set estimation, and sequence search 2 .
Introduction
Bayesian optimization (BO) is a popular method for efﬁcient global optimization of an expensive black-box function, which leverages a probabilistic model to judiciously choose a sequence of function queries.
In BO, there are a few key paradigms that motivate existing methodologies.
One paradigm is decision-theoretic BO, which includes methods such as knowledge gradient [16] and expected improvement [33, 25].
At each iteration of BO, these methods aim to make a query that maximally increases the expected value, under the posterior, of a ﬁnal estimate of the optima (sometimes referred to as a terminal action ).
Another common paradigm is based on maximal uncertainty reduction and includes information-based BO methods such as the family of entropy search methods [22, 24, 47, 35].
At each iteration of BO, these methods aim to make a query that most reduces the uncertainty, under the posterior, about a quantity of interest (such as the location of the optima).
In the uncertainty-reduction paradigm, the information-based methods have predominantly used
Shannon entropy as the measure of uncertainty.
While Shannon entropy is one measure of uncertainty that we could aim to reduce at each iteration of BO, it is not the only measure, and it is not necessarily the most ideal measure for every optimization task.
For instance, an optimal uncertainty function would, ideally, factor in how we intend to use the ﬁnal uncertainty about an inferred quantity in some downstream procedure.
In this paper, we develop a framework that aims to ﬁrst unify and then extend these two paradigms.
Speciﬁcally, we adopt a generalized deﬁnition of entropy from past work in Bayesian decision theory [13, 39, 21], which proposes a family of decision-theoretic entropies parameterized by a problem-speciﬁc loss function and action set.
This family includes Shannon entropy as a special case.
Using this generalized entropy, we can view information-based BO methods as instances of decisiontheoretic BO, with a terminal action chosen from a different type of action set.
Similarly, this framework also includes as special cases the decision-theoretic methods such as expected improvement and knowledge gradient, which yields an uncertainty-reduction view of these methods.
Beyond this uniﬁed view, our framework can be easily adapted to novel problem settings by choosing an appropriate loss and action set tailored to a given downstream use case.
This allows for handling new optimization scenarios that have not previously been studied and where no BO procedure currently exists.
As an example, there are many real-world problems where we want to estimate a set of optimal points, rather than a single global optimum.
Use cases include when we wish to ﬁnd a set of highest-value points subject to some constraints on the similarity between these points (e.g. to produce a diverse set of candidates in drug or materials design [38, 44, 45]), or points which satisfy some sequential relation (e.g. to construct a library of molecules that attains a sequence of desired measurements [15]).
Further, we may wish to estimate other properties of a black-box function, such as certain curves, surfaces, or subsets of the domain [54, 28, 40].
Due to the vast number of possibilities, most custom problem settings have not been explicitly studied in the literature.
A key advantage of our framework is that it provides a way to approach these problems where no suitable methods have been developed.
Additionally, since we deﬁne this family of generalized entropies in a standardized way, we can develop a common acquisition optimization procedure, which applies generically to many members of this family (where each member is induced by a speciﬁc loss function and action set).
In particular, we develop a fully differentiable acquisition optimization method inspired by recent work on one-shot knowledge gradient procedures [4].
This yields an effective and computationally efﬁcient algorithm for many optimization and sequential decision making tasks, as long as the problem-speciﬁc loss function is differentiable.
In summary, our main contributions are the following:
• We propose an acquisition function based on a family of decision-theoretic entropies parameterized by a loss function (cid:96) and action set A .
Under certain choices of (cid:96) and A , we can view multiple BO acquisition functions in a single decision-theoretic perspective, which sheds light on the settings for which each is best suited.
• By selecting a suitable (cid:96) and A , we can produce a problem-speciﬁc acquisition function, which is tailored to a given downstream use case.
This yields a customizable BO method that can be applied to new optimization problems and other sequential decision making tasks, where no applicable methods currently exist.
• We develop an acquisition optimization procedure that applies generically to many instance of our framework.
This procedure is computationally efﬁcient, using a gradient-based approach. • We demonstrate that our method shows strong empirical performance on a diverse set of tasks including top- k optimization with diversity, multi-level set estimation, and sequence search.
Setup
Let f : X → Y ⊂ R denote an expensive black-box function that maps from an input search space
X to an output space Y , and f ∈ F .
We assume that we can evaluate f at an input x ∈ X , and will observe a noisy function value y x
= f ( x ) + (cid:15) , where (cid:15)
[Equation]
We also assume that our uncertainty about f is captured by a probabilistic model with prior distribution p ( f ) , which reﬂects our prior beliefs about f .
Given a dataset of observed function evaluations
D t = { ( x i , y x i ) } t − 1 i =1 , our model gives a posterior distribution over F , denoted by p ( f |D t ) .
Suppose that, after a given BO procedure is complete, we intend to choose a terminal action a from some set of actions A , and then incur a loss based on both this action a and the function f .
We denote this loss as (cid:96) :
F × A → R .
As one example, after the BO procedure, suppose we make a single guess for the function maximizer, and then incur a loss based on the value of the function at this guess.
In this case, the action set is A = X and the loss is (cid:96) ( f , a ) =
− f ( a ) .
Decision-theoretic Entropy Search
In this section, we ﬁrst describe a family of decision-theoretic entropies from past work in Bayesian decision theory [13, 39, 21], which are parameterized by a problem-speciﬁc action set A and loss function (cid:96) .
This family includes Shannon entropy as a special case.
We denote this family using the symbol H (cid:96), A , and refer to it as the H (cid:96), A -entropy .
Deﬁnition 3.1.
( H (cid:96), A -entropy of f ).
Given a prior distribution p ( f ) on functions, and a dataset D of observed function evaluations, the posterior H (cid:96), A -entropy with loss (cid:96) and action set A is deﬁned as
[Equation]
Intuitively, after expending our budget of function queries, suppose that we must make a terminal action a ∗ ∈ A , where this action incurs a loss (cid:96) ( f, a ∗ ) deﬁned by the loss (cid:96) and function f .
Given a posterior p ( f |D ) that describes our belief about f after observing D , we take the terminal action a ∗ to be the Bayes action , i.e. the action that minimizes the posterior expected loss, a ∗ = arg inf a ∈A E p ( f |D )
[ (cid:96) ( f, a )] .
The H (cid:96), A -entropy can then be viewed as the posterior expected loss of the Bayes action.
We next describe how this generalizes Shannon entropy, and why it is a reasonable deﬁnition for an uncertainty measure.
Example: Shannon entropy Let P ( F ) denote a set of probability distributions on a function space F , which we assume contains the posterior distribution p ( f | D ) ∈ P ( F ) .
Suppose, for the H (cid:96), A -entropy, that we let the action set A = P ( F ) , and loss function (cid:96) ( f, a ) =
− log a ( f ) , for a ∈ P ( F ) .
Unlike the previous examples, note that the action set is now a set of distributions.
Then, the Bayes action will be a ∗ =
p ( f | D ) (this can be shown by writing out the deﬁnition of the
Bayes action as a cross entropy, see Appendix A.1), and thus
[Equation]
where H [ f | D ] = − (cid:82) p ( f | D ) log p ( f | D ) is the Shannon differential entropy.
Thus, the H (cid:96), A -entropy using the above ( (cid:96), A ) is equal to the Shannon differential entropy.
Note that we have focused here on the Shannon entropy of the posterior over functions p ( f | D ) .
In
Section 4 we show how this example can be extended to the Shannon entropy of the posterior over properties of f , such as the location (or values) of optima, which will provide a direct equivalence to entropy search methods in BO.
Why is this a reasonable measure of uncertainty?
The H (cid:96), A -entropy has been interpreted as a measurement of uncertainty in the literature because it satisﬁes a few intuitive properties.
First, similar to Shannon differential entropy, the H (cid:96), A -entropy is a concave uncertainty measure [13, 21].
Intuitively, if we have two distributions p 1 and p 2 , and ﬂip a coin to sample from p 1 or p 2 , then we should have less uncertainty if we were told the outcome of the coin ﬂip than if we weren’t.
In other words, the average uncertainty of p 1 and p 2 (i.e. coin ﬂip outcome known ) should be less than the uncertainty of 0 .
5 p 1 +0 .
5 p 2 (coin ﬂip outcome unknown ).
Since H (cid:96), A is concave, it has this property.
As a consequence—also similar to Shannon differential entropy—the H (cid:96), A -entropy of the posterior is less than the H (cid:96), A -entropy of the prior, in expectation.
Intuitively, whenever we make additional observations (i.e. gain more information), the posterior entropy is expected to decrease.
Acquisition function We propose a family of acquisition functions for BO based on the H (cid:96), Aentropy, which are similar in structure to information-theoretic acquisition functions in the entropy search family.
Like these, our acquisition function selects the query x t ∈ X that maximally reduces the uncertainty, as characterized by the H (cid:96), A -entropy, in expectation.
We refer to this quantity as the expected H (cid:96), A -information gain (EHIG).
Deﬁnition 3.2.
(Expected H (cid:96), A -information gain).
Given a prior p ( f ) on functions and a dataset of observed function evaluations D t , the expected H (cid:96), A -information gain (EHIG), with loss (cid:96) and action set A , is deﬁned as
[Equation]
There are multiple beneﬁts to developing this acquisition function.
Though similar in form to entropy search acquisition functions, the EHIG yields (based on the deﬁnition of H (cid:96), A ) the one-step Bayes optimal query for the associated decision problem speciﬁed by the given loss (cid:96) and action set A .
We prove in Section 4 that the EHIG casts both uncertainty-reduction and decision-theoretic acquisition functions under a common umbrella, using different choices of (cid:96) and A ; this standardization provides guidance on which acquisition function is optimal for a given use case, based on details of the associated terminal action.
More interestingly, in Section 5 we show how the EHIG allows us to derive problem-speciﬁc acquisition functions tailored to novel optimization and sequential decision making tasks.
And importantly, since we frame acquisition optimization of this family in a common way—as a bilevel optimization problem over the sample space and action space—we can develop a single acquisition optimization method that can generically apply to many custom tasks (Section 6).
In Algorithm 1, we present H (cid:96), A -E NTROPY S EARCH , our full Bayesian optimization procedure using the EHIG acquisition function.
This procedure takes as input a loss (cid:96) , action set A , and prior model p ( f ) .
At each iteration, the procedure optimizes EHIG t ( x ; (cid:96), A ) to select a design x t ∈ X to query, and then evaluates the black-box function on this design to observe an outcome y x t ∼ f
( x t )
+ (cid:15) .
In Section 6 we describe methods for optimizing the EHIG acquisition function via gradient-based procedures, which provide a computationally efﬁcient algorithm for many A and (cid:96) . Algorithm ← x ∈X [Equation]
( x
: y [Equation] ∼ [Equation]
+ (cid:15)
D [Equation] ← D [Equation]
y [Equation] [Table] Output: distribution p ( f | D T +1 )
In this section, we aim to show how acquisition functions commonly used in BO are special cases of the proposed EHIG family, for particular choices of (cid:96) and A . This will allow us to view each acquisition function (including information-based ones) from the perspective of a common decision problem: after the BO procedure is complete, we choose a terminal action from action set A and then incur a loss deﬁned by (cid:96) .
Each acquisition function can be viewed as reducing the posterior uncertainty over f in a way that yields a terminal action with lowest expected loss.
This uniﬁed view provides two main beneﬁts.
First, it sheds light on the particular scenarios in which one of the existing acquisition functions is optimal over the others (which we focus on in this section).
Second, it shows how using the EHIG with other choices for (cid:96) and A provides new acquisition functions for a broader set of optimization scenarios and related tasks (which is the focus of Section 5).
Information-based acquisition functions We state the family of entropy search acquisitions function in a general way that includes the entropy search (ES) [22], predictive entropy search (PES)
[24], and max-value entropy search (MES) [47] algorithms.
Let θ f ∈ Θ denote a property of f we would like to infer.
For example, we could set θ f = arg max x ∈X f ( x ) = x ∗ ∈ X , i.e. the location of the global maximizer of f , or θ f = max x ∈X f ( x ) ∈ R , i.e. the maximum value achieved by f in X .
This family of entropy search acquisition function can then be written as:
[Equation]
We can view this acquisition function as a special case of the EHIG in the following way.
Suppose, after the BO procedure is complete, we choose a distribution q from a set of distributions P (Θ) and then incur a loss equal to the negative log-likelihood of q for the true value of θ f .
In this case, we view the action set as A = P (Θ) and the loss function as (cid:96) ( f, a ) =
− log a ( θ f ) , where a ∈ A .
To visualize this, in the case where θ f
= x ∗ , see Figure 1 (left), which shows the terminal action (gold density function) and corresponding loss (horizontal dashed line).
Under this choice, the H (cid:96), A -entropy of f will be equal to the Shannon entropy of θ f , and thus the
EHIG t will be equal to ES t .
We formalize this in the following proposition.
Proposition 1 .
If we choose A = P (Θ) and (cid:96) ( f, q ) =
− log q ( θ f ) , then the EHIG is equivalent to the entropy search acquisition function, i.e. EHIG t ( x ; (cid:96), A ) =
ES t ( x ) .
[Figure]
∗
[Figure] [Figure]
Knowledge Gradient
[Figure]
[Figure]
[Figure]
Expected Improvement
[Figure]
k -Guesses
Proof of Proposition 1.
See Appendix A.1.
Decision-theoretic acquisition functions We next describe how the EHIG generalizes decisiontheoretic acquisition functions such as knowledge gradient (KG) and expected improvement (EI).
Since these acquisition functions are often motivated from a perspective of a terminal decision, it is straightforward to show how they are a special case of the EHIG .
However, the choice of A and (cid:96) here is insightful to review before extending EHIG to other scenarios.
First, the KG acquisition function can be written
[Equation]
where µ ∗ t = sup x (cid:48) ∈X E p ( f |D t ) [ f ( x (cid:48) )] is the max value of the posterior mean of f given data D t , and µ ∗ t +1 ( x, y x ) = sup x
[Equation]
∈X E p ( f |D t ∪{ ( x,y x ) } )
[ f ( x (cid:48) )] is the max value of the posterior mean, given both data D t and observation ( x, y x ) .
Second, the EI acquisition function can be written
[Equation]
where f ∗ t = max { ˆ f ( x i ) } t − 1 i =1 , for x i ∈ D t and ˆ f ( x i ) is the posterior expected value of f at x i .
This deﬁnition is equal to the standard formulation of EI in the noiseless setting (i.e. when y x = f ( x ) for queried x ) and the plug-in formulation of EI in the noisy setting, when y x = f ( x ) + (cid:15) [37, 6].
To view these decision-theoretic acquisition functions as special cases of the EHIG, suppose that after
BO is complete, we aim to make a single guess x ∗
∈
A for the maximizer of f , and then incur a loss equal to the value of the function at x ∗ , i.e. (cid:96) ( f, x ∗ ) = − f ( x ∗ ) .
For KG, we let A =
X , in which case the Bayes action a ∗ is the maximizer of the posterior mean, and for EI, we let A
= { x i } t − 1 i =1 , in which case a ∗ is the best queried point.
We visualize these as gold vertical lines in Figure 1 (center panels).
In Appendix A, we prove this equivalence with EHIG t , for KG t and EI t . Additionally, in
Appendix A.4 we discuss connections to the probability of improvement (PI) acquisition function. We thus see two key differences here, in comparison with information-based BO: (i) the terminal action a ∗ is a point estimate of the optimizer x ∗ rather than a distribution over X , and (ii) the loss does not depend on the particular value of the true optima x ∗ (nor on how accurately a ∗ provides an estimate of x ∗ ), but rather only depends on the function value of the terminal action, f ( a ∗ ) .
A Framework to Derive New Acquisition Functions for Custom Tasks
There are many real-world problems that go beyond simple black-box optimization, which have not been explicitly studied in the literature, and for which there does not exist a suitable acquisition function.
For these use cases, we can deﬁne an action set and loss based on details of the problem, and use the EHIG to provide a corresponding problem-speciﬁc acquisiton function.
As examples of this, below we apply the EHIG to a number of relevant problems where, as far as we are aware, no corresponding acquisition function has been developed by prior work.
Illustrative example: k -guesses As a simple example to illustrate our framework, suppose, after optimization is complete, we are allowed to make a batch of k guesses for the function maximizer x ∗ , and then recieve a ﬁnal reward based on the best guess.
This setup appears in cases where, after BO is complete, we can make a batch of ﬁnal designs (e.g. synthesize a ﬁnal set of materials [44] or train a ﬁnal set of models [49, 41]), and only care about the single best design of the batch.
We can thus view the action set as A = X k , and loss as (cid:96) ( f, a ) = − max ( f ( a 1 ) , . . .
, f ( a k )) .
Figure 1 (far right) provides a visualization of this for k = 2 .
In this scenario, one of the Bayes actions ( a ∗ 2 ) is near the maximizer of the posterior mean (similar to KG), while the other ( a ∗ 1 ) is separated from the ﬁrst.
Intuitively, to minimize the expected loss, the second guess should have both a high posterior mean, and also a low correlation to the ﬁrst guess—the ﬁrst guess has a better chance of a low loss, but in cases where it fails, we want the second guess to not fail (i.e. not match the ﬁrst guess), while also achieving a low loss.
In practice, this yields an EHIG acquisition function that has a similar but distinct exploration strategy from KG.
It spends a small portion of the budget on queries that give some information about not just the ﬁrst guess, but the other k - 1 guesses as well.
Top- k optimization with diversity Instead of a single optimal point in X , there are applications where we wish to estimate a set of top- k optima, i.e. the subset of X of size k
that has the highest sum of values under f .
Examples of this can be found in materials discovery [30, 44], sensor networks [2, 18], and medicine [51].
Note that the goal of this problem is distinct from the k -guesses example described above.
When X is continuous, to avoid redundant solutions, we may wish to carry out the task of top- k optimization with diversity , which aims to ﬁnd the top- k optima such that d ( x i , x j ) ≥ c , ∀ i, j ∈ { 1 , . .
. , k } , for a problem-speciﬁc distance d .
As one example of our EHIG framework, suppose that we choose A = X k (where a = ( a 1 , . .
. , a k ) ∈
A denotes a set of top- k points) and incur the loss
[Equation]
Note that we can select the distance function d here to match the problem-speciﬁc constraint.
Intuitively speaking, this choice of ( (cid:96), A ) yields an EHIG acquisition function that makes a sequence of queries which rotate focus between multiple diverse optimal points in the domain.
Multi-level set estimation The goal of level set estimation (LSE) is to estimate a subset of the design space X , where function values are larger than a given threshold c ,
S c = { x ∈ X : f ( x ) > c } .[-
-]
This task appears in a number of applications, including catalyst design [54], interactive learning [5], and environmental monitoring [40].
While many prior works have studied standard LSE, here we consider the task of multi-level set estimation (MLSE), where we are given m thresholds satisfying c 1 < . . .
<
c m and want to estimate m + 1 sets: S i = { x ∈ X : c i < f ( x ) <
c i +1 } for i = { 0 , . . .
, m } (where c 0 :=
−∞ and c m +1 := + ∞ ).
This is useful in the above applications when we have more than one threshold of interest—for example, public health policy makers must estimate regions where disease prevalence exceeds 1%, 2%, etc., for graded reopening decisions [36, 52].
As one approach to MLSE using the EHIG, we focus on settings with a discrete set of design points X 0 ⊂ X , |X
| =
J [19, 26].
Suppose, after querying is complete, we must choose a set of values a ∈ A =
[0 , 1] J × m (one for each x ∈ X
and i ∈ { 0 , . . .
, m } ), which represent level set identity variables.
Suppose we then incur a loss with the following form, that depends on these identity variables a , as well as on a ﬂexible relation r ( f ( x ) , c i ) between function values f ( x ) and thresholds c i , i.e.
[Equation]
For instance, if r ( f ( x ) , c i ) =
f
( x ) − c i , then the optimal a i ( x ) should specify the c i -super level set for each i ∈ { 1 , . . .
, m } , i.e. a i ( x ) = 1 for each x ∈ X 0 with f ( x ) >
c i and a i ( x ) = 0 otherwise.
This example loss yields an acquisition function that, empirically, focuses samples around the boundaries of multiple level sets of a black-box function simultaneously.
Sequence search We deﬁne sequence search as the task of estimating a sequence of inputs ( x 1 , . . .
, x m ) ∈
X m with outputs values matching a set of problem-speciﬁc criteria.
For example, we may wish to estimate a sequence of inputs corresponding to a predeﬁned set of function values ( y (cid:126) 1 , . . .
, y (cid:126) m ) .
This ﬁnds applications in materials science, such as in the task of synthesizing a nanoparticle library [15] (i.e. ﬁnding a set of input conditions that yield a set of nanoparticles of different pre-deﬁned sizes).
As another example, in the context of public health, we may be interested in a set of locations where vaccination rates equal some pre-speciﬁed values (e.g. (20% , . . .
, 80%) ) when making decisions involving vaccine allocations, as we describe in Section 7.
As an example of these applications in our EHIG framework, we might have the action set A = X m and loss
[Equation]
These examples all aim to show that the EHIG can be used to deﬁne a problem-speciﬁc acquisition function, which can be tailored to the details of a particular use case.
As a result, when used in
Algorithm 1, we gain a customizable optimization framework that can be applied to a variety of novel problem settings with special-purpose losses.
Gradient-based Acquisition Optimization
At each iteration of H (cid:96), A -E NTROPY S EARCH (Algorithm 1), we optimize the acquisition function to select the next query x t = arg max x ∈X EHIG t ( x ; (cid:96), A ) .
Classically, zeroth order optimization routinues have been used for acquisition optimization in BO.
However, recent work has developed gradientbased methods for optimizing certain acquisition functions [48, 4], which can allow for efﬁcient acquisition optimization over X .
We work on similar methodology here—namely, we develop a gradientbased acquisition optimization procedure for appropriate settings (i.e. assuming continuous X and A , and certain conditions on (cid:96) ).
We can, for example, apply this gradient-based optimization to each of the acquisition functions described in Section 5, for which we show experimental results in Section 7.
Similar to related work [48, 4], we give the following derivation with a focus on Gaussian process (GP) models, though the methodology can be extended to other models in which we can apply the reparameterization procedure described below to differentiate through posterior model parameters.
Differentiable loss function We ﬁrst describe a few assumptions that must be satisﬁed to carry out the gradient-based optimization procedure.
Denote the posterior expected loss given D by
L ( D , a ) :=
E p ( f |D )
[ (cid:96) ( f, a )] .
We assume that this loss function depends only on the function value of f at a ﬁnite number of points, i.e. there exists functions x 1 ( a ) , · · · , x K ( a ) , and a function (cid:96) (cid:48) : R K ×
A → R , for K ∈ N , such that
[Equation]
This requirement is satisﬁed by the loss functions in Section 5.
For brevity, denote the sequence x 1 ( a ) , · · · , x K ( a ) by x 1: K ( a ) and f ( x 1 ( a )) , · · · , f ( x K ( a )) by f ( x 1: K ( a )) .
We assume that the func- tions x k and (cid:96) (cid:48) are differentiable with respect to all arguments.
Given a dataset D and GP prior, the posterior distribution of f ( x K ( a )) is also Gaussian.
In particular, there exist functions
[Equation]
such that f ( x 1: K ( a )) =
µ ( x 1: K ( a ); D ) + U ( x 1: K ( a ); D ) (cid:15) where (cid:15) is drawn from a K -dimensional standard normal distribution.
We can combine the above results to get
[Equation]
A key property is that we can compute unbiased gradients of this with respect to both D and a , as
[Equation]
Differentiable acquisition function For a given input x ∈ X , let y ( x, D ) denote the posterior predictive distribution of our model.
Note that there exists a deterministic function ¯ y ( x,
D , λ ) such that y ( x,
D ) =
¯ y ( x, D , λ ) , where λ is drawn from a standard normal distribution.
Hence, if (cid:96) satisﬁes
Eq.
(7), then we can optimize EHIG t with gradient descent.
In particular, we can write
[Equation]
where in Eq.
(8), to avoid clutter, we use the shorthand ˆ µ ( x, a ( λ )) :
= µ ( x 1: K ( a ( λ )); D ∪ ¯ y ( x, D , λ )) , and ˆ U ( x, a ( λ )) :
= U ( x 1: K ( a ( λ )); D ∪ ¯ y ( x, D , λ )) .
Importantly, we can compute the unbiased gradient of the quantity E λ,(cid:15)
[ (cid:96) (cid:48)
(ˆ µ ( x, a ( λ ))
+ ˆ U ( x, a ( λ ))
(cid:15), a ( λ ))] .
In practice, we can also take gradients of a Monte Carlo estimate of Eq.
(8) [4], by ﬁxing samples of λ, (cid:15) throughout the optimization.
Speciﬁcally, we can sample λ 1 , · · · , λ M and (cid:15) 1 , · · · , (cid:15) N and approximate Eq.
(8) via
[Equation]
where we use a m = a ( λ m ) for brevity.
Under the assumptions above, we can compute the unbiased gradient of this quantity, and by using systems such as GPyTorch [17] and BoTorch [4] we can compute this gradient efﬁciently via automatic differentiation.
40 Iteration 13 .14 .14 .14 .14 .
N e ga t i v e
L o ss
Alpine-3, k =
RandomSearchKnowledgeGradientUncertaintySampling
H ‘, A -EntropySearch     50 Iteration 15 .
N e ga t i v e
L o ss
Alpine-5, k =      60 Iteration 8 .9 .9 .9 .
N e ga t i v e
L o ss
Vaccination, k =     10 x 1     x
Random Search     10 x 1     x
Uncertainty Sampling     10 x 1     x
Knowledge Gradient     10 x 1     x
H ‘, A -Entropy Search
Experiments
We evaluate our proposed method on the example tasks described in Section 5: top-
k optimization with diversity, multi-level set estimation, and sequence search.
For these applications, we show comparisons against a set of baselines on real and synthetic black-box functions. Comparison methods. In our experiments, we compare the following set of acquisition strategies : • H (cid:96), A -E NTROPY S EARCH (HES).
We follow Algorithm 1, using the loss and action set for each task as described in Section 5, and the gradient-based procedure outlined in Section 6.
• R ANDOM S
EARCH (RS).
At each iteration, we draw a sample x t uniformly at random from X .
• U NCERTAINTY S AMPLING (US).
At each iteration, we select the point that maximizes the posterior predictive variance, i.e. x t = arg max x ∈X Var[ p ( y x | D t )] .
• K NOWLEDGE G RADIENT (KG).
We show KG as a representative method for standard BO.
KG allows us to carry out a similar gradient-based procedure as in HES.
• P ROBABILITY OF M ISCLASSIFICATION (POM).
This is a common acquisition function for level set estimation [7].
We predict whether a point is above a threshold, represented by a binary variable z , and select the design with maximal label uncertainty x t = arg min x ∈X max z ∈{ 0 , 1 } p ( z | x ) .
Note that we are restricted to comparing against relatively general-purpose baseline methods, as more-speciﬁc acquisition functions have not previously been developed for the tasks below.
Top- k Optimization with Diversity In our ﬁrst task, the goal is to ﬁnd a set of k diverse elements in X , each with a high value of f .
To assess each method, at each iteration we record the negative loss − (cid:96) ( f , a ∗ ) using Eq.
(4)—i.e.
the negative top- k with diversity loss of the Bayes action a ∗ = arg inf a ∈A E p ( f |D t )
[ (cid:96) ( f, a )] on the true function f —using the set of queries D t produced by the given method.
Intuitively, if a method makes a set of queries that yield a good estimate of diverse top- k elements, it will score a high value under this metric.
In Figure 2 ( bottom row ) we visualize results on the multimodal Alpine- d benchmark function (see appendix for details).
Here, we can see that HES concentrates queries over ﬁve local optima of this function, while KG allocates a majority of samples on only the highest peak, and both US and RS distribute their queries over the full domain X .
In Figure 2 ( top row ), we compare performance by plotting the negative loss versus iteration on two higher dimensional examples.
We also compare each method on the Vaccination function (provided by [53]), which returns the vaccination rate for locations in the continential United States, given an input (latitude, longitude) .
The goal of this task is to efﬁciently ﬁnd a set of ﬁve diverse locations with highest vaccination rates.
We show results in
Figure 2 ( top row, right ), and see a similar advantage of HES over comparison methods. [Figure] cc u r a c y
Pennsylvania
[Figure]
Light
RandomSearchUncertaintySamplingKnowledgeGradient H ‘, A -EntropySearch ProbabilityofMisclassiﬁcation [Figure] [Figure]
Iteration 0 .
[Figure]
[Figure]
[Figure]
cc u r a c y
Alpine-
[Figure]
[Figure]
cc u r a c y
Multihills
[Figure]
Random Search 0 .
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
Uncertainty Sampling 0 .0 .
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
x
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
Knowledge Gradient 0 .0 .0 .0 .0 .1 .
0 x 1 0 .
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
H ‘,
[Figure] Entropy Search Multi-level Set Estimation In our second task, the goal is to carry out multi-level set estimation.
Here, we can assess each method using a more conventional metric: we produce an estimate of the level for every x ∈ X 0 , using the model’s posterior mean (given the queries selected by a particular method), and then record the accuracy of this estimate averaged across all level set thresholds.
Intuitively, a method will achieve a higher accuracy if it chooses queries that yield a better estimate of the function near the threshold boundaries of the level sets.
In Figure 3 ( bottom row ), we visualize results for a two-level set task on the Multihills function, deﬁned as a mixture density (details given in appendix).
We see that HES concentrates queries along both of the boundaries, which are drawn as blue and red dashed lines.
In the top row , we compare the performance of all methods, showing the accuracy vs. iteration.
Here, the Pennsylvania Night Light function [1] released by NASA (additional details in the appendix), returns the relative level of light at a location in Pennsylvania, as queried by a satellite image.
The goal of this experiment is to determine the portion of land at which night light is above a speciﬁed threshold value.
In
Appendix B, we show additional experiments, including a visualization of results on this function.
[Figure]
Sequence Search
In our third task, the goal is to ﬁnd a sequence of elements whose value under the black-box function matches a set of pre-speciﬁed function values ( y (cid:126) 1 , . . .
, y (cid:126) m ) .
To assess each method, at each iteration we record the negative loss − (cid:96) ( f , a ∗ ) from Eq.
(6)—i.e. the negative sequence search loss of the Bayes action a
[Equation]
= arg
[Equation] [Equation]
f, a )] — using the set of queries D t produced by the given method.
Intuitively, if a method makes a set of queries that yield a good estimate of a sequence of ( x (cid:126) 1 , . . .
, x (cid:126) m ) such that ( f ( x (cid:126) 1 ) , . . .
, f ( x (cid:126) m ))
≈ ( y (cid:126) 1 , . . .
, y (cid:126) m ) , it will score a high value on this metric.
In Figure 4 ( right ) we visualize results on the Vaccination function (described above).
Here, our goal is to ﬁnd a sequence of ﬁve (latitude, longitude) coordinates with vaccination rates equal to ( y (cid:126) 1 , . . .
, y (cid:126) m ) =
Estimates of locations that match these function values can be useful when making policy decisions involving a vaccine response or allocation.
In this case, we see that HES concentrates queries along a route from the relatively highly vaccinated region in the East to the relatively lowly vaccinated region in the North.
The left plots in Figure 4 provides a quantitive comparison of methods on the Vaccination function (results on additional functions are shown in the appendix), plotting the negative loss vs. iteration.
These again show the beneﬁts of query selection performed by HES relative to the comparison strategies.
Conclusion
In this paper, we take a decision making perspective on information-based acquisition functions: after querying is complete, we assume that we must make some decision a ∗ and then incur a loss (cid:96) ( f , a ∗ ) .
Our goal is thus to make a sequence of queries that reduce the uncertainty of the posterior distribution p ( f | D t ) in a way to best help make this decision with low loss.
Using H (cid:96), A -entropy [13, 39], we can deﬁne an EHIG acquisition function which carries this out directly: it selects a point that is expected to maximally reduce the posterior expected loss of the Bayes action a ∗ .
We incorporate this acquisition function into a procedure called H (cid:96), A -E NTROPY S EARCH , and show, in many cases, that we can perform efﬁcient gradient-based optimization of this acquisition function.
There are multiple interesting avenues for future work.
First, we hope to develop acquisition optimization methods for additional settings, such as for non-continuous action sets A or design spaces X [12], and for functions with multidimensional outputs [23, 11].
One interesting avenue is hybrid optimization settings, where we can only take gradient steps with respect to either the design or action variables.
Another potential direction is to incorporate cost-aware Bayesian optimization techniques into the EHIG framework [29, 50, 3]. We also wish to study how the proposed EHIG framework could be applied in practice to solve various problems in the sciences, including experimental physics [14, 32, 8], drug discovery [43, 27, 20], and materials design [31, 46]. Finally, we wish to study in further detail how the EHIG acquisition function could be implemented for Bayesian decision making with other probabilistic models beyond Gaussian processes [42, 10, 9, 34]. Acknowledgments
We thank the anonymous reviewers, members of the Stanford SAIL community, and members of the CMU Auton Lab for helpful feedback on this paper. This work was supported by NSF (#1651565),
AFOSR (FA95501910024), ARO (W911NF-21-1-0125), CZ Biohub, and Sloan Fellowship. features/NightLights .
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sectionand appendix Section A. (b) Did you include complete proofs of all theoretical results?
[Yes] See appendix Section A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] All code and instructions are included in supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] All training details are speciﬁed in the paper and included code.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 7.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See appendix B. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] All existing assets were cited.
(b) Did you mention the license of the assets?
[Yes] Licences of all assets are properly attributed.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] New assets are included in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Here we prove the propositions stated in Section 4. A.1 Entropy Search Proposition 1.
If we choose A = P (Θ) and (cid:96) ( f, q ) = − log q ( θ f ) , then the EHIG is equivalent to the entropy search acquisition function, i.e. EHIG t ( x ; (cid:96), A ) = ES t ( x ) . Proof of Proposition 1.
We ﬁrst prove that under our deﬁnition of loss (cid:96) , the H (cid:96), A -entropy H [ f | D t ] is equivalent to the Shannon entropy of the posterior distribution over θ f (where θ f denotes a property of f that we would like to infer—as an example, θ f could be equal to the global maximizer x ∗ of f ).
Note that the H (cid:96), A -entropy is the expected loss of the Bayes action
[Equation]
We want to show that q ∗ deﬁned above is equal to p ( θ f | D t ) .
To do so, note that
[Equation]
[Equation]
where the ﬁrst equality holds since
[Equation]
(13) and the second equality holds since we can view E p ( θ f |D t )
[ − log q ( θ f |D t )] as a cross entropy, which is minimized when q ( θ f |D t )
=
p ( θ f |D t ) .
Therefore, under this loss and action set, using the deﬁnition of the EHIG we can write
[Equation]
A.2 Knowledge Gradient
Proposition 2.
If we choose A = X and (cid:96) ( f, x ) = − f ( x ) , then the EHIG is equivalent to the knowledge gradient acquisition function, i
[Equation]
e. EHIG t ( x ; (cid:96), A ) =
KG t ( x ) .
Proof of Proposition 2.
The proof follows directly from the deﬁnition of H (cid:96)
[Equation]
A -entropy and the EHIG, namely
[Equation]
A.3 Expected Improvement
Proposition 3.
If we choose A t = { x i } t − 1 i =1 , where x i ∈ D t , and (cid:96) ( f, x i ) =
− f ( x i ) , then the EHIG is equal to the expected improvement acquisition function, i.e. EHIG t ( x ; (cid:96),
[Equation]
Proof of Proposition 3.
The ﬁrst term of EHIG t in Eq.
(3) is equal to:
[Equation]
where ˆ f ( x i ) is the posterior expected value of f at x i .
The second term in Eq.
(3) is:
[Equation]
Putting it together
[Equation]
the EHIG t acquisition function in Eq.
(3) will reduce to:
[Equation]
A.4 Probability of Improvement
We additionally include a result below showing that the probability of improvement (PI) acquisition function can similarly be viewed as a special case of the proposed EHIG family.
Proposition 4.
For some constant τ , the acquisition function of PI is deﬁned as PI τ
( x ; D t ) =
E p ( f |D t )
[ I ( f ( x ) − τ > 0)] , where I ( · ) is the indicator function, and typically τ is taken to be equal to f ∗ t = max i ≤ t − 1 ˆ f ( x i ) for x i ∈ D t .
If we choose A t = { x t − 1 } , where x t − 1 ∈ D t , and (cid:96) τ ( f, x ) = − I ( f ( x ) − τ > 0) , then maximizing EHIG is equivalent to maximizing the probability of improvement acquisition function, i.e. arg max x ∈X EHIG t ( x ; (cid:96) τ , A ) =
arg max x ∈X PI τ ( x ) .
Proof of Proposition 4.
The ﬁrst term of EHIG t in Eq.
(3) is equal to:
[Equation]
where ˆ f ( x t − 1 ) is the posterior expected value of f at x t − 1 .
More importantly, H (cid:96), A t [ f | D t ] is a constant with respect to x that we are optimizing.
The second term in Eq.
(3) is:
[Equation]
Putting it together, the EHIG t acquisition function in Eq.
(3) will reduce to:
[Equation]
Thus maximizing EHIG is equivalent to maximizing the probability of improvement acquisition function.
B Additional Experimental Details and Results
Details on the Alpine- d function.
The multimodal Alpine- d function is deﬁned as Alpine- d ( x ) = (cid:80) di
=1 | x i sin( x i ) + 0 .
1 x i | , for x ∈ R d .
Details on the Vaccination function.
The vaccination function is obtained by training a MultiLayer Perceptron (MLP) network based on the data from [53], which uses county-level vaccination data provided by the CDC, and uses small area estimation 3 to interpolate the vaccination rate of every location.
We restrict the optimization domain to be a rectangle focusing on the state of Pennsylvania.
Details on the Multihills function.
The Multihills function is deﬁned as a mixture density as follows.
Multihills ( x ) =
(cid:80)
Jj =1 w j N ( x | µ j , C j ) , for x ∈ R d , where N denotes a multivariate normal density, { µ j } are a set of J means, { C j } are a set of J covarance matrices, and { w j } are a set of J weights.
Details on the Pennsylvania Night Light function.
We consider the 2012 gray scale global nightlight raster with resolution 0.1 degree per pixel.
The data is downloaded from NASA Earth Observatory 4 .
We restrict the optimization domain to be a rectangle focusing on the state of Pennsylvania and normalize all raster data before use.
Each location query gives a value proportional to the average amount of night light at that location.
Computational Cost.
While using the EHIG t ( x ; (cid:96), A ) acquisition function in Bayesian optimization (Algorithm 1) is more expensive than simpler methods (e.g. expected improvement (EI)), in many cases it has a comparable computational cost to methods such as knowledge gradient (KG) or entropy search (ES) methods, when applied to the same task—in fact, our implementation has a similar structure as one-shot knowledge gradient acquisition optimization methods.
The following timing results compare the average cost ( mean wall clock time in seconds ) of acquisition optimization for a set of comparison methods, including EI as an additional method, on the Alpinefunction from the ﬁrst experiment in our paper: EHIG: 6.9s, KG: 6.6s, EI: 0.5s, US: 0.3s .
B.1 Additional Experiment Results and Visualizations.
We show further experiment results for multi-level set estimation and sequence search (Figure 5), visualizations for multi-level set estimation (Figure 6), and an additional comparisons of classic BO acquisition functions on the initial top- k optimization experiments (Figure 7).
[Figure]
[Figure]
x
Random Search     10 x 1     x
Uncertainty Sampling     10 x 1     x
Knowledge Gradient     10 x 1     x
H ‘, A -Entropy Search 0 .0 .0 .0 .0 .1 .
{+0 x 1 0 .0 .0 .
+}
.0 .1 .x
Random Search 0 .0 .0 .0 .0 .1 .
{+0 x 1 0 .0 .0 .
+}
.0 .1 .x
Uncertainty Sampling 0 .0 .0 .0 .0 .1 .
{+0 x 1 0 .0 .0 .
+}
.0 .1 .x
Knowledge Gradient 0 .0 .0 .0 .0 .1 .
{+0 x 1 0 .0 .0 .
+}
.0 .1 .x
H ‘, A -Entropy Search
Random Search
Uncertainty Sampling
Knowledge Gradient
H ‘, A -Entropy Search