Joint Entropy Search for Multi-Objective Bayesian Optimization
Abstract
Many real-world problems can be phrased as a multi-objective optimization problem, where the goal is to identify the best set of compromises between the competing objectives.
Multi-objective Bayesian optimization (BO) is a sample efﬁcient strategy that can be deployed to solve these vector-valued optimization problems where access is limited to a number of noisy objective function evaluations.
In this paper, we propose a novel information-theoretic acquisition function for BO called Joint Entropy Search (JES), which considers the joint information gain for the optimal set of inputs and outputs.
We present several analytical approximations to the JES acquisition function and also introduce an extension to the batch setting.
We showcase the effectiveness of this new approach on a range of synthetic and real-world problems in terms of the hypervolume and its weighted variants.
Introduction
Bayesian optimization (BO) has demonstrated a lot of success in solving black-box optimization problems in various domains such as machine learning [75, 76, 90], chemistry [27, 35], robotics [7, 14] and clinical trials [62, 78].
The procedure works by maintaining a probabilistic model of the observed data in order to guide the optimization procedure into regions of interest.
Speciﬁcally, at each iteration the black-box function is evaluated at one or more input locations that maximizes an acquisition function on the model.
Implicitly, this function strikes a balance between exploring new areas and exploiting areas that have been shown to be promising.
In this work, we consider the more general problem where the black-box function of interest is vector-valued.
This increases the difﬁculty of the problem because there are now many directions in which the objectives can be improved, in contrast to the single-objective setting where there is only one.
Informally, the end goal of multi-objective optimization is to identify a collection of points that describe the best trade-offs between the different [-objectives.
-]objectives.
There are several ways to deﬁne an acquisition function for multi-objective BO.
A popular strategy is random scalarization [50, 63], which works by transforming the multi-objective problem into a distribution of single-objective problems.
These approaches are appealing because they enable the use of standard single-objective acquisition functions.
A weakness of this approach is that it relies on random sampling to encourage exploration and therefore the performance of this method might suffer early on when the scale of the objectives is unknown or when either the input space or the objective space is high-dimensional [21, 63].
Another popular class of multi-objective acquisition functions are improvement-based.
These strategies focus on improving a performance metric over sets, for example the hypervolume indicator [18, 19, 26, 92] or the R2 indicator [24].
The main drawback of these approaches is that the performance of these methods can be biased towards a single performance metric, which can be inadequate to assess the multi-objective aspects of the problem [97].
There are also many other multi-objective acquisition functions discussed in the litera36th Conference on Neural Information Processing Systems (NeurIPS 2022). ture, which mainly differ by how they navigate the exploration-exploitation trade-off [8, 9, 51, 67, 68]. Instead of relying on scalarizations or an improvement-based criterion, this paper considers the perspective where the goal of interest is to improve the posterior distribution over the optimal points.
We propose a novel information-theoretic acquisition function called the Joint Entropy Search (JES), which assesses how informative an observation will be in learning more about the joint distribution of the optimal inputs and outputs.
This acquisition function combines the advantages of existing information-theoretic methods, which focus solely on improving the posterior of either the optimal inputs [31, 33, 39] or the optimal outputs [4, 6, 79].
We connect JES with the existing information-theoretic acquisition functions by showing that it is an upper bound to these utilities.
Contributions and organization.
In Section 2, we set up the problem and introduce the novel
JES acquisition function.
In Section 3, we present a catalogue of conditional entropy estimates to approximate this utility and present a simple extension to the batch setting.
These approximations are analytically tractable and differentiable, which means that we can take advantage of gradient-based optimization.
The main results that we developed here can be viewed as direct extensions to the recent work in the Bayesian optimization literature presented in [58, 79].
In Section 4, we present a discussion on the hypervolume indicator and explain how it can be a misleading performance criterion because it is sensitive to the scale of the objectives.
We show that information-theoretic approaches are naturally invariant to reparameterization of the objectives, which make them well-suited for multi-objective black-box optimization.
For a more complete picture of performance, we propose a novel weighted hypervolume strategy (Appendix K), which allows us to assess the performance of a multi-objective algorithm over different parts of the objective space.
In Section 5, we demonstrate the effectiveness of JES on some synthetic and real-life multi-objective problems.
Finally in Section 6, we provide some concluding remarks.
Additional results and proofs are presented in the Appendix.
Preliminaries
We consider the problem of maximizing a vector-valued function f : X → R M over a bounded space of inputs X ⊂ R D .
To deﬁne the maximum max x ∈ X f ( x ) , we appeal to the Pareto partial ordering in R M .
For the rest of this paper, we will denote vectors by y =
( y (1) , . . .
, y ( M ) )
∈ R M , the non-negative real numbers by R ≥ 0 and diagonal matrices by diag ( · ) .
Pareto domination.
We say a vector y ∈ R M weakly Pareto dominates another vector y (cid:48)
∈ R M if it performs just as well in all objectives if not better: y (cid:23) y (cid:48)
⇐⇒ y − y (cid:
) ∈ R M ≥ 0 .
Additionally, if the vectors are not equivalent, y (cid:54) =
y (cid:48) , then we say strict Pareto domination holds:
y (cid:31) y (cid:48)
⇐⇒ y − y {+(cid:
+}
) ∈ R M ≥ 0 \ { 0 M } , where 0 M is the M -dimensional zero vector.
This binary relation can be further extended to deﬁne domination among sets.
Let A, B ⊂ R M be sets, if the set B lies in the weakly dominated region of A , namely B ⊆ D (cid:22) ( A ) = ∪ a ∈ A { y ∈ R M : y (cid:22) a } , then we say A weakly dominates B , denoted by A (cid:23) B .
In addition, if it also holds that the dominated regions are not equal, D (cid:22) ( A ) (cid:54) =
D (cid:22) ( B ) , we say strict Pareto domination holds, denoted by A (cid:31) B .
Multi-objective optimization.
The goal of multi-objective optimization is to identify the Pareto optimal set of inputs X ∗ =
arg max x ∈ X f ( x ) ⊆ X .
The Pareto set is deﬁned as the set of inputs whose objective vectors are not strictly Pareto dominated by another: x ∗ ∈
X ∗ ⇐⇒ x ∗ ∈ X and (cid:64) x ∈ X such that f ( x ) (cid:31) f
( x ∗ ) .
The image of the Pareto set in the objective space Y ∗ =
f ( X ∗ )
= max x ∈ X f ( x ) is called the Pareto front.
For convenience of notation, we will denote the set of Pareto optimal input-output pairs by ( X ∗ , Y ∗ ) .
Bayesian Optimization is a sample efﬁcient global optimization strategy, which relies on a probabilistic model in order to decide which points to query.
In Appendix A.1, we present the pseudo-code for the standard BO procedure—for more details see [13, 29, 74].
In this work, we will use independent Gaussian process priors [70] on each objective, f ( m ) ∼ GP ( µ ( m ) 0 , Σ ( m ) 0 ) , where µ ( m ) :
X → R is the mean function and Σ ( m ) :
X × X
→ R is the covariance function for objective m .
The observations at location x ∈ X will be assumed to be corrupted with additive Gaussian noise, y = f
( x ) + (cid:15) , where (cid:15) ∼ N (0 , diag ( σ ( x ))) denotes the observation noise with variance σ ( x ) ∈ R M ≥ 0 .
After n evaluations, we have a data set D n = { ( x t , y t ) } t =1 ,...,n .
The posterior model p ( f | D n ) is a collection of independent Gaussian processes f ( m ) | D n ∼ GP ( µ ( m ) n , Σ ( m ) n ) . The explicit expressions for the mean and covariance are presented in Appendix A.2. The main focus of this work is on designing the acquisition function, α : X → R , which is used to select the inputs: x n +1 = arg max x ∈ X α ( x | D n ) .
Information-theoretic acquisition functions focus on maximizing the gain in information from the next observation and a function of the probabilistic model.
Initial work in BO focussed on picking points to learn more about the distribution of the maximizer p ( X ∗ | D n ) .
Speciﬁcally, the goal of interest was to maximize the mutual information between the observation y and the Pareto set X ∗ conditional on the current data set D n :
[Equation]
where H [ p ( x )]
= − (cid:82) p ( x ) log p ( x ) d x represents the differential entropy.
This acquisition function is commonly referred to as predictive entropy search (PES) [39, 40, 73], but it was formerlyknown as entropy search (ES) [38, 83].
Despite the importance of obtaining more information about the maximizer, the PES acquisition function is heavily dependent on the approximation of p ( y | x , D n , X ∗ ) , which is both computationally difﬁcult to implement and optimize.
This motivated researchers to consider a simpler scheme that focusses on learning more about the distribution of the maximum p ( Y ∗ | D n ) .
The resulting acquisition function is known as the max-value entropy search (MES) [4, 44, 79, 85]:
[Equation]
Unlike PES, the conditional probability p ( y | x , D n , Y ∗ ) arising in MES can be approximated and optimized more easily because some approximations lead to closed-form expressions.
Despite the favourable properties of MES, the primary goal of interest is to identify the location of the maximizer X ∗ and not necessarily the value of the maximum Y ∗ .
To combine the advantages of both of these approaches, we propose the joint entropy search acquisition function, which focusses on learning more about the joint distribution of the optimal points p (( X ∗ , Y ∗ ) | D n ) :
[Equation]
The JES acquisition function inherits the advantages of the PES and MES acquisition functions because it considers the knowledge learnt about the optimal points and is also simple to implement— more details in the next section.
The following proposition shows that we can also interpret JES as an upper bound to both the PES and MES acquisition function.
Proposition 1.
The JES is an upper bound to any convex combination of the PES and MES acquisition functions:
[Equation]
|
[Equation]
n ) , for any β ∈ [0 , 1] .
In Figure 1, we illustrate the subtle differences between the different information-theoretic acquisition functions.
More speciﬁcally, we visualise the difference between the conditional distributions arising in each acquisition function for a single-objective problem using one sample of the optimal points.
Remark.
In the BO literature it is common to distinguish between single-objective and multiobjective acquisition functions by appending ‘MO’ to the end of the acronym.
For notational simplicity, we opt against this convention in this paper.
In Appendix C, we emphasize the main differences that arise when computing the information-theoretic algorithms in both settings.
Approximating JES
In this section, we present several approximations to the JES acquisition function (3) and a simple extension to the batch setting.
The ﬁrst term in the JES criterion (3) is the entropy of a multivariate normal distribution:
[Equation]
The second term is an intractable expectation which is approximated by drawing Monte Carlo samples from p (( X ∗ , Y ∗ )
| D n ) .
The conditional entropy H [ p ( y | x , D n , ( X ∗ , Y ∗ ))] is also an intractable quantity which has to be estimated.
The overall approximation of (3) will take the form
[Equation]
where h denotes the conditional entropy estimate and ( X ∗ s , Y ∗ s ) ∼ p (( X ∗ , Y ∗ )
| D n ) are the Monte
Carlo samples.
The distribution p ( y | x , D n , ( X ∗ , Y ∗ )) is very challenging to work with because it enforces the global optimality condition that the function lies below the Pareto front f ( X ) (cid:22) Y ∗ .
Instead of enforcing global optimality, we make the common simplifying assumption as in [58, 79, 85] and only enforce the optimality condition at the considered location: f ( x ) (cid:22) Y ∗ .
By applying Bayes’ theorem, the resulting density of interest becomes
[Equation]
where we have denoted the augmented data sets by D n ∗
=
D n ∪ ( X ∗ , Y ∗ ) and D n + = D n ∗ ∪{ ( x , y ) } .
We will refer to the quantity p ( f ( x ) (cid:22) Y ∗ ) as the cumulative distribution function (CDF).
The following lemma shows that this CDF can be computed analytically when the set Y ∗ ⊂ R M is discrete.
This is a standard result [16, 49, 67, 79] which can be derived by ﬁrst partitioning the region of integration, D (cid:22) ( Y ∗ ) = ∪ y
∗ ∈
Y ∗ { z ∈ R M : z (cid:22) y ∗ } , into a collection of hyperrectangle subsets and then summing up the individual contributions—see Figure 2 for a visual.
This partition can be computed using an incremental approach (Algorithm 1 of [54]), which has a cost of O ( | Y ∗ | (cid:98) M/ 2 (cid:99) +1 ) .
In the single-objective setting, the maximum is a single point y ∗ ∈ R and the box-decomposition is simply the interval D (cid:22) ( { y ∗ } ) =
( −∞ , y ∗ ] .
Lemma 1.
Let Y ∗ ⊂ R M be a ﬁnite set and z ∼ N ( a , diag ( b )) be an M -dimensional multivariate normal with mean a ∈ R M and variances b ∈ R M ≥ 0 .
Let D (cid:22) ( Y ∗ ) =
(cid:83)
Jj =1 B j = (cid:83)
Jj =1 (cid:81)
Mm =1 ( l ( m ) j , u ( m ) j ] be the box decomposition of the dominated space, then
[Equation]
In Algorithm 1, we present the pseudo-code for the estimation of the JES acquisition function at a single candidate input. Several variables that calculated within the algorithm are independent of the input (coloured in blue). For computational efﬁciency, we only compute these variables once and then save them to memory for later use.
[Figure]
do
Sample a path f
s
Compute optimal points
∗
X f Y ∗ s f Compute Compute
8 end
3.1 Estimating the conditional entropy
The entropy of (6) can be written as an M -dimensional expectation over the multivariate normal distribution p ( y | x , D n ∗ ) =
N ( y ; µ n ∗ ( x ) , Σ n ∗ ( x , x )) :
H [ p ( y | x , D n ∗ , f ( x ) (cid:22) Y ∗ )] = − E p ( y | x ,D n ∗ ) (cid:20) p ( f ( x ) ( cid:22) Y ∗ | x , D n + ) p ( f ( x ) (cid:22) Y ∗ | x , D n ∗ ) log (cid:18) p ( f ( x ) (cid:22) Y ∗ | x , D n + ) p ( f ( x ) (cid:22) Y ∗ | x , D n ∗ ) p ( y | x , D n ∗ ) (cid:19)(cid:21) .
[Equation]
To simplify the notation, we deﬁne the m -th standardized value by
[Equation] for any scalar z ∈ R .
Using this function together with Lemma 1, we denote the cumulative distribution p ( f ( x ) (cid:22) Y ∗ | x , D n ∗ ) by W = (cid:80)
Jj =1 W j = (cid:80)
Jj =1 (cid:81)
Mm =1 W j,m , where [Equation] are the differences appearing in (7).
Moreover, we denote the differences of the ﬁrst derivative of W j,m and the negative of the second derivative (with respect to γ m ) by
[Equation]
where φ is the probability density function of a standard normal distribution.
In the setting where the observation noise is zero, the conditional distribution is a truncated multivariate normal, which is known to have an analytical equation formula for the entropy (Theorem 3.1.
in [79]).
In Appendix E, we construct an ad hoc extension to this expression when the observation noise is non-zero. In the noisy setting, the distribution of interest is a type of multivariate skew normal distribution, which is known to not have an analytical form for the entropy [1].
As a result, we propose two approximation strategies to estimate this entropy.
The ﬁrst strategy is to approximate the integral using Monte Carlo.
The details of the Monte Carlo estimate h JES-MC is described in y (1)
y ( 2 )
Skew Multivariate Normal 0 .
( (1) n * ( x ), (2) n * ( x ))
Pareto front 0.25 0.
(a) y (1) y ( 2 )
Truncated Multivariate Normal 0 .0 .0 .
(b) y (1) y ( 2 )
Multivariate Normal (Full Covariance) 0 .0 .0 .0 .1 . 0 1 .1 .
(c) y (1) y ( 2 )
Multivariate Normal (Diagonal Covariance) 0 .0 .0 .
(d)
Appendix F. The second strategy is to directly approximate the distribution with one that exhibits an analytical entropy. We consider the most obvious choice, which is a multivariate normal distribution with the same ﬁrst two moments.
The same strategy was proposed in [58] for the single-objective multi-ﬁdelity MES acquisition function. By a standard result (Chapter 12 of [17]), the entropy of this approximating distribution is actually an upper bound for the entropy of interest: H [ p ( y | x , D n ∗ , f ( x ) (cid:22) Y ∗ )] ≤ M 2 log(2 πe ) + 12 log det V ar ( y | x , D n ∗ , f ( x ) (cid:22) Y ∗ ) . The following result shows that the these central moments can be computed analytically.
Proposition 2. Under the modelling set-up outlined in Section 2, for an input x ∈ X the ﬁrst and second central moment of p ( y | x , D n ∗ , f ( x ) (cid:22) Y ∗ ) are [Equation] and
[Equation]
As an upper bound on the conditional entropy leads to a lower bound on the mutual information, we will refer to the resulting conditional entropy estimate as the JES-LB estimate:
[Equation]
We could obtain a further lower bound by ignoring the off-diagonal terms in the covariance matrix.
We dub the resulting approximation as the JES-LB2 entropy estimate:
[Equation]
Figure 3 presents an illustration of the different density approximations that are used within the various conditional entropy estimates.
An important remark is that all the conditional entropy estimates that we have developed here can also be applied to estimate MES.
The only difference in the MES algorithm is that we no longer apply the conditioning step (line 6 of Algorithm 1) because we are interested in estimating H [ p ( y | x , D n , f ( x ) (cid:22) Y ∗ )] as opposed to (8).
Consequently, the MES acquisition function is cheaper to evaluate because the cost of evaluating the posterior variance at a single input is O ( n 2 ) , whereas JES incurs a cost of O (( n + | Y ∗ | ) 2 ) —more details are presented in the cost analysis in Appendix H.
3.2 Batch evaluations
Evaluating the JES acquisition functions for a batch of points x [1: q ] =
( x [1] , . . .
, x [ q ] ) ∈
X q is expensive because the entropy estimates now depends on the q -dimensional normal CDF and its derivatives.
To circumvent this issue, we follow the example of [58] and propose a suboptimal batch approach by upper bounding the expensive joint conditional entropy term by the sum of the individual entropies: H [ p ( y [1: q ] | x [1: q ] , D n ∗ , f ( X ) (cid:22) Y ∗ )] ≤ (cid:80) qi
=1 H [ p ( y [ i ] | x [ i ] , D n ∗ , f ( X ) (cid:22) Y ∗ )] .
The resulting q -batch lower bound JES estimate is given by
[Equation]
where h is the conditional entropy estimate and
[Equation]
[-U HV =0.
-]
[-U HV =0.
-]
is the initial entropy.
This acquisition function is deﬁned over a qD -dimensional space, which becomes more difﬁcult to optimize as q increases.
Alternatively, we can maximize this function greedily by sequentially selecting the best input conditioned on the previously chosen points.
This greedy procedure satisﬁes an e − 1 regret bound when the acquisition function is submodular [87].
In
Appendix G, we show that this batch acquisition function is indeed submodular.
Performance criteria
In multi-objective optimization, the most common way to measure performance is by comparing the approximate Pareto set ˆ X ∗ against the optimal Pareto set X ∗ in the objective space: d ( f (ˆ X ∗ ) , f ( X ∗ )) where d : 2 R M × 2 R M → R is a function that measures the discrepancy between the sets of objective vectors.
Existing work in multi-objective BO mainly focusses on the hypervolume (HV) discrepancy, d HV ( A, B ) =
| U HV ( A ) − U HV ( B ) | , where the HV indicator, U HV ( A ) =
(cid:82) R M I [ r (cid:22) z (cid:22) A ] d z , is deﬁned as the volume between a reference point r ∈ R M and a set
A ⊂ R M .
The general guidance is to set reference point to be slightly worse than the nadir, which is the vector consisting of the worst possible points, min x ∈ X f ( m ) ( x ) , for objectives m = 1 , . .
, M —see [46] for more details. An attractive feature of the HV indicator is that it is Pareto complete (or compliant) in the sense that a better set will lead to a larger HV [97]: A (cid:31) B =
⇒ U HV
( A ) > U HV ( B ) , if we assume the sets A and B are ﬁnite.
The reverse implication known as Pareto compatibility does not hold for the HV indicator [97].
In other words, the HV can be used to discriminate between sets where one dominates another, but it cannot be relied upon when the sets are incomparable.
Not all incomparable sets are treated equally by the HV indicator [95].
For instance Figure 4a shows an example where the HV indicator places more emphasis on the end points of the Pareto front.
On other hand, if we apply a monotonically increasing transformation g m : R → R to each objective, the Pareto set will not change, whereas the HV comparison will (Figure 4b).
Implicitly, the HV indicator assumes that a linear change in one objective is equivalent to a linear change in another.
This assumption might not necessarily reﬂect the decision maker’s outlook and this is something that is typically overlooked when designing and benchmarking multi-objective optimization algorithms.
The following result shows that information-theoretic acquisitions function are in fact agnostic to the choice of parameterization.
Proposition 3.
The information-theoretic acquisition functions α PES , α MES and α JES are invariant to reparameterization of the objective space that are consistent with the Pareto ordering relations.
For example, α JES ( x | D n ) =
MI ( y ; ( X ∗ , Y ∗ )
| D n )
=
MI ( g ( y ); ( X ∗ , g ( Y ∗ ))
| D n ) , where the g m : R → R is a strictly monotonically increasing function acting only on the m -th objective.
To benchmark the algorithms, we use both the standard HV discrepancy (Section 5) and the HV discrepancy under different parameterizations (Appendix L).
To easily obtain a family of parameterizations, we devise a novel weighting approach in Appendix K, which exploits the fact that the HV indicator can be written as an expectation over a uniform distribution on the ( M − 1)
-dimensional hypercube [23, 93]. We observe that it is possible to assess the performance at different locations [Figure] [Figure]
[Figure]
[Figure]
U
{+[Figure] [Figure] y (1) y ( 2 ) U HV =0.
+}
Objective space (a) g 1 ( y (1) ) g 2 ( y ( 2 ) )
{+U HV =0.
+}
Transformed objective space g 1 ( y (1) )
g 2 ( y ( 2 ) )
U HV =
0.822 Transformed objective space (
[Figure]
of the objective space by using alternate distributions over hypercube. We call the resulting metric the generalized hypervolume (GHV). In our experiments, we found that the performance of each algorithm changed with regards to the choice of parameterization, but the JES approaches tended to be one of the strongest performers throughout.
Experiments
We empirically evaluate the JES acquisition function on a range of synthetic and real-world benchmark problems.
We compare this approach with some popular acquisition functions in multi-objective
BO: TSEMO [12], ParEGO [50], NParEGO [19], EHVI [18], NEHVI [19], PES [31, 33] and MES- [79].
We have also included the MES-LB, MES-LB2 and MES-MC acquisition functions, which can be easily derived from the conditional entropy estimates that we developed here.
All algorithms are based on the open source Python library BoTorch [3], which uses features from GPyTorch [30] for Gaussian process regression and PyTorch [65] for automatic differentiation.
All experiments are repeated using 100 different initial seeds and we generate the Pareto set recommendation ˆ X ∗ of 50 points by maximizing the posterior mean using a multi-objective solver (NSGA2 [22] from the
Pymoo library [10]).
The complete details of the experiments are outlined in Appendix L, whilst the code is available at https://github.com/benmltu/JES.
5.1 Benchmarks
Synthetic benchmark.
We consider the ZDT2 [22] benchmark with D = 6 inputs and M =objectives.
We corrupt the observations with additive Gaussian noise with zero-mean and standard deviation set to approximately 10% of the objective ranges.
Chemical reaction.
This benchmark considers a nucleophilic aromatic substitution reaction (SnAr) between 2,4-diﬂuoronitrobenzene and pyrrolidine in ethanol to produce a mixture of a desired product and two side-products [45].
The design space comprises of D = 4 components relating to the initial conditions.
The goal is to optimize M = 2 objectives, namely the space time yield and the environmental impact.
We apply a logarithm transform to the objectives and contaminate the observations with additive Gaussian noise with zero-mean and standard deviation set to approximately 3% of the resulting objective ranges in order to emulate a potential real-world scenario.
Pharmaceutical manufacturing.
This problem is concerned with optimizing the Penicillin production process outlined in [55].
The design space is made up of D = 7 elements that control the initial condition of the reactions.
The goal is to optimize M = 3 objectives, which relates to the yield, the amount of carbon dioxide released and the time to ferment.
We include additive zero-mean
Gaussian noise with a standard deviation set to approximately 1% of the objective ranges.
Marine design.
This problem considers optimizing a family of bulk carriers subject to the constraints imposed for ships travelling through the Panama Canal [64, 72].
The design space is made up of D = 6 variables that determine the architecture of the carriers.
The goal of this problem is to [Figure] maximize the annual cargo, whilst minimizing the transportation cost and the ship weight subject to some design constraints.
We consider the reformulation in [82], which converts the constraints into another objective.
For this reformulated M = 4 objective problem, we corrupt the observations with additive zero-mean Gaussian noise with standard deviation set to approximately 0 .
5% of the objective ranges.
5.2 Results and discussion
We present the log HV discrepancy results for both the sequential and batch experiments in Figure 5.
The JES approach is consistently one of the stronger performing algorithms for these set of experiments.
A similar conclusion is reached when we consider the weighted variant of the hypervolume in
Appendix L.8.
Conditional entropy estimates.
We compared the performance of the different conditional entropy estimates for both the JES and MES acquisition function in Appendix L.6. We observed that in the majority of cases all the estimates exhibit similar performance.
As a result, we recommend using the cheapest approximation, which is usually the lower bound estimates, judging from the wall times presented in Appendix L.9.
Acquisition wall times.
The wall times in Appendix L.9 indicate that the cost of acquiring a new point with JES is comparable with NEHVI, slightly more expensive than MES, but cheaper than PES.
We note that the wall times for all methods can be improved by taking advantage of parallelization.
In particular, for entropy based methods we used a gradient-free optimizer to sequentially optimize the multi-objective samples (line 4 in Algorithm 1), whereas in practice we should of ideally solved these problems in parallel using a gradient-based optimizer such as [56].
Querying high performing points.
In certain domains it might be useful to directly query highperforming points because the ﬁnal decision will be restricted to only the sampled locations X N .
In Appendix L.5, we investigated the performance when such a restriction was made.
In this setting, we observed that the information-theoretic approaches were occasionally outperformed by the improvement and scalarization based acquisition functions, which picked points more greedily.
We observed that the entropy based approaches had a tendency to pick points that are more informative for the posterior over the optimal points as opposed to directly selecting a point that is known to perform well.
To address this setting, we recommend combining information-theoretic acquisition [Figure] (d [Figure]
q=1)
[Figure]
[Figure]
NParEGO
NEHVI
[Figure]
Region
[Figure]
: f ( * )
[Figure]
vectors: ( * )
Objective vectors: f ( * )
[Figure]
( d
[Figure]
[Figure]
log( d HV ) =
[Figure]
NParEGO log( d HV ) =
[Figure]
NEHVI log( d HV ) =
[Figure] PES log( d HV ) = 1. MES-LB log( d HV ) = 1.
JES-LB functions with an epsilon greedy approach, where points are occasionally picked according to a greedy strategy such as maximizing a function of the posterior mean.
Assessing local performance.
Using the generalized hypervolume, we can target different parts of the objective space in order to get a much better picture of performance.
We demonstrate this on a simple bi-objective example in Figure 6, where we assess that quality of the approximations at three different regions of the objective space.
We observe that all of the BO algorithms were quickly able to identify the right section of the Pareto front.
Evidently, the main source of difﬁculty for this problem arises from approximating the points in the left section of the Pareto front, which favours the second objective.
This observation would not be apparent if we focussed solely on the standard HV.
General guidance.
The ideal acquisition functions is problem dependent and strongly depends on the decision maker’s plans and goals.
In a completely black-box setting, where there is no immediate preferences, Proposition 3 and the empirical results motivates the usage of information-theoretic acquisition functions, which treats all points on the Pareto front as equally desirable a priori.
Conclusion
We introduced JES, a novel information-theoretic acquisition function for multi-objective BO.
To approximate this acquisition function, we presented several approximations to the conditional entropy and also a simple extension for the batch setting.
Experimental results suggest that JES is very competitive with existing acquisition functions in terms of the HV discrepancy and its weighted variants.
The main limitation of the JES acquisition function is that it relies on routines such as box decompositions and multi-objective optimization of function samples, which can be expensive to execute for very large problems.
Future work could focus on improving the scalability of these information-theoretic methods and extending it to more general settings, which include constrained, decoupled and multi-ﬁdelity optimization—see Appendix M for more details.
Acknowledgments and Disclosure of Funding
BT was supported by the EPSRC StatML CDT programme EP/S023151/1 and BASF SE, Ludwigshafen am Rhein.
NK was partially funded by JPMorgan Chase & Co. under J.P. Morgan A.I.
Faculty Research Awards 2021.
Joint Entropy
Search
for
Multi
-Objective Bayesian
Optimization
A Basics A.1 Bayesian optimization pseudocode
Bayesian optimization consists of two steps that are iterated until the budget of function evaluations
N is exhausted.
The ﬁrst step is modelling, this is where posterior of the probabilistic model p ( f | D n ) is computed.
The second step is acquisition, this is where a utility function is optimized in order to determine the next location to query.
The pseudo-code for this procedure is presented in Algorithm 2.
For a more thorough overview on Bayesian optimization consult the references [13, 29, 74].
Input : A black-box function f .
1 Initialize the probabilistic model p ( f ) .
2 for n = 0 , . . .
, N − 1 do
Optimize for the next point x n +1 = arg max x ∈ X α
( x | D n ) .
Evaluate the function y n +1 = f ( x n +1 ) + (cid:15) .
Augment the data set D n +1 =
D n ∪ { ( x n +1 , y n +1 ) } .
Compute the posterior p ( f | D n ) .
7 end 8 return D N and p ( f | D N ) .
A.2 Posterior Gaussian process
Under the independent Gaussian observation model described in Section 2, the posterior p ( f ( X ) | D n ) evaluated at a vector X ⊂ X , is a collection of Gaussian processes [70] with mean µ ( m ) n ( X
[Equation]
and covariance
[Equation]
[Equation]
[-( Y ( m ) n − µ ( m ) 
-]
where we denote the collection of sampled locations by ( X n ) t = x t and observations ( Y ( m ) n ) t = y ( m ) t for objectives m = 1 , . .
, M and data points t = 1 , . . . , n .
A.3 Sampling from a Gaussian process
Exact sampling of a one-dimensional Gaussian process over a ﬁnite set, X ⊂ X , scales cubically with the number of points | X | .
This cubic cost arises form the inversion of the covariance matrix.
As we want to sample global maximizers and maximums ( X ∗ , Y ∗ ) , exact sampling over the whole space X is not computational feasible.
We could restrict the sampling to a discrete subset of X , but this would require designing a principled strategy to select a reasonable subset, for example we could try triangulating between existing points [36].
To avoid this difﬁculty, we follow the example of previous work [40, 85], which considers generating approximate samples via random Fourier features [69, 88, 89].
The strategy centres around approximating the covariance kernel using a feature representation Σ ( m )
( x , x (cid:48) )
≈ ϕ ( x ) T ϕ
( x (cid:48) ) .
For a stationary covariance kernel Σ ( m ) 0 , the Fourier transform of the kernel is a non-negative measure that can be normalized to obtain a probability distribution p ( θ ) —this result follows from Bochner’s Theorem [11] .
For the exponential and Matern kernels, this probability distribution is known in closed-form [12, 69].
Assuming Σ ( m )is stationary, an approximate sample of the Gaussian process prior GP ( µ ( m ) 0 , Σ ( m ) 0 ) can be written as a Bayesian linear model
[Equation] [Equation]
where ϕ i ( x ) = (cid:112) 2 /L
cos( θ Ti x + τ i ) are the Fourier features depending on the random variables τ i ∼ U (0 , 2 π ) and θ i ∼ p
( θ ) , whilst ω i ∼ N (0 , 1) are the random weights for i = 1 , . . .
, L .
The posterior samples of GP ( µ ( m ) n , Σ ( m ) n ) can be obtained by adding an additional pathwise update via
Matherons’ rule [88, 89]:
[Equation]
where κ t =
(Σ ( m )
( X n , X n ) + diag ( σ ( m ) ( X n )))
− 1 ( Y n − f ( m )( X n )) .
For a more comprehensive overview of sampling from a Gaussian process refer to [89].
B Related work B.1 Conditional entropy estimates
In this work we devised a number of conditional entropy estimates for the JES acquisition function.
These estimates can also be used by the MES acquisition function.
In this brief section, we elaborate on the similarities of our estimates with the existing work relating to the MES acquisition function. Firstly, the noiseless entropy estimate (31) was derived recently in [79] to extend the noiseless MES acquisition function to the multi-objective setting.
This extension was called the Pareto frontier entropy search (PFES) and it is equivalent to the acquisition function we call MES-0.
An earlier attempt to extend the MES to noiseless setting resulted in the MESMO [4] acquisition function.
The MESMO acquisition function is equivalent to the PFES acquisition function if we crudely approximates the dominated space with a single box D (cid:22) ( Y ∗ )
≈ B = ( −∞ ,
max y ∈ Y ∗ y (1) ] × · · · × ( −∞ , max y ∈ Y ∗ y ( M ) ] .
Empirically, this approximation leads to a deterioration in performance [79]. The closest work to the lower bound entropy estimate is the GIBBON [58] acquisition function.
The GIBBON acquisition function was derived as a lower bound to the multi-ﬁdelity MES acquisition function for a single-objective problem.
The MES-LB and MES-LB2 can be interpreted as the multi-objective extensions of GIBBON for the single ﬁdelity setting. The Monte Carlo entropy estimate has been used before for the multi-ﬁdelity MES acquisition in the single-objective setting [59, 80].
The MES-MC estimate can be interpreted as the multi-objective extension of these approaches for the single ﬁdelity setting.
B.2 BAX
The JES acquisition function bares some similarity with a special case of the Bayesian Algorithm
Execution (BAX) algorithm proposed in [60].
Speciﬁcally, if we set O A ( f )
=
( X ∗ , Y ∗ ) in [60], then the corresponding BAX acquisition function would reduce to
[Equation]
In the JES acquisition function, we condition on the augmented data set D n ∗ =
D n ∪ ( X ∗ , Y ∗ ) and the optimality condition f ( X ) (cid:22) Y ∗ for the density arising in the conditional entropy term.
Whereas in BAX, we only condition on the augmented data set.
The BAX acquisition function is a lower bound to JES because conditioning never increases the entropy (Theorem 2.6.5 of [17]). C Single-objective setting
In this section, we discuss the main differences between the single-objective and multi-objective information-theoretic acquisition functions.
At a high level, the only difference between the two settings is whether we use the total ordering over real numbers or the Pareto partial ordering over vectors.
Note that in the single-objective setting, M = 1 , the Pareto partial ordering coincides with the standard total ordering order real numbers, that is to say the deﬁnition of the binary relations (cid:22) and ≺ coincides with the standard inequality signs ≤ and < , respectively. As a result, it is generally possible for multi-objective acquisition functions to be used in the single-objective setting, which is deﬁnitely the case for the estimates of our JES acquisition function. We will now comment individually on each information-theoretic acquisition functions on the more subtle differences between the single and multi-objective algorithms. PES.
There are two main difference between the single-objective and multi-objective PES algorithm.
Firstly, in the single-objective setting, we sample a single maximizer, x ∗ ∼ p ( x ∗ | D n ) , based on the standard total ordering, whereas in the multi-objective setting we sample a discrete set of maximizers, X ∗ ∼ p
( X ∗ | D n ) , based on the Pareto partial ordering.
Secondly, the equations governing the expectation propagation updates can be different depending on how the target density is factorised and the modelling assumptions that are made.
Therefore, setting M = 1 for the multi-objective PES algorithm (also known as PESMO) described in [31] might not exactly recover the same result for a different single-objective implementation of the PES algorithm.
In our code we follow the equations proposed in [31], which accounts for single, multi-objective, batch and/or constrained setting.
MES.
There are two main difference between the single-objective and multi-objective MES algorithm.
Firstly, in the single-objective setting, we sample a single maximum, y ∗ ∼ p ( y ∗ | D n ) , based on the standard total ordering, whereas in the multi-objective setting we sample a discrete set of maximums, Y ∗ ∼ p
( Y ∗ | D n ) , based on the Pareto partial ordering.
Secondly, in the single-objective setting the box decomposition is readily available without any effort, D (cid:22) ( { y ∗ } ) =
( −∞ , y ∗ ] , whereas in the multi-objective setting we have to compute it.
The MESMO algorithm [4] and the PFES algorithm
[79] both reduce to the single-objective MES algorithm [85] when we set M = 1 .
As mentioned before in Appendix B, the MESMO algorithm is a special case of the PFES algorithm when using a crude approximation to the box decomposition of D (cid:22) ( Y ∗ ) , which turns out to be exact when there is only one objective.
JES.
There are two main difference between the single-objective and multi-objective JES algorithm.
Firstly, in the single-objective setting, we sample a single optimal point, ( x ∗ , y ∗ ) ∼ p (( x ∗ , y ∗ ) | D n ) , based on the standard total ordering, whereas in the multi-objective setting we sample a discrete set of optimal points, ( X ∗ , Y ∗ ) ∼ p (( X ∗ , Y ∗ ) | D n ) , based on the Pareto partial ordering.
Secondly, in the single-objective setting the box decomposition is readily available without any effort, D (cid:22) ( { y ∗ } ) =
( −∞ , y ∗ ] , whereas in the multi-objective setting we have to compute it.
As mentioned before in
Section 3, if we do not perform the conditioning step in Algorithm 1, we obtain the MES algorithm.
D Proof of results
Before we begin the proofs, we will restate some important notation.
Let p ( y | x , D n ∗ ) =
N ( y ; µ n ∗ ( x ) , Σ n ∗ ( x , x )) denote the probability density at a point x ∈ X conditional on the data set
D n ∗ =
D n ∪ ( X ∗ , Y ∗ ) .
Then we denote the m -th standardized function by
[Equation]
(22) for m = 1 , . . .
, M .
For the discrete set of points Y ∗ ⊂ R M , we decompose the dominated region into J boxes:
[Equation]
[Equation]
where l j = ( l (1) j , . . .
, l ( M ) j ) are lower bounds and u j = ( u (1)1 , . .
, u ( M ) J ) are the upper bounds for boxes j = 1 , . . .
, J .
Using the box decomposition, we deﬁne
[Equation]
[Equation]
for boxes j = 1 , . . .
, J and objectives m = 1 , . . .
, M , where Φ is the cumulative distribution function of a standard normal distribution.
The ﬁrst derivative of W j,m (with respect to γ m ) is denoted by
[Equation]
and the negative of the second derivative by
[Equation]
for boxes j = 1 , . . .
, J and objectives m = 1 , . . .
, M , where φ is the probability density function of a standard normal distribution.
D.1 Proof of Proposition
Proposition 1.
The JES is an upper bound to any convex combination of the PES and MES acquisition functions: α JES ( x | D n ) ≥ βα PES ( x | D n ) + (1 − β ) α MES ( x | D n ) , for any β ∈ [0 , 1] .
Proof.
The upper bound property follows from the standard result that conditioning on more variables will never increase the entropy (Theorem 2.6.5 of [17]):
H ( A | B ) ≤
H ( A ) for random variables A and B .
Using this result, H [ p ( y | x , D n , ( X ∗ , Y ∗ ))] ≤ max( H [ p ( y | x , D n , X ∗ )] , H [ p ( y | x , D n , Y ∗ )]) .
Plugging this inequality into (3), we obtain the α JES ( x | D n ) ≥ max( α PES ( x | D n ) , α MES ( x | D n )) , which implies the result. (cid:4)
D.2 Proof of Lemma
Lemma 1.
Let Y ∗ ⊂ R M be a ﬁnite set and z ∼ N ( a , diag ( b )) be an M -dimensional multivariate normal with mean a ∈ R M and variances b ∈ R M ≥ 0 .
Let D (cid:22) ( Y ∗ ) =
(cid:83)
Jj =1 B j = (cid:83) Jj =1 (cid:81)
Mm =1 ( l ( m ) [Equation] , u ( m ) j ] be the box decomposition of the dominated space, then p
[Equation]
z
[Equation]
Proof.
Under the assumptions of Lemma 1, the following series of equations holds:
[Equation]
(1) This step follows from the box decomposition of D (cid:22) ( Y ∗ ) and the independence between the components of z .
This step follows from the deﬁnition of the CDF.
[Equation]
D.3 Proof of Proposition
Proposition 2.
Under the modelling set-up outlined in Section 2, for an input x ∈ X the ﬁrst and second central moment of p ( y | x , D n ∗ , f ( x ) (cid:22) Y ∗ ) are
[Equation]
and
[Equation]
Proof.
In the noisy setting, the density of interest is
[Equation]
From Lemma
[Equation]
To obtain a tractable expression for the cumulative distribution p ( f ( x ) (cid:22) Y ∗ | x , D n + ) , we ﬁrst compute the conditional distribution p ( y | x , D n + ) .
By standard Gaussian conditioning p ( y | x , D n + ) =
N ( y ; µ n + ( x ) , Σ ( m ) n + ( x , x )) where
[Equation]
and
[Equation]
) with ρ m := ρ m ( x ) = (cid:113) Σ ( m ) n ∗ ( x , x ) / (cid:113) Σ ( m ) n ∗ ( x , x ) + σ ( m ) ( x ) denoting correlation between the observation y ( m ) and the objective value f ( m ) ( x ) , for objectives m = 1 , . . .
, M .
Using the box decomposition, the posterior CDF is equal to
[Equation]
To simplify the notation we perform a change of variable:
[Equation]
where ¯ y ( m ) = ( y ( m ) − µ [Equation] m ) n ∗
[Equation] x )) / (cid:113) Σ ( m ) n ∗ ( x , x ) + σ ( m ) ( x ) is the standardized observation, which is distributed according to a standard normal random variable for objectives m = 1 , . . .
, M .
To derive the moments of p ( y | x , D n ∗ , f ( x ) (cid:22) Y ∗ ) , we ﬁrst obtain the moment generating function of the standardized observation ¯y .
[Equation]
By Lemma 2 in [2], the expectation of the normal CDF is given by (cid:82) R Φ( az + b ) φ ( z ) dz =
Φ( b/ √ 1 + a 2 ) for any constants a, b ∈ R .
Using this result, [Equation]
R
[Equation]
[Equation] The ﬁrst moment p (¯ y ( m ) | x , D n ∗ , f ( x ) (cid:22) Y ∗ ) can be obtained by evaluating the ﬁrst derivative of
M ¯y ( t ) with respect to t ( m ) at t = {+
+}
M
[Equation]
Differentiating a second time, we can obtain the second moments.
For m (cid:54) = m (cid:48) ,
[Equation]
and for m = m (cid:48) ,
[Equation]
=
[Equation]
W
[Equation]
The moments of y ( m ) can be now derived by reversing the initial linear transformation: y ( m ) =
¯ y ( m ) (cid:113) Σ ( m ) n ∗ ( x , x ) + σ ( m ) ( x ) + µ ( m ) n ∗ ( x ) .
The ﬁrst moment is
[Equation]
The second moment:
[Equation]
x , D n ∗ , f ( x ) (cid:22) Y ∗   .
This implies
[Equation]
Substituting in the expressions before
[Equation]
we have that for m (cid:54) = m (cid:48) ,
[Equation]
and for m = m (cid:48)
[Equation]
By some additional algebraic manipulation
[Equation]
the covariance for m (cid:54) = m (cid:48) is given
[Equation]
and the variance is
[Equation]
D.4 Proof of Proposition
Proposition 3.
The information-theoretic acquisition functions α PES , α MES and α JES are invariant to reparameterization of the objective space that are consistent with the Pareto ordering relations.
For example, α JES ( x | D n ) =
MI ( y ; ( X ∗ , Y ∗ )
| D n )
=
MI ( g ( y ); ( X ∗ , g ( Y ∗ ))
| D n ) , where the g m : R → R is a strictly monotonically increasing function acting only on the m -th objective.
Proof.
We will prove the general statement that the mutual information is invariant under smooth bijective transformations.
Consider two random variables X and Y then following equations hold:
[Equation]
where X (cid:48)
=
g x ( X ) and Y (cid:48) =
g y ( Y ) represents the transformed variables under some suitably deﬁned
The result follows by restricting the class of bijective functions to ones where the Pareto set remains unchanged.
For example, the class of monotonic increasing functions in each objective ensures this property holds: g : R M → R M such that g ( y )
= ( g 1 ( y (1) ) , . . .
g M ( y ( M ) )) with g m being monotonically increasing. [Equation]
E Noiseless entropy estimate
In this section, we present the conditional entropy estimate for the zero observation variance setting and an ad hoc extension for noisy setting.
Firstly, if we assume the observation variance is zero, the conditional distribution of interest is a truncated multivariate normal:
[Equation]
which is known to have the following analytical equation for the entropy:
Proposition 4.
(Theorem 3.1.
in [79])
Under the modelling set-up outlined in Section 2, if x ∈ X is an input with zero observation variance, σ ( x ) =
M , then the entropy of the truncated multivariate normal distribution p ( y | x , D n ∗ , f ( x ) (cid:22) Y ∗ ) is given by
[Equation]
Proof.
See [79] for the proof of this result.
[Equation]
Assuming σ ( x ) =
M , we have
[Equation]
The ﬁrst term is equal to expectation of the negative log-probability of the noiseless observation lying below Pareto front:
[Equation]
This term accounts for the exploitation because it puts more emphasis on the points that are likely to lie above the Pareto front.
The remaining terms account for the exploration by placing more emphasis on reducing the uncertainty at input location.
To still take advantage of the result in Proposition 4 for the noisy observation setting, we propose an ad hoc extension that adjusts the exploration term to include the effects of the observation noise.
Speciﬁcally, we propose a modiﬁcation which replaces the difference in log variances in the exploration term,
[Equation]
with the differences in log variances plus observation noise,
[Equation]
where σ ( m ) ( x ))( x ) is the observation variance for objectives m = 1 , . .
, M at x ∈ X .
Using this adjustment, we deﬁne the resulting conditional entropy estimate by
[Equation]
Empirically, we observe that the performance of this conditional entropy estimate is in-line with the other conditional entropy approximations.
F Monte Carlo entropy estimate
In this section [Equation] we consider estimating the entropy of p ( y | x , D n ∗ , f ( x ) (cid:22) Y ∗ ) via Monte Carlo.
As a reminder, the entropy of interest can be written as an expectation over p ( y | x , D n ∗ ) : [Equation] [ p [Equation]
x
[Equation]
∗ , f ( x ) (cid:22) Y ∗
[Equation]
The CDF in the denominator is p ( f ( x ) (cid:22) Y ∗ | x , D n ∗ )
=
W from (28), whilst the CDF in the numerator is
[Equation]
from (29).
By sampling y i ∼ p ( y | x , D n ∗ ) for i = 1 , . . .
, I , we can approximate the entropy with the following Monte Carlo average:
[Equation]
where the Monte Carlo entropy estimate is given by
[Equation]
Instead of generating new samples for each call of the acquisition function, we follow the general wisdom in BO [87] and apply the reparameterization trick on the sampling distribution: y i = µ n ∗ ( x ) + C n ∗ ( x ) z i , where C n ∗ ( x ) ∈ R M × M is the Cholesky factor of Σ n ∗ ( x , x ) and z i ∼ N (0 , I M ) are the base samples that only need to be initialized once.
The variance of this estimate could potentially be reduced by including control variates.
For example,
[Equation]
[Equation]
where the expectations are known,
[Equation]
and β 1 , β 2 ∈ R are the regression coefﬁcients.
We did not assess the effect adding control variates to our Monte Carlo estimate because the standard quasi-Monte Carlo scheme with the reparameterization trick worked reasonably well out of the box.
One word of caution [43]: combining both control variates and quasi-Monte Carlo could lead to an increase in variance if the coefﬁcients are naively estimated.
G Submodularity
A set function g : 2 V → R is submodular if for every A, B ⊂ V , g ( A ∩ B )+ g ( A ∪ B ) ≤ g ( A )+ g ( B ) , where V is the set of interest—for further details on submodularity refer to [52].
The following proposition states that the lower bound batch acquisition function and its approximation for the JES and MES are submodular functions deﬁned over subsets of the input space.
Proposition 5.
The lower bound batch acquisition function α qLB-JES ( X
| D n ) and its approximations ˆ α qLB-JES ( X | D n ) are submodular functions deﬁned over subsets of X .
Proof.
Let X
= { x i } i =1 ,...,q ⊂ X denote a set of inputs.
If K X ∈ R q × q is a positive semi-deﬁnite matrix such that K i,j depends only on the inputs x i ∈ X and x j ∈ X , then the log-determinant function log det K X deﬁned over sets X is submodular [53].
Similar to [58], we will show that this lower bound batch acquisition function can be written as a sum of these log-determinants.
[Equation]
where ζ
[Equation]
For notational convenience, let A ( m ) =
Σ ( m ) n ( x [
[Equation]
q ] , x [ [Equation] q ] ) + diag ( σ ( m ) ( x [1: q ] )) for m = 1 , . .
, M and B = diag (cid:16) e ζ ( x [1: q ] ) /M
(cid:17) .
Combining the terms, we obtain
[Equation]
where K ( m ) =
BA ( m ) B with K ( m ) i,j = e ζ ( x [ i ] ) /M
e ζ ( x [ j ] ) /M
A ( m ) i,j for i, j = 1 , . . .
, q and m = 1 , . .
, M .
Each summand, log det K ( m ) , is a submodular function [53].
As a sum of submodular functions is submodular, we conclude using that the lower bound batch acquisition function is submodular.
For the approximate batch acquisition function, the expectation in (35) is replaced by the corresponding Monte Carlo estimate.
The submodularity derivation above continues to hold when using the conditional entropy approximations.
(cid:4)
H Cost analysis
In this section we consider the costs involved in evaluating the JES acquisition function according to
Algorithm 1.
We also include some discussion of the cost of the other information-theoretic criterion, namely MES and PES.
Sampling cost.
The cost of approximate sampling from a single Gaussian processes p ( f ( m ) | D n ) using the random Fourier features (described brieﬂy in Appendix A.3) is O (min( n, L ) 3 ) , where L is the number of Fourier features—for more details refer to [88].
An evaluation of a sample at a set of inputs X ⊂ X has a linear cost depending on | X | [88].
Multi-objective optimization cost.
Given a sample f s , we optimize for p Pareto optimal points ( X ∗ s , Y ∗ s ) using the popular genetic algorithm known as NSGA2 [22].
The cost of this algorithm is
O ( MN 2 pop N gen N off ) , where N pop is the size of the population, N gen is the number of generations and
N off is the number of offspring [22].
At a high level, NSGA2 works by evaluating the function at
N pop locations at time t = 1 and then moves on to evaluate N off candidates for the rest of the time t = 2 , . .
N gen .
The location of the offspring evaluations are determined by a random heuristic motivated by the mechanisms involved in the theory of evolution: crossover, mutation, elitism and diversity.
Box decomposition.
The cost of performing a single the box decomposition based on the incremental algorithm in [54] is O ( p (cid:98) M/ 2 (cid:99) +1 ) , where p is the number of Pareto optimal points.
Conditioning cost.
Conditioning on an additional p data-points requires updating the Cholesky decomposition of the input covariance matrix.
The cost of a single update of this kind relies on a triangular solve, which has a quadratic complexity O ( M ( n + p ) 2 ) .
Initial entropy evaluation cost.
The initial entropy (4) can be computed directly from the posterior covariance.
The cost of instantiating the caches of the Gaussian process covariance, namely (Σ ( m ) {+
+} ( X n , X n ) + diag ( σ ( m ) ( X n )) − 1 is O ( n 3 ) .
The cost of evaluating the posterior covariance (18) at point x ∈ X is O ( n 2 ) .
In the batch case, we also have to compute a log-determinant of the q × q covariance matrix, which has a cost of O ( q 3 ) .
We use the implementation in the GPyTorch [30], which computes the approximate log-determinant, which only uses a linear cost of O ( q ) —this approximation approach is outlined in [25].
Conditional entropy evaluation cost.
Assuming the posterior model and conditioned model are instantiated, we will consider the operations involved when evaluating the conditional entropy estimates.
The cost of evaluating the probability density function and CDF of a univariate normal distribution at a point x ∈ X are both assumed to be O (1) .
As a result, the dominant cost of evaluating the h JES- 0 (31) and h JES-LB 2 (14) comes from the evaluation of the variance.
For h JES-LB (13), we additionally have to populate an M × M matrix and compute its log-determinant.
For h JES-MC (32), we need to initialize a set of I base samples for the reparameterization trick [87] described in
Appendix F—this has a linear one time cost of O ( MI ) .
Assuming the Monte Carlo samples are generated, the rest of the operations depend linear on I .
Expectation propagation.
To approximate the density p ( y | x , D n , X ∗ ) in the PES acquisition function (1), the authors of [31] consider using expectation propagation [57].
The dominant cost of this is from matrix inversions of the covariance matrix, which scales cubically with the number of data points in consideration.
In the initialization phase, we prepare the expectation propagation caches, which require inverting an ( n + p ) × ( n + p ) matrix, whilst during testing we invert a ( q + p ) ×
( q + p ) matrix, where q is the batch size.
Total cost.
The total cost is the sum of the initialization cost and the query cost.
Using the variables deﬁned in Table 1 and Table 2, we can write down the initialization and query cost of JES, MES and PES.
For the calculations, we assume that we have already queried n points and we want to compute the acquisition function at a batch of q points using S Monte Carlo samples of the Pareto set and/or front comprised of p points.
• JES.
The initialization cost is C init + SC sample ( L )+ SC moo ( N pop , N gen , N off )
+ SC bd ( p )+ SC cond ( p ) . There is an additional initial cost of SC base-sample ( I ) for the Monte Carlo conditional entropy estimate.
The query cost is C init-h ( q ) + SqC h ( p ) , where C h ∈ { C h-0 , C h-lb , C h-lb2 , C h-mc } is the conditional entropy estimation strategy.
Keeping the other parameters ﬁxed, the dominant query cost is Mq 2 when using the approximate log-determinant and Mq 3 when using the standard log-determinant for large q . • MES.
We consider the MES algorithm obtained by excluding the conditioning step described in Algorithm 1.
The initialization cost of this approach is C init ( q ) + SC sample ( L )
+ SC moo ( N pop , N gen , N off )
+ SC bd ( p ) .
The query cost is C init-h ( q ) + SqC h (0) , where C h ∈ { C h-0 , C h-lb , C h-lb2 , C h-mc } is the conditional entropy estimation strategy. Keeping the other param- eters
ﬁxed the dominant query cost is Mq 2 when using the approximate log-determinant and Mq
when using the standard log-determinant for large q
• PES.
We consider the expectation propagation approach described in [31] to approximate the PES acquisition.
The batch extension is derived in a follow up work [33]
The initialization cost of this
approach is C init ( q
+ SC sample L
+ SC
moo ( N pop , N gen , N off ) + SC ep0 ( p ) .
The query cost is C init -h ( q ) + SC ep ( q,
p Keeping the other parameters ﬁxed, the dominant query cost is SMq 3 for large q
Naturally, the initialization phase can be executed in parallel because we are running S independent operations.
For our experiments, we did not take advantage of this advantageous property when performing the sampling, multi-objective optimization and box decomposition.
[Table]
Initializing the posterior p ( f | D n )
[Table]
Log-determinant of a K × K matrix
[Table]
Approximate log-determinant of a K × K matrix
[Table]
Generating I base samples from N (0 , diag ( 1 M ))
[Table]
Approximate sampling of p ( f | D n )
[Table]
Multi-objective optimization of f s
[Table]
Box decomposition of Y ∗ s with p points
[Table]
Conditioning on the Pareto optimal point p ( f | D n ∗ )
[Table] p ) [Table]
Initialization of expectation propagation caches
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table]
[Table]
, n )
M ( | X | n 2 + | X | 2 ) Initial entropy H [ p ( y [1: q ] | x [1: q ] , D n )] C init-h ( q )
C cov ( q, n ) + MC logdet ( q )
Conditional entropy estimate h JESC h-0 ( p )
C cov (1 , n + p )
Conditional entropy estimate h JES-LB
C h-lb ( p )
C cov (1 , n + p ) + M 2 + C logdet ( M )
Conditional entropy estimate h JES-LB
C h-lb2 ( p )
C cov (1 , n + p )
Conditional entropy estimate h JES-MC
C h-mc ( p )
C cov (1 , n + p ) + MI
Expectation propagation
C ep ( q, p )
M ( q + p )
I Estimation error
Overall, there are ﬁve sources of estimation error when approximating the batch JES acquisition function.
In this section, we brieﬂy enumerate and discuss these errors below.
The ﬁrst source of error arises from replacing the global optimality condition with the local optimality condition.
This turns out to be a lower bound approximation: [Equation] because conditioning on more variables will never increase the entropy. The error from this lower bound approximation appears also in the majority of the work on entropy based acquisition
functions.
Empirically, we observed that this approximation leads to a very minor change to the selected point for the sequential acquisition function.
The effects of this error on the batch acquisition function is unclear because computing an unbiased estimate of the exact batch acquisition function over the whole input space is too expensive to do in practice.
The second and third source of error arises from the Monte Carlo approximation of the intractable expectation and the discrete approximation of the Pareto optimal points, respectively.
The error of this step could be reduced by increasing the number of samples or optimal points at the expense of more computation.
In the experiments we conducted, we set the number of Monte Carlo samples as S = 10 and the number of optimal points as p = 10 .
This combination seemed to work well under our collection of problems and the sensitivity analysis we conducted in Appendix L. seems to indicate that there is a diminishing gain in performance when we increase the number of samples or optimal points further.
The fourth source of error comes from estimating the conditional entropy.
This can be computed exactly in the noiseless setting, but has to be estimated in the other settings.
The Monte Carlo estimate gives an unbiased estimate of this term, which can be made more accurate by increasing the number of samples.
In the experiments presented in Appendix L, we observed very little difference in the points that are selected when using the different conditional entropy estimates.
As a result, we recommend using the cheapest estimates which in most cases is the one based on the moment-matching approach.
The ﬁfth source of error comes from estimating the batch acquisition function using the lower bound.
This error is only present when the batch size is greater than one: q > 1 .
Quantifying this error is hard to do empirically because the exact batch acquisition function is too expensive to estimate unbiasedly.
Nevertheless, the lower bound batch acquisition function and its approximation are still principled acquisition functions because they can be written as determinantal point processes (DPPs) [53].
This property was derived and used in the proof of submodularity in Appendix G.
As discussed in the BO literature [15, 48, 86], batch acquisition functions based on
DPPs are powerful because they can promote diverse batches in high-quality regions.
The trade-off between diversity and quality (Section 3.1 in [53]) is evident in the form of the DPP kernel, K ( m ) i,j = q i q j A ( m ) i,j ,
where q i = exp( ζ i /M ) can be interpreted as the quality of item i, whilst A ( m ) i,j corresponds to a notion of similarity.
In the approximation, we use a Monte Carlo estimate for the term ζ i , which only arises when computing the quality term.
As a result, the batch that is selected with this approximation might differ slightly from the optimal batch but the diversity of the batch will still be high because of the matrix A ( m ) .
J Contour plots
In this section we present some contour plots to illustrate visually the differences that emerges between the different approximation strategies for the information-theoretic acquisition functions on a single-objective problem.
The approximation for PES, MES and JES are presented in Figure 8,
Figure 9 and Figure 10, respectively.
To obtain the ground truth for the acquisition function we run the rejection sampling scheme discussed in [40].
As a comparison we also include the contours of some popular single-objective acquisition functions.
Visually the estimates all appear to be reasonably effective for this set-up.
Interestingly, the PES and MES prefer querying around the lower mode, whilst JES prefers the upper mode which perhaps offers a better trade-off between the gain in information about both the optimal input and output.
1.0 x 1 0.
x
Rastrigin 0.
1.0 x 1 0.
x0 .0 .0 .0 .0 .
Gaussian Process Mean 0.
1.0 x 1 0.
x
Gaussian Process Entropy 1 .
1.0 x 1 0.
x
PES ( x | D n ) 0 .0 .0 .0 .0 .0 .0 .0 .
1.0 x 1 0.
x
PES ( x | D n ) 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
1.0 x 1 0.
x
TS ( x | D n ) 0 .
[Figure]
[Figure]
1.0 x 1 0.
x
MES MC ( x | D n ) 0 .
1.0 x 1 0.
x0 .0 .0 .
JES ( x | D n ) 0.
1.0 x 1 0.
x
JES 0 ( x | D n ) 0 .0 .0 .0 .
1.0 x 1 0.
x
JES LB ( x | D n ) 0 .0 .0 .0 .0 .0 .
1.0 x 1 0.
x
JES MC ( x | D n ) 0 . 015 0 .0 .0 .0 .
1.0 x 1 0.
x
EI ( x | D n ) 0 .0 .
0.8 1.0 x 1 0.
x
PoI ( x | D n ) 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0.
1.0 x 1 0.
x
UCB ( x | D n ) 0 . 72 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
0.8 1.0 x 1 0.
x
KG ( x | D n ) 0 .0 .0 .0 .
K Hypervolume indicator
The hypervolume indicator is deﬁned as the area of the dominated region between a reference point r ∈ R M and the set of interest A ⊂ R M :
[Equation]
As discussed in the main text (Section 4), the HV indicator is sensitive to the parameterizations of the objective space.
In order to build a more complete picture about the performance of a multi-objective optimization algorithm, we consider tracking the HV under a number of different parameterizations.
Instead of deriving a family of transformation function, we consider a more general approach based on a different formulation of the HV indicator described in [23, 93].
In particular, the HV indicator can be written as an expectation over a probability distribution by performing a change of variables into spherical polar coordinates:
[Equation]
36) where Γ( · ) is the Gamma function and the scalarization function s λ : R M → R is deﬁned as
[Equation]
For the standard HV, the inverse weight distribution p ( λ ) is a uniform distribution over the surface of the M -dimensional unit sphere in the non-negative orthant S M − 1 + = { λ ∈ R M
[Equation]
: (cid:80)
[Equation]
( λ ( m ) )
[Equation]
The inverse weight distribution p ( λ ) controls the radial contribution for each point of A towards the HV.
Existing work mainly considers a uniform distribution over the weights in order to compute the standard HV.
We make the novel observation that we can assess the quality of the Pareto front in different regions of the objective space by varying this weight distribution.
We call the resulting utility function the generalized hypervolume (GHV) indicator, denoted by U GHV [-.
-].
This GHV satisﬁes a weaker version of the Pareto complete property: A (cid:23) B = ⇒ U GHV ( A ) ≥
U GHV ( B ) .
This property can be proved from the fact that the scalarization function s λ ( a ) is a [Figure] [Figure] monotonic increasing function in a ∈ A , which implies that the Pareto order is maintained.
This performance criteria satisﬁes the strict Pareto complete property when all of the weights have non-zero density p ( λ ) > 0 .
Intuitively, if a set of weights had zero density, then an improvement along these direction will not always result in a larger GHV.
This means that we will not always be able to distinguish between a set which strictly dominates another, which is essential for strict Pareto completeness. To generate different distributions p ( λ ) , we exploit the fact that the set S M − 1 + is isomorphic to the ( M − 1) -dimensional unit hypercube [0 , 1] M − 1 .
In particular, any point w ∈ [0 , 1] M −can be mapped onto the sphere to a point λ ( w )
∈ S M − 1 + with
[Equation]
Consequently, we can use any distribution with a ﬁnite support to generate samples on S M − 1 + .
For example, we use M − 1 independent Beta distribution Beta ( a ( m ) , b ( m ) ) to generate samples from w ∈ [0 , 1] M − 1 .
In Figure 13, we present the radial contributions to the GHV for different weight distributions p ( w ) .
To isolate performance in one particular radial region, we propose the use of a uniform distribution over some subset of [0 , 1] M − 1 .
For example in Figure 6, we chose three different subsets of [0 , 1] to isolate three different regions of the two-dimensional objective space.
Remark.
We are not the ﬁrst to observe the importance of weighting the HV towards regions of interest.
For example, an early paper [98] proposed another a weighted hypervolume indicator (WHV), U WHV ( A ) =
(cid:82) R M w ( z ) I
[ r (cid:22) z (cid:22) A ] d z , which introduces a weight function w : R M → R directly into the integral.
This approach is not as ﬂexible as the one presented here because it relied on designing hand-crafted weight functions and an involved symbolic integration procedure.
L Experiments L.1 Implementation details
All of the numerical results was implemented on Python 3.8 using open-source libraries: BoTorch (0.5.1) [3], GPyTorch (1.6.0) [30], NumPy (1.21.2), Pymoo (0.5.0)
[10] [37], PyTorch (1.9.0) [65] and SciPy (1.7.3) [84].
All computations were performed on a computing cluster, where we restricted the computation to a single CPU core of an AMD EPYC 7742 64-Core Processor @ 2.25GHz.
The code is available at https://github.com/benmltu/JES.
Gaussian process model.
For all the problems, we normalized the inputs and standardized the observations before performing Gaussian process regression.
We assume an independent model for each objective with a constant mean and Matérn 5/2 ARD kernel.
Additionally, we assume a zero-mean Gaussian observation noise with a homogeneous variance σ ( x ) = σ ∈ R M .
For convenience, we rely on the default model in BoTorch, which additionally places Gamma priors for the kernel hyperparameters and observation variance.
At each iteration of BO, we update the point estimates for the model hyperparameters and observation variance by maximizing the exact marginal log-likelihood using SciPy’s default gradient optimizer.
Optimizing the acquisition function.
To determine the next point, we optimize the acquisition function using multi-start L-BFGS-B [84].
We use the exact gradient inferred using automatic differentiation [65] for all algorithms except for the PES.
The automatic gradients inferred for the
PES acquisition function occasionally failed due to the differentiability issues arising from the damping procedure within the expectation propagation update [31].
As a result, we used the default 2-point ﬁnite difference method in SciPy to approximate the gradients of the PES acquisition function.
To initialize the multi-start gradient optimizer, we evaluated the acquisition function on a space-ﬁlling design of 1000 D random points.
The starting points were then chosen as the best performing points from the initial check.
The number of starting points that we used differed for each algorithm based on the overall expenses.
For the sequential experiments, we used 20 starting points for PES, 5 D starting points for the MES and JES, and 10 D starting points for the rest.
For the batch experiments, we used 20 starting points for PES and 5 D starting points for the rest.
For the information-theoretic acquisition functions, we could have initialized the starts near the sampled Pareto optimal points in order to promote faster convergence.
We abstained from doing this because we wanted to give a fair comparison between the existing approaches.
Speciﬁcally, we wanted to see how well informationbased acquisition function worked out of the box without any optimization compared to the existing optimized approaches.
Sampling the Pareto points.
To sample the Pareto optimal inputs and outputs, we ﬁrst sample
Gaussian process paths.
We achieved this by using an approximate sampling strategy based on random Fourier feaures (Appendix A.3).
We used the implementation in BoTorch and set the number of features to L = 500 .
To solve for the Pareto set and front, we used the multi-objective solver
NSGA2 [22], which was implemented in Pymoo.
We set the population size to be N pop = 100 , the number of generations to be N gen = 500 and the number of offspring to be N off = 10 .
In general, the multi-objective solver outputs an approximate set of Pareto optimal points with a size less than or equal to N pop .
To truncate this set into a size p , we used a HV truncation strategy.
In particular, we greedily selected points based on their contribution to the sample HV generated by the sample f s ( X n ) —the reference point is set to the current estimate of the nadir minus some error ˆ r n − 0 .
1 | ˆ r n | , where ˆ r ( m ) n = min t =1 ,...,n y ( m ) t .
This reﬁnement strategy can be implemented quickly using the expected HV improvement strategy discussed in [18].
We implemented this sampling and optimizing procedure in sequence, but naturally this can be executed in parallel because we are performing S independent computations.
As motivated by the wall times in Appendix L.9, implementing this step in parallel could be very beneﬁcial computationally.
Box decompositions.
We used the BoTorch implementation of the box decomposition strategy discussed in Algorithm 1 of [54].
The box decompositions were performed in sequence instead of being executed in parallel.
From the wall times presented in Appendix L.9, the time required to compute the box decompositions becomes more demanding as the number of objectives increases.
Conditioning.
We used the fantasizing feature in BoTorch to condition the current posterior on a collection of S independent Pareto samples of size p .
We treated the Pareto samples as noisy pseudo-observations in the conditioning.
Acquisition functions.
The benchmark comparison considers a range of popular acquisition functions that have all been implemented in BoTorch.
We implemented our own version of the multi- objective PES, MES and JES.
We now elaborate on the implementation details for each acquisition functions in the benchmark experiments. • TSEMO.
The Thomson sampling algorithm (TSEMO) [12] is a random acquisition function, which selects the point that maximizes the HV improvement according to a single sample of the Pareto front.
Unlike the original paper [12], we select the point that improves the HV of the sample frontier f s ( X n ) as opposed to the observation frontier generated by Y n .
We ﬁnd this adjustment to be sensible when there is observation noise.
We use the same modiﬁcation for the batch extension. • ParEGO/ NParEGO.
The random scalarization strategy (ParEGO) [50] and its noisy counterpart (NParEGO) [19] are one of the most popular strategies for multi-objective BO.
At each iteration, a random scalarization of the objectives is obtained by randomly drawing a weight w ∈ R M from the ( M − 1) -simplex.
To target all the Pareto optimal points, we use the augmented Chebyshev scalarization: min m =1 ,...,M w ( m ) ˜ f
( m ) + 0 .
01 (cid:80) m
=1 w ( m ) ˜
f
( m ) .
Here we have denoted ˜ f
∝ f as the objective function, which has been approximately normalized to [0 , 1] M using the observations Y n .
For the single-objective problem, a Monte Carlo estimate of the expected improvement and the noisy expected improvement for the ParEGO and NParEGO algorithm are used respectively.
The number of base samples for the Monte Carlo estimates is set to 128 .
For the batch setting, we sampled q different weights and optimized the acquisition function sequentially.
• EHVI/ NEHVI.
The expected hypervolume improvement (EHVI) [18] and its noisy counterpart (NEHVI) [19] are an improvement-based acquisition function for the HV indicator.
The number of base samples for the Monte Carlo estimates is set to 128 .
We set the reference point of the HV indicator to be equal to the observed nadir minus some error, ˆ r n − 0 .
1 | ˆ r n | , where ˆ r ( m ) n = min t =1 ,...,n y ( m ) t .
This dynamic reference point strategy was recommended in the supplementary material of [18].
For the batch setting, we considered a sequentially greedy optimization strategy. • PES.
The predictive entropy search (PES) [31, 33, 39, 40] acquisition function is approximated using expectation propagation.
We implemented this algorithm from scratch in BoTorch under the guidance of the supplementary material in [31].
We used S = {+
+} Monte Carlo samples and p = 10 number of Pareto optimal inputs.
We set the jitter for the matrix inversion to be 0 .
001 and the convergence threshold for the initialization stage to be 5% relative change in the mean and covariance.
If the expectation propagation failed to converge, we outputted a random vector from the already sampled Pareto sets as the ofﬁcial recommendation.
For the batch setting, we extended the approach outlined in [31].
This extension appears to be equivalent to the approach described in [33].
We optimized the resulting acquisition function using a joint optimization approach. • MES.
The maximum value entropy search (MES) [79, 85] can be approximated using all the conditional entropy estimates we devised in this paper.
We used S = {+
+} Monte Carlo samples and p =
10 number of Pareto optimal outputs.
For the Monte Carlo estimate MES-MC we set the number of base samples to 128.
For the batch setting, we considered a greedy optimization strategy [18, 19].
For the batch setting, we consider the lower bound described by (15).
We optimized the resulting acquisition function using a greedy optimization approach.
• JES.
The joint entropy search (JES) [79, 85] can be approximated using all the conditional entropy estimates we devised in this paper.
We used S =
Monte Carlo samples and p = 10 number of Pareto optimal points.
For the Monte Carlo estimate JES-MC we set the number of base samples to 128.
For the batch setting, we consider the lower bound described by (15).
We optimized the resulting acquisition function using a greedy optimization approach.
Optimizing for the recommendation.
To obtain the recommendation set, ˆ X ∗ n , we used the NSGA multi-objective solver to optimize the posterior mean µ n .
Using the Pymoo implementation, we set the population size to be N pop = 500 , the number of generations to be N gen = 500 and the number of offspring to be N off = 10 .
We select the p = 50 points that greedily maximizes the HV generated by the mean objectives.
The reference point for the HV truncation was set to the current estimate of the nadir minus some error: ˆ r n − 0 .
1 | ˆ r n | , where ˆ r ( m ) n = min t =1 ,...,n y ( m ) t .
Before applying the truncation, we ﬁrst augmented the Pareto set found at this iteration with the previous recommendation set ˆ X ∗ n − 1 in order to guarantee that some promising solutions weren’t missed due to randomness of the solver.
Optimizing for the Pareto set.
To obtain the baseline Pareto set, X ∗ , we used the Pymoo’s NSGA with a population size to be N pop = 1000 , the number of generations to be N gen = 5000 and the number of offspring to be N off = 10 .
L.2 Benchmark problems
We initialized each test problem with with 2( D + 1) training points using a random space ﬁlling design.
The details of the individual benchmark problems is presented below.
ZDT2 (D=2, M=2, Noise=10%).
The ZDT2 benchmark [96] is a bi-objective minimization problem min x ∈ X f ( x ) , over the D -dimensional hypercube X
=
[0 , 1] D , where
[Equation]
with g ( x ) = 1 − 9 D − 1 (cid:80) Di =2 x ( i ) .
For the experiments, we considered the maximization problem by negating the objective: max x ∈ X ( − f ( x )) .
The standard deviation of the Gaussian observation noise is set to σ 1 / 2 = (0 . 1 , 0 .
8) , which is estimated to be around 10% the range of the objectives from 1000 function evaluations.
For the HV indicators we use the reference point of r = ( − 11 , − 11) .
SnAr (D=4, M=2, Noise=3%).
The SnAr benchmark [27] is a bi-objective minimization problem min x ∈ X f ( x ) , which is deﬁned over a four-dimensional space, X =
[0 . 5 , 2 . 0] × [1 . 0 , 5 . 0] × [0 . 1 , 0 . 5] × [30 , 120] , where
[Equation]
with STY and E-Factor denoting the space-time yield and environmental factor, respectively.
The functions STY and E-Factor depend on the solution of an ordinary differential equation governed by a kinetic model, where the rate constants have been estimated by empirical tests [45].
Unlike the original paper [27], we have taken the logarithm of the output to accommodate for the additive noise in the objective space.
For the experiments, we considered the maximization problem by negating the objective: max x ∈ X ( − f ( x )) .
The standard deviation of the Gaussian observation noise is set to σ 1 / 2 = (0 . 12 , 0 . 08) , which is estimated to be around 3% the range of the objectives fromfunction evaluations.
For the HV indicators we use the reference point of r = (5 . 5 , − 5) .
Penicillin (D=7, M=3, Noise=1%).
The Penicillin benchmark [55] is a three objective minimization problem min x ∈ X f ( x ) , which is deﬁned over a seven-dimensional space, X = [60 , 120] × [0 .
05 , 18] × [293 , 303] × [0 . 05 , 18] × [0 . 01 , 0 . 5] × [500 , 700] × [5 , 6 . 5] , where
[Equation]
with PenicillinConcentration , CO2Concentration and TimeToFerment denoting the concentration of the desirable product, the concentration of the subproduct and the time to ferment, respectively.
The equations describing the pharmaceutical simulation are given in the original reference [55].
We used the open-source implementation available in BoTorch [3, 20].
For the experiments, we considered the maximization problem by negating the objective: max x ∈ X ( − f ( x )) .
The standard deviation of the Gaussian observation noise is set to σ 1 / 2 = (0 . 14 , 0 . 8 , 3 .
8) , which is estimated to be around 1% the range of the objectives from 1000 function evaluations.
For the HV indicators we use the reference point of r = ( − 1 . 85 , − 86 .
93 , − 514 .
Marine Design (D=6, M=4, Noise=0.5%).
This marine design benchmark [64, 72, 82] is a four objective minimization problem min x ∈ X f ( x ) , which is deﬁned over a six-dimensional space, X = [150 , 274 . 32] × [20 , 32 . 31] × [13 , 25] × [10 , 11 .
71] × [14 , 18] × [0 . 63 , 0 .
75] , where
[Equation]
[Equation]
with TransportationCost , Weight , AnnualCargo and SumOfConstraints denoting the transportation cost, the ship weight, the annual cargo transport capacity and the sum of the constraint violations, respectively.
The formal equations describing the functions are presented in the original references [64, 72, 82].
For the experiments, we considered the maximization problem by negating the objective: max x ∈ X ( − f ( x )) .
The standard deviation of the Gaussian observation noise is set to σ 1 / 2 = (10 , 77 , 132 , 0 . 07) , which is estimated to be around 0 .
5% the range of the objectives from 1000 function evaluations.
For the HV indicators we use the reference point of r = (250 , − 20000 , − 25000 , − 15 . 0) .
L.3 Sensitivity analysis
In the section we empirically analyse how sensitive the approximations to the information-theoretic acquisition functions are with different choices of Monte Carlo samples S and number of Pareto optimal points p .
For the experiments in the main section, we set S = 10 and p = 10 for all information-theoretic acquisition functions.
This selection is comparable to the existing literature [4, 31, 79, 85].
For the sake of brevity we present results only for one benchmark problem: ZDT (D=2, M=2, Noise=10%, [-q=1).
-]q=1).
To test the sensitivity with regards to the number of Monte Carlo samples S , we ﬁx p = 10 and ran the benchmark problem 100 times with S ∈ { 1 , 5 , 10 , 25 , 50 } .
To test the sensitivity with regards to the number of Pareto optimal points p , we ﬁx S = 10 and ran the benchmark problem 100 times with p ∈ { 1 , 5 , 10 , 25 , 50 } .
We report the mean log HV discrepancy with two standard errors over the runs in Figure 14 and Figure 15.
We report the wall times in Figure 16. Performance-wise there does not appear to be much variation for acquisition function when
S > 1 and p > 1 .
Naturally the wall times increase with the number of Monte Carlo samples S because the sampling and optimization of the Gaussian process paths are done in sequence (this step could be parallelized in practice).
Overall, for this example problem, there does not appear to be much beneﬁt in using a larger number of Monte Carlo samples S or Pareto optimal points p
[Figure]
[Figure] [Figure]
L.4 Noise levels
In the section we empirically analyse how sensitive the approximations to the information-theoretic acquisition functions are when we increase noise levels. In Figure 17 and Figure
[Figure] we plot the mean logarithm HV discrepancy for the JES and MES estimates on the ZDT2 (D=2, M=2, q=1) benchmark when the recommended points are obtained by maximizing the posterior mean.
In both examples, we observe that the performance decreases as the noise levels increases. On the whole the JES estimates perform very similarly across the board. For the MES results, we see that the MES-0 estimate is noticeable weaker than the rest even after the ad hoc correction described in Appendix E.
[Figure]
[Figure]
In this section we present the results of the standard hypervolume when we restrict the recommended Pareto set ˆ X ∗ to be a subset of the sampled locations: ˆ X ∗ = arg max x ∈ X N µ N ( x ) .
The complete results are presented in Figure 19.
Our main ﬁnding is that information-theoretic strategies have a tendency to not directly query the best performing points but instead opt for more informative points that will reduce the overall model uncertainty over the optimal points.
As a result of this behaviour, information-theoretic algorithms tend to perform well when we assess the Pareto set over the whole input space X and less so when we only assess the performance over the sampled locations X N .
If directly querying high-performing points is important, it might be advantageous to use an epsilon greedy strategy, where points are occasionally picked greedily according to the posterior mean
[Figure]
L.6 Comparing the entropy estimates
In this section we compare the results using the different conditional entropy estimates for the different experiments.
In Figure 20 and Figure 21 we present the results for the JES and MES estimates respectively.
We observed that the different conditional entropy estimates seem to perform similarly across the board.
As a result, we advocate the use of the cheapest estimate, which are typically the lower bound estimates.
On the Marine experiment, we observed that the zero-variance estimate was noticeably weaker even after we applied the ad hoc correction described in
Appendix E. Therefore, we generally recommend against using the zero-variance estimate if possible.
[Figure]
of the mean logarithm HV discrepancy with two standard errors over one hundred runs for the four benchmark problems for the JES algorithms.
On the top, we present the results for the sequential experiment, whilst on the bottom we present the ﬁnal results on the batch experiments with different batch sizes.
[Figure]
l og ( d HV )
[Figure]
L.7 Comparing the improvement-based algorithms
In this section we present the results for the improvement-based algorithms: NParEGO, ParEGO,
EHVI and NEHVI.
On the whole, we observe that the greedier strategy which ignores the noise seems to perform reasonably well compared to the strategy which accounts for the noise.
[Figure]
, we present the results for the sequential experiment, whilst on the bottom we present the ﬁnal results on the batch experiments with different batch sizes.
L.8 Generalized hypervolume
In this section we produce proﬁle plots of the generalized hypervolume at the ﬁnal time instance.
We only consider the median performance from the multiple runs when setting the approximation set to be the maximum of the ﬁnal posterior mean: ˆ X ∗ =
arg max x ∈ X µ N ( x )
L.8.1 Weight distributions
[Figure]
[Figure]
[Figure]
[Figure]
NParEGOParEGOTSEMOSobol
[Figure]
(D=
M=
Noise=10%
[Figure]
q=
[Figure]
L.8.3 SnAr (D=4, M=2, Noise=3%, q=1)
[Figure]
NEHVIEHVI
NParEGOParEGOTSEMOSobol
[Figure]
L.8.4 Penicillin (D=7, M=3, Noise=1%, q
=
[Figure]
NEHVIEHVI
NParEGOParEGOTSEMOSobol log( d GHV ):Penicillin (D=7, M=3, Noise=1%, q=1)
[Figure] [Figure]
NParEGOParEGOTSEMOSobol
[Figure]
L.9 Wall times
[Table]
[Table] [Figure]
[Figure]
[Table]
[Figure]
M Extensions
The discussion so far focussed on the problem of unconstrained multi-objective optimization where evaluations of the vector-valued function are executed individually or in batch.
Not all multi-objective optimization fall into this category, so we propose some additional extensions to handle more general problems.
Constrained optimization.
In the literature there are several examples of how to generalize information-theoretic acquisition functions to constrained optimization problems [6, 28, 31, 33, 41, 42, 66, 81].
We propose a simple extension for the JES acquisition function, similar to [81], to handle inequality constraints.
In particular, suppose we are interested in maximizing an M -dimensional black-box function f (1: M ) ( x ) subject to K black-box inequality constraints f
( M + k ) ( x ) ≤ 0 for k = 1 , . .
, K .
We can model the constraints f
( M + k : M + K ) ( x ) as additional independent objectives using the same observation model described in Section 2.
The only difference between the constrained setting and the unconstrained setting is the region of integration for the CDF.
In the constrained setting, the sampled Pareto front dominates only the vectors satisfying the constraint.
Hence the constrained CDF is now of the form p ( z ∈ D K (cid:22) ( Y ∗ ))
where
D K
[Equation]
( Y
[Equation]
) = { z ∈ R M + K : ( z (1: M ) (cid:22) Y ∗ and z ( M +1: M + K ) (cid:22) 0 K ) or ( z ( M +1: M + K ) (cid:31) 0 K ) } = { z
[Equation]
R M + K : z (1: M + K ) (cid:22) ( Y
[Equation]
, 0 K ) } ∪ { z ∈ R M + K : z (1: M + K ) (cid:31) ( − ∞ M , 0 K ) } =
[Equation]
[Equation]
[Equation]
[Equation]
[-(cid:31) 
-](( − ∞ M , 0 K )) .
This region can be decomposed into boxes in the almost same way as before.
The only difference is that we now have an additional box arising from region where the constraint is not satisﬁed.
In general, the JES (and MES) acquisition function estimates described here can handle any type of black-box constraint as long as we are able decompose the feasible objective region into boxes.
For example, interval constraints of the form f ( M + k ) ( x ) ∈ [ a, b ] , can also be readily handled in this framework.
Decoupled evaluations.
Evaluating all objectives at each iteration can be costly and perhaps unnecessary for practical problems.
To address this problem, researchers in BO have considered the use of decoupled [34] acquisition functions α M , which considers the quality of querying a subset of objectives { f ( m ) : m ∈ M} .
To the best of our knowledge, all of the existing decoupled acquisition functions in multi-objective BO are based on information-theoretic acquisition functions [31, 39, 42, 79].
The novel JES-LB2 and MES-LB2 described in this paper possesses this decoupling property because it can be decomposed into a sum of acquisition functions for each objective: α JES-LB2 ( x | D n ) =
(cid:80)
Mm =
α m ( x | D n ) .
By Theorem 4.1 of [79], the JES-0 and MES-0 can also be generalized to the decoupled setting via a marginalization argument.
Upon reviewing the experimental results of the cited papers, we see that decoupled evaluations typically provide only a marginal improvement over the non-decoupled strategies.
This is possibly down to heuristic choice to search the space of subsets.
A more principled search method based on some emerging ideas about how to optimize over categorical inputs [32, 61, 71] might be useful to obtain further improvements.
Multi-ﬁdelity Bayesian optimization.
For practical optimization problems of interest, it is occasionally possible to evaluate approximations of the true objectives that are much cheaper.
This additional degree of freedom has been exploited before in literature under the name of multi-ﬁdelity
Bayesian optimization [5, 6, 47, 58, 59, 77, 80, 91, 94].
These strategies have demonstrated some beneﬁt when optimizing with cost constraints on the function evaluations.
It is possible to adapt
JES to the multi-ﬁdelity by combining the ideas introduced here with the ideas before in the papers referred to above such as using cost weights and conditioning arguments on lower ﬁdelities.
The main obstacle to extending this work to the multi-ﬁdelity setting will likely arise from some lengthy algebraic exercises relating to the conditional entropy and box decompositions.